So I've been going on about advanced web scraping for the past few minutes now.
What exactly does advanced web scraping constitute?
To be honest,
it is a fairly arbitrary term, and you could include pretty
much whatever you wanted to inside it.
What does it mean in this course's context though? Let's talk through it with a
few examples. To provide a better perspective, let's see what basic web
scraping means. In basic web scraping, you have a web page. You apply some
scraping process, and you get to required data.
That's it.
Other than things like honeypot traps and ethical issues, you won't
encounter much resistance. What differentiates advanced web scraping
from this? In the context of this course,
when a web page or website that you want to scrape data from has an
additional layer of difficulty or resistance,
it comes under advanced web scraping.
It could be things like login pages, the necessity of having to
submit input values, etc. Let me explain in a little more detail.
In this course under advanced web scraping, we'll be dealing with
four specific things.
First, we'll be handling form data.
A web form on a web page is something that allows a user to enter
data that is then sent to a server for processing.
For example,
forms can be used to enter shipping or credit card data to order a product or
can be used to retrieve search results from a search engine.
The latter is the example we'll be focused on during the demo section.
Cookies are small files which are stored on a user's computer.
They are designed to hold a modest amount of data specific to a
particular client and website and can be accessed either by the
web server or the client computer.
This allows the server to deliver a page tailored to a particular
user, or the page itself can contain some script which is aware of
the data in the cookie and so is able to carry information from one
visit to the website to the next.
How are they relevant to web scraping?
We'll learn soon enough.
Next up, we'll be handling web pages that require user
input in the form of checkboxes,
radio buttons, etc. This is pretty relatable in the sense that all of us
have come across web pages which only gives you access once you click on
the Terms and Conditions checkbox or select the radio button corresponding
to your gender. And finally, uploading files to a web page during scraping.
All of us at some point have uploaded files like profile pictures or
resumes in order to create a profile and gain access to the data within the
website.
So, in this section, I'll teach you to upload an image to a website.
Learning to do this will come in handy as the process is the
same for other types of files as well.