I'm Clarke Bishop, a big data engineer,
and in this module you'll learn about the web scraping process.
We'll be looking at human browsing versus automated scraping,
an overview of HTTP, the HTTP protocol, and URL hacking.
Let's go.
You can think of web scraping as automated browsing.
It works just like a human using a web browser but
without requiring human interaction.
Let's compare a human using a web browser to a web scraper.
A human starts by entering a URL or clicking a bookmark.
A web scraper begins by setting a start_url variable.
The human's browser then downloads the HTML at that URL.
Well, guess what?
The web scraper does the same thing, downloads the HTML.
Next, the human's browser parses the HTML,
pulls in all the images and other content,
and renders the page for the human.
A web scraper also parses the HTML, only it converts the HTML into data.
The human reviews the page for relevant information.
Well, that's what we'll do with a web scraper, extract useful information.
The human tries to interpret and understand the web page
to derive meaning. The web scraper, though,
transforms or aggregates the data to get it ready for further analysis.
Ultimately, it has the same objective: to derive meaning and insights.
Finally, the human will hopefully remember the information,
or at least bookmark the page.
The web scraper just saves the data.
Web scrapers never forget.
All this happens very quickly, and we humans do it automatically.
Load a page, review a page, click a link, and repeat.
Web scrapers aren't that different.
They just move onto the next scraping job.
The bottom line is that web scrapers are simply automated web browsers.
Humans like pretty pictures and process the information more emotionally.
Web scrapers simply capture and store the data for further analysis.