I'm Clarke Bishop, a big data engineer.
You already know about basic web scraping and how to build a spider with Scrapy.
There's still one common scraping problem we've not solved,
handling web pages that are generated via JavaScript.
In this module,
I'll show you two tools that can solve the JavaScript
problem: Selenium and Requests‑HTML.
First, we'll explore the JavaScript scraping problem in more detail.
It's important that you understand what's happening. Then a newer
library that's built on top of the Requests library that you already
know about, it's called Requests‑HTML. And Selenium, Selenium was
built to automate website testing, but it's useful for much more,
including web scraping.
Web scraping used to be easier, aww, the good old days.
All you had to do was download a page and parse the HTML.
Then JavaScript went and ruined everything.
Let's examine a specific example of the JavaScript problem.
ISeeCars uses JavaScript and that causes problems.
I'll show you exactly what's happening in the demo.
To understand the problem, we'll use two more Chrome Developer Tools.
One slows down page rendering,
and will temporarily disable JavaScript to see the page the way
our scraper sees it. We used the long URL for iSeeCars as an
example when we explored URL hacking.
It's a good example for another common scraping problem too.
I've got a Chrome browser window open in incognito mode.
First,
I'll show you another Chrome Developer Tools trick. Open Developer Tools, go to
the Network tab, and click the drop down to select Slow 3G.
This simulates a low bandwidth connection so we can better
visualize what happens. Now let's load our iSeeCars URL. It's going
to take longer than normal because we're running in Slow 3G mode,
and notice it says Loading for several seconds, with a 2017 Tesla
Model S in the background.
Wait, in the URL I asked for only Model 3s.
What's with the Model S? And loading seems to go on forever.
Finally, Loading goes away and you see a 2018 Tesla Model 3.
What is going on?
I asked for Model 3s, got a Model S, then it
suddenly changed to a Model 3 again.
One more Chrome Developer Tools trick to see exactly what's happening.
First, switch your browser back from Slow 3G to Online.
I don't want your browser going slowly.
Actually,
you don't really have to worry much about this, as the bandwidth
gets reset as soon as you close Developer Tools.
Now, click the three dot icon and pick Settings, or
just use the keyboard shortcut, F1.
Find the checkbox to disable JavaScript and check the checkbox.
Keep Developer Tools open and reload the page.
Now it's stuck on Loading, permanently.
This is happening because iSeeCars uses JavaScript to load up the final page.
JavaScript can't load because we've disabled it.
So no matter how long you wait,
this icon is just going to keep going round and round,
saying Loading. Uncheck Disable JavaScript,
then close Developer Tools and reload the page.
You'll see the wrong car again, a Model X, then the JavaScript will run,
and the data will load.
Scroll down below the ad to see the real data, the
Teslas we might actually want.
ISeeCars uses some custom JavaScript,
but more and more websites are using JavaScript
frameworks, like Angular or React.
The browser starts by downloading an initial HTML page that
includes some JavaScript, then the JavaScript automatically
runs and modifies the page.
You actually see the final rendered version that's
not at all what you started with.
The JavaScript rendering process can be so fast that you hardly notice,
iSeeCars with their custom JavaScript happens to be a bit slower,
so we can actually watch the page change.
The scraping problem, though, is we actually need the final rendered page,
not the initial HTML.
Here's the problem.
Requester Scrapy only downloads the original HTML
page, Python is not a web browser,
so there's no JavaScript and no way to render the updated page,
but you need the data in the updated page.
The original page doesn't have your data, it's no help.
What can you do?
I guess we're just going to have to add JavaScript to Python.
That's what we'll look at next.