Kenneth Reitz is a prolific Python developer.
He's behind the Requests library I showed you in an earlier part of this course.
Well,
he's also solved the JavaScript problem for us with a
newer library called Requests‑HTML.
To take your web scraping to the next level,
I want you to understand more about how Requests‑HTML
actually works and how it delivers its magic.
What we really need is a way to run JavaScript inside of Python.
We've relied heavily on Chrome DevTools throughout this course.
Google's provided another key project: Puppeteer.
It's named after the person that controls a puppet.
Puppeteer is a headless Chrome browser.
You can think of it as a browser without a user interface,
only it's a JavaScript tool.
If only there was a Python version.
Fortunately, there is a Python port of Puppeteer,
it's called pyppeteer, clever huh,
and Requests‑HTML uses pyppeteer to fully render JavaScript.
Now, we can get the after JavaScript version of a page.
Headless browsers are the last piece of the web scraping puzzle.
We'll use pyppeteer inside of Requests‑HTML,
and in the next section, I'll show you Selenium.
Even better, while learning about web scraping, along the way
you've also learned how to develop web testing code.
Automated testing is another big use case for pyppeteer and Selenium.
Let's set up a project to explore Requests‑HTML, you've
seen these steps before. Make a new project folder and go
into the folder. Requests‑HTML, though, requires python 3.6.
Fortunately,
we can easily handle this requirement with pyenv. Then set up pipenv with python
3.6.9, install requests‑html, and launch the pipenv shell.
Since you already know how to use the Requests library,
this Python code should look familiar.
We specify the start_url, then get the page. The
response is saved in a variable named r.
What's different is that we're importing an HTML session and
assigning it to a variable named session.
This helps Requests‑HTML manage the process,
but it's not that different from what you've done before.
Requests‑HTML has built‑in CSS and XPath selectors, cool.
This code should seem familiar now. We want to select a div
with ID equal to cars_v2‑result‑list.
Then look for a child article tag.
There may be more than one result, and first=true
just gives us the first result, then assigns it to the variable, tesla.
Then we can use the tesla object to find an h3
object and extract the text. Easy.
Well, it would have been easy if not for JavaScript.
The result comes from the initial HTML page, so we get that
2017 Model S, not the Model 3 we wanted.
Requests‑HTML is amazing.
All we have to do is add one line to render the page with
JavaScript. Behind the scenes, it launches pyppeteer,
which runs JavaScript inside the headless Chrome browser.
Remember watching the page load in the browser back in the last section?
It took a few seconds, that's why I added sleep=5.
This setting has pyppeteer wait 5 seconds to ensure the page fully loads.
When you try this on a different website, you may have to adjust the delay.
The rest of the code is the same and the end result is the
Model 3 we wanted in the first place. Yay! Like pyppeteer,
Selenium is another automated browser.
Let's explore Selenium, next.