You already know the first step in web scraping, download the page.
In this section, we'll write code to download a page,
and I'll show you some JupyterLab tricks to explore the downloaded page.
We're going to import the needed Python packages,
use Python to download and save the page,
use JupyterLab to explore local files, inspect HTML with Jupyter,
and I'll show you some more Chrome developer tools tricks.
In the next cell, we're importing our packages.
There's still a version 3 of Beautiful Soup,
so be sure to install Beautiful Soup 4, then import from bs4.
I'm importing pandas so we can create a pandas
DataFrame with the data we scrape.
Pandas can then easily save the data as CSV,
or store it in a database, or many other things.
All right, that's done.
You've seen the next cell before.
Set a start_url, use request to download the URL,
then use Beautiful Soup to parse the HTML.
Saving a local copy of the HTML is handy,
especially if you're working with a dynamic page that's frequently changing.
I save the file using the w, or write mode flag,
but you can read the file by using an r for read mode.
Then just send the contents back into Beautiful
Soup to restore your soup object.
Okay, I've run that cell.
Notice in JupyterLab there's a sidebar panel that shows all our local files.
Pipfile and Pipfile.lock were created by pipenv to manage all our packages.
And the HTML file we just downloaded and saved to disk is there too.
See?
Downloaded.html.
With JupyterLab, we can right‑click on download.html,
then Open With, and pick Editor.
A new tab will open that shows all the nicely formatted source for the file.
Or right‑click on download.html, then Open With, HTML Viewer.
Jupyter will render the actual HTML page.
There's no CSS, so it's not as pretty, but try this.
Scroll down to the production and sales table.
That's the data we want to scrape.
Guess what?
Chrome developer tools works right inside of Jupyter.
Right‑click, Inspect, and voila!
Now I'll move developer tools down to the bottom,
but it's otherwise the same as before.
The top pane we used before is now the left pane,
and the bottom pane from before is the right pane.
You can move developer tools around by clicking the three‑dot icon,
then dock to the right side, dock to the bottom,
dock to the left side, or even undock and split out into a separate window.
Click table in the breadcrumbs, and there you go.
Table.wikitable is what we want to scrape and put into a pandas DataFrame.
Close developer tools and switch back to the Jupyter tab,
and let's move on to the next cell.