I'm Clarke Bishop, a big data engineer.
You've learned some of the ways to stay out of legal trouble,
and you got a foundation in selecting elements from a web page.
It's time to make you dangerous and write some actual Python scraping code.
You'll see how to set up a Python scraping project
and install the required packages,
how to download a page to scrape and how to use JupyterLab to inspect the page,
then how to extract data from the page, clean it up, and get it into pandas.
You can't do much development without creating the
right Python scraping environment.
We'll get you set up in this section.
You're about to learn the basics of JupyterLab.
It's the next evolution of Jupyter Notebooks, and it's
perfect for exploring scraping with Python.
Pyenv and pipenv,
these are the latest tools for managing Python versions and installing packages.
We'll review and install the required packages,
and I'll show you some ways to verify your development
environment is set up and ready to go.
If you're a data scientist,
it's likely that you already know about Jupyter Notebooks.
Well, this is JupyterLab, the next evolution for Jupyter.
Jupyter Notebooks are great, but JupyterLab is fabulous.
We're exploring web scraping basics with Python,
but along the way, I'll show you some JupyterLab tricks.
The right pane in JupyterLab is just like the original Jupyter Notebook.
Let's start with the first cell.
Everything is commented out, but I want you to know how I set up my environment.
The best tools I've found to manage Python projects are pyenv and pipenv.
Pyenv lets you easily manage different Python versions.
Pyenv install 3.7.4 makes Python 3.7.4 available on your local machine.
Then pyenv local 3.7.4 sets the Python version for
the local project to that version, 3.7.4.
Pyenv exists to make it easy to set and switch between various versions, 3.7,
3.6, even a 2.7 version if you need it for some reason.
Then pipenv is the official replacement for pip.
It exists to manage Python packages in virtual environments.
Pipenv works well with pyenv, and the next statement,
pipenv ‑‑python 3.7.4 sets the virtual environment to use 3.7.4.
You must already have Python 3.7.4 available or you'll get an error.
Fortunately, we already took care of this with pyenv.
The rest of this will look familiar to anyone who's used to pip.
Pipenv install requests, well, it installs the request library.
Install beautifulsoup4, pandas, jupyterlab,
we're just installing all the packages we need.
With pyenv and pipenv installed on your development machine,
run these commands from the command line and you'll be ready to go.
Sometimes I like to verify that my environment is right,
and these commands let you ensure everything is set up correctly.
An exclamation point, or you can also just say bang,
tells Jupyter to run the statement on the command line.
So !pyenv local is just like running pyenv local on the command line.
It will report the Python version set by pyenv for the local project.
!python ‑V or python ‑‑version will report the local
python version that's currently available.
It should say 3.7.4.
Finally, pip and pipenv are compatible,
so !pip list will show all the packages that are installed and available.
We've gotten our environment set up.
Let's get to some scraping and write some Python code.