I'm Clarke Bishop, a big data engineer.
In the last module,
you learned the basics of web scraping with Python using the
Requests and Beautiful Soup libraries.
Now let's explore the full‑featured Python web scraping tool named Scrapy.
You'll see how to get started with the Scrapy framework,
how to use the Scrapy shell to discover and refine selectors,
and you'll learn how to write a working scraper for truecar.com.
Scrapy is the most popular framework for writing Python web scrapers.
There's a lot to it,
so let's get started. Okay, Scrapy is an application framework.
It's for crawling websites and extracting data.
Sounds good, huh?
There's a command line interface, or CLI, to generate an
initial project with all the required files.
Now Scrapy expects code in a certain format to build a spider.
Once you learn the template,
it's much easier to code a spider, and the Scrapy shell is a
wonderful tool for troubleshooting and working out your
selectors. I said Scrapy is a framework,
but what exactly does that mean?
In the last module, we looked at Requests and Beautiful Soup.
Well, those are both libraries, Scrapy, though,
is a framework. A library makes available a set of
related functions you can call, but a framework gives you a complete application.
With a library,
you make calls to its functions, with a framework, it calls code that
you write according to the framework conventions.
You can do what you want with a library, just call the functions when needed.
With a framework,
you've got to put code in specific places so it can find things.
The main difference is that frameworks follow the Hollywood principle,
you know, don't call us, we'll call you. Now if you want to impress your friends,
tell them that a framework uses inversion of control.
That just means the framework is in charge.
Here's an overview of the exact steps you need to take to use Scrapy.
Let's start with the development environment.
We're going to let the Scrapy CLI set things up.
So create a new directory in your dev workspace folder.
I called mine scrapy and changed into the scrapy folder. Then
use pyenv to set up a local Python version.
We did this before.
Use pipenv to set up python and install scrapy, and finally, run
pipenv shell to start your virtual environment.
You'll need this running to use Scrapy's CLI in the next step.
Earlier, I mentioned that Scrapy has a command line interface, or CLI.
We can use the CLI to create all the files we need in our project.
Just type scrapy startproject, and name the project. I called
mine truecar, as we'll be scraping truecar.com.
Scrapy actually builds a complete project for you.
There's a scrapy.cfg file for config,
a bunch of Python files, and a spiders folder for our scraping code.
It doesn't create truecar_spider.py though, we have to write a little bit of
code after all, and that's where it goes. After we finish our spider and save
it in truecar_spider.py, all we have to do is tell Scrapy to run the spider,
and the command, scrapy crawl truecar, causes Scrapy to run the spider, then
dash ‑o is for output, and we save the scraping results to truecar.csv.
That's it.
Scrapy reads the page,
extracts the data, and saves the results in the file we specified.
Oh,
you want JSON format instead of CSV, just change
the output file to truecar.json.
Scrapy is smart enough to save the result in several common formats.
Here's a sample of the file Scrapy produced. I coded the
spider to extract a link to each vehicle,
along with the model information, mileage, and price.
The actual website had 30 cars listed on each page,
and Scrapy extracted the specific data for each and every car.
I could have extracted more data if needed.
If you want more data, just create more selectors.
Scrapy is very powerful. On this page, there were only 30 cars per page,
so a human would have to click the Next button.
With a little configuration, Scrapy can automatically
detect and follow all those next page links.
It's scraping nirvana.
I showed you everything except the actual spider code.
Don't try to read this now.
We'll go over all the details of the code in a bit. Just
realize all the power of scraping meant we only had to write
about 20 lines of code, that's it.