With Scrapy, you can build complex scraping applications,
but a key part of the work is building a spider.
That's what we'll dive into next.
Remember, the Scrapy CLI set up our entire project for us.
We just need to write the code for truecar_spider.py.
Let's do it.
I showed you the full code before.
It's only about 20 lines, but there's plenty going on, so let's take it apart.
Here's the overall structure.
We have to import Scrapy, but you already knew that.
Then we've got the TruecarSpider class.
It doesn't matter what you name it,
but the convention is to name it after the domain it scrapes.
We're going to scrape truecar.com.
That's why our Spider is named the TruecarSpider.
Notice that scrapy.Spider is passed inside parentheses in the class statement.
That's the Python way for inheritance.
Our TruecarSpider is going to inherit from the base class scrapy.Spider,
and that's how the framework knows how to run everything.
Now let's unpack each part of the TruecarSpider class.
First, there's the name that's set to Truecar, all lowercase.
The name is what we use in the Scrapy crawl command to say which Spider to run.
A good practice is to match the name to the domain we're planning to scrape.
And finally, two methods, start_requests and parse.
The Scrapy framework expects both of these methods and
uses them to control the Spider.
They do what you expect, start_requests lets Scrapy start a scraping session,
and parse reads the page response and extracts the data.
This is pretty cool.
Make a class and write two methods then let the
scraping begin, just two methods, and the first one,
start_requests, is pretty easy, make a list with the
URLs you want to scrape, in this case, I only used one URL,
but you can have as many as you need,
then loop over the URLs and call scrapy.requests for each one.
Only, what does yield mean?
I'll get to that in a minute.
For now,
just notice scrapy.requests is called with a URL and with a callback
Scrapy.requests will go download the page from the URL and send the
HTML content along to the callback.
In this case, the callback is self.parse.
So Scrapy sends the HTML right back to the parse method
in the Spider class we're writing.
A full discussion of yield is beyond what you need to understand Scrapy,
but here are the essentials.
When you call scrapy.requests,
there's going to be a delay while the page is loaded.
Yield is helpful in this type of situation where there's a delay.
Naturally,
you think of code as starting at the top and running all the way to completion.
Yield though, lets you pause than pick back up where you left off.
It's great for creating a series of data over time,
like a stream of web pages to scrape.
Pausing and resuming helps Python be faster and more memory efficient.
It definitely helps if you're scraping a lot of pages.
Essentially, you can think of yield like return, only lazy.
It kind of returns when it gets around to it.
Now, the parse method.
This is where the excitement happens.
Well, really, it's just a bunch of selectors to extract the data we want.
First, we get a list called all_listings using an XPath selector.
I used Scrapy shell and Chrome developer tools to discover
that all of the car listings were in divs.
Each div had a custom attributes named data‑qa that was set equal the listings.
So this XPath grabs all the div sections and puts them into a Python list.
There were 30 cars on each page, so the list ends up with 30 items.
Next,
we loop through each listing and build a Python
dictionary with the data we want, url, model,
mileage, and price.
Remember, these were the columns in the CSV file I showed you.
Scrapy expects to get back a dictionary,
and it uses it to create the output CSV.
Let's examine the entire loop.
In the last slide,
I skipped over a cleaning section to help you see the overall flow.
It turned out that for this specific website,
I had to do some extra selecting and data cleaning to get what I wanted.
I pulled out the make_model data to make it easier to drill down
then got the year and role model information.
To get the actual model string,
I had to do a replace clean out some unwanted HTML comments.
Next, I'll show you the HTML that we've been scraping.
Each item in the for loop is an object named Tesla.
Then we can easily use Scrapy's CSS selector to get just the make_model div.
We told Scrapy to go get a div that has a custom attributes named
data‑test that's set equal to vehicleListingCardTitle.
Then go get the first div that's inside.
We're left with just the inner div.
Here's the inner div we selected before and that we stored in make_model.
Now, it's easy to get the year.
It's the text inside of the CSS selector span.vehicle‑car‑year.
Double colon text tells Scrapy just to give us back to text, not the entire tag.
All right, year is now set to 2019.
We're getting there.
Model is a little trickier because of those HTML comments.
We can get model_raw the same way we got the year,
only we have to find and replace those HTML comments.
The end result is the string Tesla Model 3 with spaces separating each word.
You'll often see problems like this in real‑world scraping projects.
I don't know why this website inserted those HTML comments,
perhaps there's a business purpose,
or maybe they just wanted to make things hard on a determined web scraper.
Well, it didn't work.
Speaking of problems,
the way I got each listing was to look for a div that had the
custom attribute data‑qa set to Listings.
In practice though, it seemed like the name was inconsistent.
Inspect the site with Chrome developer tools, and it might say
VehicleListings instead of just Listings, just another real‑world
challenge you might have to code around.
I'll share a few good tricks for real‑world scraping.
Set your browser to use private or incognito mode, the results will be
closer to what Scrapy sees as there are no cookies, save a copy of the
downloaded page locally, that lets you troubleshoot problems in case the
selector labels change or something else changes, use an IDE and a
debugger to set breakpoints and step through the code to verify all the
selectors are working as intended, and break down the problem one HTML
chunk at a time,
just like I showed you. One last warning, web scraping is a brittle.
All it takes is a change in the website to break all your selectors.
And of course, we're dealing with live data.
So what you scrape right now will be different from what you get any
other time, just know, things are going to change.
We've covered a lot in this module.
Congratulations! We started with the Scrapy framework.
You learned the difference between libraries and
frameworks, and specifically, how to use Scrapy to set up
a project and how to launch a spider.
Then we explored Scrapy shell,
including how to specify the user agent and how to test out your selectors.
And we reviewed the TruecarSpider class that scrapes
a site in 20 lines or so of code.
We even covered Python's yield statement for doing a lazy return and saw how
to solve some common data scraping and cleaning problems.
There's one more big scraping problem we'll cover next, what to do about
pages that are dynamically generated with JavaScript.