In a practical web scraping project,
most of the work is in fine‑tuning the selectors.
You already know how to use Chrome developer tools.
Well, Scrapy shell is another invaluable part of your scraping tool kit.
Before we explore the spider code, let's understand Scrapy shell.
Start with the Wikipedia page we scraped in the last module.
At the Command Prompt, type scrapy, space, shell, followed by
space and the URL for the page.
Scrapy prints a bunch of logging information to the console.
What it's actually doing is the hard work of downloading and parsing the page.
The result is a response object. Response.css lets us use CSS
selectors, just like we did before. .Get at the end,
just extracts the text. Fire up Scrapy for homework and have
some fun with any page you want to scrape.
Here's another trick, type view with response in parentheses.
Scrapy will launch the full page in your web browser.
There's a whole lot more to Scrapy shell.
Remember, I told you,
Scrapy is a full application. When you're ready to quit the shell, type
Ctrl+D. Remember when you learned about user agents in the module on HTTP? I
ran into an interesting problem when scraping the Truecar website, and it's a
common problem you may see too. Scrapy's default user agent just says Scrapy
with some version information.
Well, Truecar did not like that user agent and returned a 403 error.
That means forbidden.
Fortunately, with Scrapy, there's an easy fix.
How about this?
You can specify the user agent right inside the scrapy shell
command. No more 403 error, just scraping fine. Yay! I know
it's kind of ugly to look at, but hey,
that's what a user agent string looks like. In the actual code,
there's a section in setting.py to configure whatever user agent
you want. Scrapy thinks of everything.
Finish the command with the actual URL and start using
Scrapy shell to test out your selectors.
We'll look at the spider code next, and I'll show you the selectors I created.
I want you to know about my process though.
Guess what I used. Chrome developer tools, Scrapy shell, and my Python IDE.
That's all it took to find and verify every selector I needed.