If you're scraping web pages, and you want to go beyond the basics,
Scrapy is what you use.
So what is Scrapy?
It's an application framework for crawling websites
and extracting structured data.
When you work with Scrapy,
you'll see that it's not about accessing a single web page,
downloading its content, and extracting information.
It's for crawling websites.
Scrapy was originally built for web scraping,
but it's now used for web crawling.
There's actually a difference between web scraping and web crawling.
Let's compare and contrast the two.
Web scraping involves extracting data directly from websites.
Web crawling involves downloading the contents of websites,
extracting the information, and then indexing that information.
You're storing the data in a structured format and
then adding indexing capabilities.
Web scraping is what one would use for quick and dirty data analysis.
It has a somewhat unsavory reputation.
Web crawling, though, is performed by all search engines,
and its associated with legitimate use.
This is, of course, just its reputation.
There's really nothing wrong with web scraping per se.
The scope of web scraping tends to be rather specific.
Scrape prices from Amazon because I have a competitive e-commerce site and I
want to see what Amazon charges for different products.
That's web scraping.
Web crawling, on the other hand,
tends to have a broader scope and is more general purpose.
Crawl all sites linked off of Amazon so I can access
all of the seller information.
I think the primary difference between scraping and crawling lies in the scale.
Web scraping is small scale and it results in a very specialized dataset,
which you need for a specific purpose.
Web crawling is large scale, you create a document corpus,
which can be used in the future.
Another thing you should realize when you use Scrapy is that it's
an application framework and not a library.
framework versus library involves an inversion of control.
Frameworks call your code.
You call code which live within libraries.
When you work with libraries, such as, say,
the HTTP client libraries that you used earlier,
you specifically invoked functions from the libraries.
for your use case.
You call the library functions.
When you use application frameworks,
you write your little bit of code in a specific place,
the framework will call your code at the right time,
it will do a bunch of stuff behind the scenes.
With the library, you're writing some application that you want to use,
and you invoke the library for specific portions that you want implemented.
In the case of a framework,
the framework defines the application and invokes
your code for specific portions.
So what we've used so far, that is Beautiful Soup, that is a parsing library.
Scrapy, what we're going to use now, is a web scraping framework.
When you use Scrapy, you can extract data in a very structured format.
However, you need to know what you're looking for.
You need to be able to parse the HTML and extract the
information that you're looking for.
And because parsing the HTML structure is involved,
using Scrapy is inherently somewhat fragile,
like regular expressions, Beautiful Soup,
and other tools that you might use for scraping.
If the website changes the structure of its content,
then your code needs to change as well.
Scrapy allows you to specify expressions that can select HTML elements.
They can be selected for processing using selectors.
You can specify selectors and Scrapy using CSS class names,
where the class is an attribute of an HTML element,
or using XPath expressions to navigate down your tree.
A selector and Scrapy is nothing but a specification of
what HTML elements on a web page,
ought to be selected for further processing and parsing.
Scrapy supports both XPath, as well as CSS selectors.
And we'll study both in our demos.