Before we start writing Scrapy spiders to crawl web pages,
let's get a big picture understanding of how Scrapy works.
And here is an architecture diagram for Scrapy.
There are many components in there, and it's rather confusing,
but don't worry, we'll break this down.
At the heart of the Scrapy application framework is the Scrapy
engine that controls the data flow and triggers events to which the
rest of the Scrapy components respond.
All of the code that you write to pass and extract information will
be accessed and invoked using the Scrapy engine.
Scrapy spiders are classes that you write as a user.
You'll write code, which goes into spiders,
and this code only involves extracting content from
the web pages that you want passed.
Essentially, when you're working with Scrapy,
you don't have to worry at all about how web pages are accessed,
what requests are made, how are they crawled.
All you need to do is write code to pass responses and
extract the items that you're interested in.
Once you get your Scrapy spider set up and you invoke it,
the engine will forward the request to download content from the
website to something known as a scheduler.
All of this is happening behind the scenes.
You don't have to worry about these.
The engine also asks for the next request that needs
to be made from the scheduler.
The scheduler will essentially say to the engine,
Here is the next URL that needs to be hit,
here is the next request that you need to make.
The engine will then process this request.
It will look at it and pass it on to a downloader.
And this downloader is responsible for downloading
the content from the internet,
from making the request to the site and downloading content.
The downloader fetches the URL from the internet using
some kind of HTTP client library.
And it sends this content back to the Scrapy engine.
Once the Scrapy engine has this response from the URL,
it'll take this content and forward that response object back to the spider.
So your code, which is in the spider, will have a response object to work with.
You just need to write code to process the response using selectors.
Whether it's XPath or CSS, extract the content that you need.
The spider will then return the items that have been scraped,
and these scraped items are then sent to item pipelines,
which can perform further transformations on the data.
Item pipelines can contain many kinds of data transformations.
You can clean the data, validate it, persist it to a database, anything.
When you work with Scrapy, most of your code will be written within the spider.
These are custom classes where you define custom behavior for crawling
and parsing pages from a site or a group of sites.
Any additional code that you might write might be for
data transformation operations, which are part of the Scrapy pipeline.
When you set up your custom spider,
you need to specify three bits of information,
what you want to crawl, the URLs that you want to download the content from.
You may also specify how you want this page to be crawled.
You'll access the request, you can access the items that you've extracted, etc.
You also need to specify how the downloaded content needs to be
parsed and what information you want to extract.
This you'll do using selectors.