[Autogenerated] there is one common theme about all great software
frameworks and it is that it's well architected and Scrapy is no exception
and that's exactly what we're going to talk about Now,
one of the great things about a really well architected framework
is that as a developer you are given find were in control of what
you can do and achieve with it.
There are three fundamental aspects of Scrapy that allow
you to have that fine grained control.
These are the ability to limit download request,
which allows you to have complete control of what
information can be downloaded by the framework.
The ability to handle concurrent connections which is related to how Scrapy
handles tasks in parallel is also an essential feature and finally how Scrapy
is able to automatically follow request is another very important feature
which gives the system the capability to not become overloaded and is also
key for preventing crawl _____.
These three features get Scrapy developers great control
of what they can do with the framework.
So how is Scrapy architected first we have spiders.
These are custom classes written by developers to parse
responses and extract items from sites.
Then we have a core engine responsible for controlling the data
flow between all components of the system and triggering events
when certain actions and events occur.
Another essential component is the scheduler which receives
requests from the engine and accused them for feeding them
later when the engine request them,
then we have the downloader module refreshing
webpages and feeding them to the engine,
which in turn feeds them to the spiders and finally we have the
items pipeline which is responsible for processing the items once
they have been extracted by the spiders,
items represent data retrieved from websites.
Typical tasks include cleaning validating and data persistence,
such as storing the items in a database.
So to understand the architectural process better,
let's have a look at the following.
We have the web which contains millions of pages and vast amounts
of information to make use of that information.
We need an engine.
That engine needs a module that can schedule tasks which then
creates a list of requests that will be processed.
Those requests are then processed by the spiders with the
objective to produce items that can be later consumed.
The downloader is then responsible for actually fetching the
html content which will be parsed so the data required can be
extracted and the items populated.
Now that we have a good overview of scribes architecture.
Let's talk a bit about beautiful soup