[Autogenerated] So let's talk about spiders.
Not those amazing creatures that spin up webs to catch insects at
night but instead fetch data from the internet.
So how can we define the term?
Spider spiders could be defined as python classes,
where very specific custom behaviors are defined for
crawling and parsing pages from websites.
So, in essence, they are classes that are organized in a way,
which makes it easy for the scrapie engine to retrieve content from the web.
So harsh scrapie spiders implemented.
There are three fundamental aspects that need to be taken into
account when implementing a scrappy spider.
The first aspect that needs to be considered when implementing a spider is to
understand and scope what could be crawled in practice.
This means that you are elves that we want to crawl,
are specified within the starch request method.
Once that is known, it is important to look at how the content can be crawled,
essentially, how the quality mechanism is going to be implemented.
This in practice means that if we want to follow the links in that page,
or whether we want to look at a portion of the page,
and finally, once that is known,
it's important to consider how the data is going to be parse,
so, in essence,
how the data is going to be extracted after it has been retrieved.
This in practice means that we have to specify and determine what parts
of the web page are going to be process and extracted.
So basically what information we're interested in for post processing.