Hi, and welcome to this module on scraping websites using Scrapy spiders.
Now when you're working with a web page,
you'll initially prototype how you want to scrape the content
from that web page using the Scrapy shell.
But when you want to productionize your code, you'll write spiders.
You'll define crawling tasks in classes using spiders.
And the Scrapy application framework will invoke your spider code
once it's downloaded content from the website and will then pass
this content onto your spider class.
We'll see how we can make our scraping process more logical and extract
structured data using Scrapy items and processors.
Processors are used to pre-process and post-process the scraped data.
We'll introduce additional automation into the crawling
and scraping process using item loaders.
Item loaders allow you to zoom into the portion of the document that you are
interested in and extract content in a more automated manner,
making your spider easier to maintain.
And, finally,
we'll see how you can chain data transformations for the
scraped data using Scrapy pipelines.