And this hands-on demo on using Scrapy item pipelines
brings us to the very end of this module.
We started this module off by getting a big picture
understanding of how Scrapy works.
We saw how the Scrapy engine interacts with the
custom spider classes that we built.
We saw the spiders were Scrapy classes where we define our crawling task.
Within spiders,
we wrote code to extract logical information using Scrapy items and
processors to pre-process and post-process the scraped data.
We introduced further automation into the scraping
process using Scrapy item loaders,
which allowed us to zoom in the specific portion of the
web page which we were interested in,
and then extract data in an automated manner.
We then brought everything together by applying data
transformations using Scrapy pipelines.
And with this,
we come to the very end of this course on scraping
your first web page with Python.
Now if you're interested in web scraping,
there are other courses that you can watch on Pluralsight,
Web Scraping: The Python Data Playbook,
Extracting Data from HTML with Beautiful Soup will explore the Beautiful
Soup parser library beyond what we covered in this course.
But if it's Scrapy that you're interested in exploring,
Extracting Structured Data from the Web Using Scrapy is the course for you.
And that's it for me here today.
I hope you enjoyed this course.
Thank you for listening.