And with this demo,
we come to the very end of this module where we were introduced to
this Scrapy framework for crawling web pages.
We started this module off by understanding the differences
between web scraping and web crawling.
Beautiful Soup is typically used for web scraping.
Scrapy is an application framework for crawling websites.
Once we got set up with the Scrapy package in our local
machine using a simple pip install,
we saw how we could extract data from web pages in a
structured format using this Scrapy shell.
The Scrappy shell is great for prototyping.
It allows you to interactively test extraction of content from web pages.
We saw that the Scrapy shell downloads the contents of the web page
that you want to parse onto your local machine,
and it uses this cached page to extract content.
We saw how we could select HTML elements within a web page using XPath,
as well as CSS selectors from within the Scrapy shell.
In the next module,
we'll harness some of the powerful features that Scrapy
has to offer for crawling websites.
We'll write custom spider classes in Scrapy.