Hi, and welcome to this module on selecting elements using this Scrapy shell.
In this module, we'll move beyond Beautiful Soup HTML parsing,
and we'll work with the Scrapy application framework.
The Scrapy application framework is typically used for
web crawling at production scale.
Unlike Beautiful Soup, Scrapy is not a library.
It's a framework, which means you write a little bit of code,
which is then invoked by the Scrapy framework at the right time.
Before we get started with this Scrapy shell,
we'll talk about the differences between web scraping and web crawling.
We'll then see how we can extract data in a structured format using Scrapy.
We'll download and install the Scrapy package on our local machine,
and we'll interactively test extraction of content from
web pages using this Scrapy shell.
The Scrapy shell is great for prototyping your spiders,
which we'll write in the next module.
We'll understand how we can work with Scrapy selectors.
We'll see that Scrapy supports selectors for both XPath as well as CSS.