Before you build a Scrapy spider to crawl and extract content from web pages,
you might want to prototype and play around with the
content that you want to extract.
And that's exactly where you use this Scrapy shell.
This is an interactive shell where you can prototype how you plan to
crawl a site without actually building a spider.
The Scrapy shell is accessible by invoking the scrapy shell
command on your terminal window and specifying the URL of
the web page you want to scrape.
The Scrapy shell we make a request to that web page,
download its HTML contents,
and instantiate a bunch of Scrapy objects for you to work with.
Within this Scrapy shell,
you can use these Scrapy objects to extract and pass
the contents of your HTML page.
If you want to see what Scrapy objects are available,
the shell command will bring up all of the details.
Let's explore a few of these Scrapy objects to see exactly what they are.
The spider object within your Scrapy shell contains the definitions and
details of how a site will be scraped and the content extracted.
When you use the Scrapy shell,
the contents of your web page are extracted using the default spider.
It's possible for you to specify your own custom spider,
and we'll see how we can do that a little later on in this course.
The Scrapy object within the Scrapy shell is the main Scrapy module that gives
you access to the other Scrapy objects such as the request,
the selectors that explicitly select portions of your HTML document,
etc.
The crawler object in Scrapy is the API that provides access to
Scrapy's components such as its execution engine,
other settings, and so on.
You may not interact with the crawler directly,
especially for simple web pages,
but you should know that Scrapy uses it under the hood.
A Scrapy item is a container for the structured data
that you extract from a web page.
We haven't used an item yet, we simply opened up a Scrapy shell,
which is why it's empty.
We'll be working with items a lot more when we build our custom Scrapy spiders.
A request object within the Scrapy shell gives you access to the
URL request that was made to your web page.
The response that Scrapy receives from the URL target is
stored within the response object.
You can see that the response returned successfully with the status of 200.
The URL property of the response object shows you the contents of the URL
that have been downloaded and loaded into this Scrapy shell.
The status property on the response object gives you the
status code returned by the server, the status was 200.
Invoking the view command within the Scrapy shell is the equivalent of
running the scrapy view command from our terminal window.
The view object will bring up the web page whose contents were downloaded and
loaded into the Scrappy shell for parsing and extraction.
Once again,
I'll bring to your attention the fact that the web page URL that
is shown here is a file on our local machine,
which is where Scrapy has cached the contents of this page.
Make a note of the location of this file.
In fact, you can copy this URL over,
and you can use this local file to bring up a Scrapy shell as well.
Let's exit from our current Scrapy shell.
And I'm going to invoke the Scrapy shell command once again.
And this time, I'll specify an HTML file on my local machine.
This is a part to the location of the file where Scrapy had cached it earlier.
This will bring up the Scrapy shell once again,
and observe that the request was made to a local file,
and the response was received from the local file as well.
Thus, you can see that Scrapy can be used with live URLs,
as well as local HTML files.
Currently, our Scrapy shell has the HTML contents of this local HTML file.
Now if you want to load in the contents of another URL,
you can do that within the same Scrapy shell as well.
Here is the stackoverflow.com site.
Within the Scrapy shell,
I'll use the fetch command to fetch the contents of stackoverflow.com.
This will update the Scrapy shell with the contents of this new URL.
What has essentially happened is that the Scrapy shell has been re-loaded
with the contents of this new URL that you fetched.
Response.url shows you stackoverflow.com.
All of the Scrapy objects within the shell have been
updated with the contents of this new URL.