In this demo, we'll get set up with the Scrapy framework on our local machine.
And we'll see how we can use this Scrapy shell to download and
work with the HTML contents of a web page.
Here I am on the terminal window of my local machine,
and in order to get Scrapy set up on the machine,
I installed the Scrapy package with a simple pip install.
If you already have Scrapy installed,
the --upgrade flag will ensure that you get the latest version.
The latest version at the time of this recording is 1.7 .3,
and that's what we'll use across all of our demos.
Once Scrapy has been successfully installed on your local machine,
running the Scrapy command on your terminal window
will bring up this Scrapy mime page,
and this will show you all the commands that you can use with Scrapy.
We'll explore a few of these commands in just a bit.
This Scrapy version command will show you the current
version of Scrapy that you're working with.
The demos in this course work with Scrapy version 1.7 .3.
Scrapy is a web framework that allows you to create
spiders that help you crawl websites.
The scrapy bench command runs a quick benchmark on your local machine to
check how Scrapy performs on the hardware that you have.
The scrapy bench command will run a local web server and try to crawl it as
quickly as possible and give you performance statistics.
If you scroll down below,
you'll see a few details about what exactly Scrapy bench did.
It crawled 485 pages at 2400 pages a minute.
You can see the request that it made, the responses that it received,
the number of bytes that it received in the response and so on.
Let's get started with Scrapy by downloading the contents of this URL here.
This is a Stack Overflow link that talks about the
difference between Beautiful Soup and Scrapy.
Copy over this URL here, and let's switch back to our terminal window.
I am going to run this scrapy fetch command and specify the Stack Overflow URL.
Scrapy fetch will download the contents of the HTML file from
this URL and write the contents out to screen.
When you hit Enter,
this will spit out onto your terminal window the entire
HTML contents of this Stack Overflow link.
If you want to store these HTML contents onto a file,
you can use the scrapy fetch command and re-direct the output of this
command into an HTML file on your local machine.
The downloaded HTML contents are now present in my current working
directory in the file stackoverflow_file.html.
You can open up the file on your local machine and view the HTML
associated with the Stack Overflow URL that we just saw.
You should know though by default,
Scrapy just scrapes the static content of any web page.
If your web page has dynamic content,
dynamic content can be scraped only using other extensions of Scrapy,
not the default Scrapy installed.
Dynamic content is content loaded into your web page
after the initial HTML has been rendered.
Scrapy comes with a bunch of configuration settings that you can tweak.
These settings determine how you crawl your website.
Scrapy settings will give you access to those settings.
When you first start off with Scrapy, your settings are set to the default,
which is why Scrapy settings return nothing.
When you use Scrapy to download the content of an HTML page,
Scrapy views the page or sees the page in a certain manner.
The scrapy view command opens your URL in the browser
the way Scrapy will see the URL.
So, if you have dynamic content within this URL, that will not be rendered.
Executing this command brings up the Chrome browser on my local
machine and renders the Stack Overflow page.
Take a look at this URL though.
It's not the original Stack Overflow URL.
Instead, it points to a file on my local machine.
The scrapy view command has downloaded the HTML contents of this URL,
saved it somewhere on my local machine,
and has displayed the page using that local file.
When you use the Scrapy shell to prototype your web
scraping command as we'll do in the next clip,
Scrapy will work on this locally cached file.