Once you've used the Scrapy shell to explore the
web page that you want to parse,
the next step is to set up a custom spider which you can run on demand.
In this demo,
we'll see how we can set up a simple Scrapy spider in order to
programmatically extract the contents of a web page.
I'm going to use Scrapy to extract the contents of this page here,
finance.yahoo .com /sector/ms_technology.
This gives you the stock price information for the
most popular technology stocks.
Here is a table,
and the specific columns that we're interested in in this table are the Symbol,
Name, and Intraday Price columns.
Before we extract any data, we need to select and explore this web page,
how it's structured.
I've selected this Microsoft Corporation text here.
I hit Inspect, and this will bring up Chrome browser tools.
We have a table element.
This table has a thead and a tbody.
And under the tbody element, we have multiple tr elements.
The tr elements make up the rows of the table.
And td elements make up the individual cells in a row.
This table, which has this information, has a unique identifier, scr-res-table.
And the actual text content that we want to extract is
present in the individual cells of each row.
I'm going to extract information using XPath.
I'm going to select a cell, right-click, and select the Copy XPath option.
However, I won't be using the XPath directly.
Instead, I'm going to edit this XPath to be more generic.
Let's head back to our terminal window.
In our current working directory,
I'm going to create a new directory to hold all of my Scrapy projects,
mkdir scrapy_projects.
If you run an ls command,
you'll see that this directory has been created under here.
In this module,
we'll create all of our custom spiders within this scrapy_projects directory.
This directory is currently empty.
We can create a project to hold our first spider by
invoking the scrapy startproject command.
The name of our project is SimpleSpider.
This is the first spider, so it's going to be a fairly simple one.
A project in Scrapy will generate all of the files that you
need to use to set up your own custom spider.
This command created a new directory under scrapy_projects called SimpleSpider.
Observe that Scrapy also gives you a few basic instructions on
how you can get started with your spider.
I'm now going to cd into this SimpleSpider folder that was
created within the scrapy_projects directory.
And that's where we'll work.
Once you're within this project directory,
you can run the ls command and see what it contains.
You see here that we have a config file that contains the
configuration settings for our Scrapy project.
We'll work with the default settings for this module.
Under that, we have another subfolder called SimpleSpider.
And if you cd into this SimpleSpider subfolder,
once again, you'll find a bunch of interesting files in here.
Run the ls command and observe that a number of Python
files have been generated for you.
All of these files and folders have special meaning,
and they allow you to plug into the Scrapy framework.
The actual code for your spider,
which parses your website and extracts the information,
is written within the spiders subfolder.
So far we've just created the project for a spider.
We haven't created any spiders yet.
Now this folder structure is confusing.
So let's view this within a Finder or an Explorer window.
We have our scrapy_projects folder that we had created.
Under that we created a SimpleSpider project.
You can expand this and view the contents and structure
of the project that Scrapy set up.
The actual code for our spider will be written under the spiders subfolder,
and that's where we'll work now.
We can create a new spider file manually,
or you can use the scrapy genspider command.
The name of our spider is company_details.
And we'll use this to parse the site finance.yahoo .com.
This genspider command will create a Python file,
which contains the template for a basic spider.
If you run an ls command in this spiders directory,
you'll find a new file in here called company_details.py.
If you open up this file in a Sublime Text editor,
you'll find that it contains the basic structure of a spider.
Import the scrapy module, and this class is called CompanyDetailsSpider,
which inherits from the scrapy.Spider object.
We have a class variable within the spider,
which contains the name of the spider.
This is important because this is what you'll use to
invoke and execute the spider.
This is the spider name that we had specified when
we used the genspider command.
The allowed_domains class variable allows you to
restrict what domains your spider can pass.
We'll restrict our spider to finance.yahoo .com.
And I'll update the start URL to be more specific.
We want our spider to crawl the page at
finance.yahoo .com /sector/ms_technology.
The parse method within your spider is where you specify how you
want to parse the contents of this web page.
I'm now going to edit the code within this function to
specify how I plan to parse this Yahoo!
Finance page.
The response object from the URL that we want to parse is
available as an import argument to this function.
I've specified an XPath selector that will allow me to access
all of the company names within this table.
Along with the names of all of the companies in that table,
I want to extract the intraday price information as well.
And I use a separate XPath to get this information.
Use the extract function so that all of the company prices are
extracted into the company_price_list variable.
I now have two lists, one of company names and one of company prices.
All I need to do is to run a little for loop from 0 to count and
extract the individual company names and the corresponding
prices and print them out to screen.
And that's all I plan to do within our very first simple spider.
Save the edits that you've made to this spider file,
and let's switch back to our terminal window.
I'm now going to move up two directories so that I'm
in the SimpleSpider directory, the outer directory.
In order to run the spider, to crawl the Yahoo!
Finance page that we had specified,
all I need to do is specify scrapy crawl and the name of the spider,
company_details.
And Scrapy will take care of the rest.
It'll connect to that web page, crawl the contents of the web page,
and apply the parse function that you had specified to extract the company
names and the intraday prices and print them out to screen.
If you need to debug your spider,
there is other metadata information available as well.
You can see that our spider made two requests to the Yahoo!
Finance page and got two successful 200 status responses.