In this demo,
we'll see how we can use items in Scrapy to extract
structured details from a web page.
Before we use Scrapy items within an actual spider,
let's explore these items using the Scrapy shell.
I've brought up the Scrapy shell here, which passes the technology page on Yahoo!
Finance.
Within the Scrapy Python shell,
let's go ahead and import the scrapy module and create a new item object.
This is the CompanyDetailsItem.
An item in Scrapy is just a container object,
which is used to collect and store the scraped data in a structured format.
The individual bits of content that we extract from the web
page are stored as fields within an item.
The item here represents one row of stock information from the Yahoo!
Finance page.
It contains the company name, the intraday price,
the symbol for a particular company, and a link to that company's page on Yahoo!
Finance.
An item that you declared in Scrapy inherits from the scrapy.Item base class.
Once we've defined the item class, we can instantiate a new item object.
This is the CompanyDetailsItem, and it's currently empty.
Import the pretty-print Python module so that we can view
the extracted content from our web page.
The first bit of content that I'm going to extract is the list of companies.
Here is the XPath that we are familiar with.
The company name information is present in the cell at index 2,
that is, the third column.
This XPath matches all company names within the same table and
returns a list of companies as you see here on screen.
If you want to access the name of just the first company here in this list,
you can use the get function on your XPath selector,
and that will give you Microsoft Corporation.
Another way to access the first item that was matched using
the selector is to use extract_first,
and that gives you Microsoft Corporation once again.
The company_details_item can be accessed like you would a
Python dictionary using square brackets.
Right now we have no company_name specified, which is why we get the KeyError.
We'll now use our XPath selector to select the first
company in our list and assign this to the company_name
field in our company_details_item.
The company_name key within our item is now associated
with the value Microsoft Corporation.
I'll now specify the XPath to extract the other bits of
content that we're interested in on this page.
This is the XPath to get the intraday price for each company.
The get function returns the first element that matches,
which is the price for Microsoft.
We can now assign this price to the
company_price_intraday field within our Scrapy item.
Let's now access the symbol information for Microsoft Corporation.
This is present at index 1 within our table.
And let's assign this as a field to our company_details_item,
the company_symbol field.
And this is how we extract the specific bits of information that we are
interested in from a web page and assign it to the item container.
Here is another bit of information that we want to extract,
the URL to that particular company's page on Yahoo!
Finance.
The URL is available as an href attribute on the company_symbol field,
and this is the URL for Microsoft.
Observe that it's a relative URL.
In order to get the absolute URL,
you need to prefix it with the finance.yahoo .com /sector/ms_technology URL.
Let's go ahead and assign this absolute URL to the company_symbol_link field.
This now contains the complete URL to the page for Microsoft.
Now that we've seen how we can extract information from a web
page and store it within the Scrapy item,
let's incorporate this in a Scrapy spider.