In this demo,
we'll create a Scrapy spider to extract the same contents from the Yahoo!
Finance page that we've seen before, but this time,
we'll store the extracted content using Scrapy items.
Let's create a new Scrapy project in order to work with items and spiders.
I'll call this project the ItemSpider project.
This creates a subfolder called ItemSpider,
and within that, another subfolder called ItemSpider once again.
Cd into the subfolder.
And if you run an ls command here,
you'll find that Scrapy has generated a number of Python files.
All items that you define that will store extracted
content go within this items.py file.
So open up this file using your favorite code editor,
and let's create a scrapy.Item Python object.
I'm going to get rid of the default code that has been generated here and
create a new scrapy.Item called CompanyDetailsItem,
which inherits from the scrapy.Item base class.
All of the content that we extract from the Yahoo!
Finance page will be stored within the individual fields of this item.
So set up all of the fields that we want to get content for.
These are the same four fields that we worked with
using the Scrapy shell earlier.
Save the edits that you've made to this items.py file and create
a new file called company_details_item.py.
This is the file for your spider,
and we'll create this Python file within the spider's subfolder.
You'll find that the spider definition does not change much.
The way we extract data is the same.
However, we want to store our data in this CompanyDetailsItem.
So set up the import for this item.
Here is our spider class definition that inherits from scrapy.Spider.
The name of the spider is company_details_item.
And the remaining details remain the same.
The next step is to define the parse function that will be
executed when the spider is invoked.
The url variable holds the URL of the current scraped page.
This is the prefix URL for the relative URLs that we'll extract from the Yahoo!
Finance page.
Here is our XPath specification to select the HTML elements that
we're interested in: the name of the company,
the intraday price, the symbol of the company,
and the relative URLs.
The company symbol is available within an a link,
and the href attribute of that a link gives us the
relative URL for that company.
The extract function will allow us to extract all of the text
that we're interested in in each column.
We instantiate the count variable with the number of companies which
we are going to parse and set up a little for loop.
Within this for loop,
we'll instantiate the CompanyDetailsItem that will serve as
a container for the extracted content.
Assign the extracted content to the specified fields,
and make sure that the company_symbol_link has the URL as the prefix.
The Yahoo!
Finance page contains just the relative URL.
And we use this URL prefix to convert it to an absolute URL.
You'll specify the yield command to create a generator object that returns
the individual items to the invoker of this function.
And that's it.
Save the edits that you've made to this file,
and switch over to your terminal window.
Let's run this spider by calling scrapy crawl company_details_item.
That is the name of our spider.
You can see that by default,
the spider prints out all of the extracted contents from the Yahoo!
Finance page onto our terminal window.
Every individual item here is a JSON object
containing scraped content from the Yahoo!
Finance page.
If you want the scraped content to be saved out to a file,
you can specify the -o option when you invoke the spider to crawl your web page.
These additional command line arguments that you've specified indicate to Scrapy
that you want each item to be stored within this CSV file.
Hit Enter, execute this spider, and once the spider has completed,
you'll find that company_details.csv is present
within your current working directory.
If you open up this file,
you'll find all of the items that were extracted from the Yahoo!
Finance page are present here in the CSV format.