In this demo, we'll see how we can use Scrapy item loaders,
which make it easy to populate scraped content into an item.
We'll also see how we can perform processing on content loaded into Scrapy
item fields by using default input and output processors.
I'm going to start a brand new Scrapy project here called ItemLoaderSpider.
Once this project has been created,
let's move into the subfolder within this project where the
Scrapy-generated Python files are present.
Under our current working directory, we have the items.py file.
This is where we'll define our Scrapy item.
I'm going to edit this file and remove this code that was generated by default.
And I'm going to import the MapCompose object from scrapy.loader .processors.
This MapCompose object will allow me to specify a custom function to
process the scraped data that I extract from a URL.
I'm going to define a helper function within this
items.py file called full_links.
The input argument to this helper function is the relative company_symbol_link.
In order to convert this relative URL to an absolute URL,
I'll prefix the URL of the scraped page.
This is the finance.yahoo .com page.
Earlier,
we were explicitly adding this prefix into every URL that we
extracted here and defined the helper function.
Let's go ahead and define our item here exactly as we've done earlier.
This is the CompanyDetailsItem that inherits from scrapy.Item.
I'll use this item to store three bits of information
that I'll extract from the Yahoo!
Finance page, the name of the company, the company symbol,
link, and its intraday price.
Observe that the Scrapy field object that I've instantiated for
the company_symbol_link has an input processor.
The input processor allows us to process the data as
soon as it's scraped from the web page.
The extracted content is then parsed through this input processor that
we have specified before it's stored in the item.
The final processed data is what the item will have.
And the processor that we've specified will convert the
relative URL links to absolute links.
Save the edits that you've made to this items.py file and
create a new Python file called
company_details_itemloader.py within the spiders folder.
This is the Python file for our spider.
Within this spider, we'll extract content as we did before from the Yahoo!
Finance page, but this time,
we'll use an item loader to make this a little easier.
The basic definition of the spider remains the same.
The spider is called CompanyDetailsItemLoaderSpider,
and its name is company_details_itemloader.
Within the parse function, we'll use an ItemLoader to extract information.
Instead of selecting individual cells at a time,
we'll use the XPath selector to select one row at a time.
Remember, one row contains the details of a single company.
The rows that are selected using this XPath selector are
stored in the company_results variable.
The company_results variable will contain as many selected
elements as there are rows in that Yahoo!
Finance table.
Iterate over each company_result,
and let's use an ItemLoader to extract information.
An ItemLoader is a convenience mechanism within
Scrapy to populate scraped items.
It automates common tasks like parsing and extracting raw data.
For each bit of data that you want to extract,
you'll instantiate an ItemLoader object.
And within that, you'll specify the item whose fields you want to populate.
In our case, it's the CompanyDetailsItem.
The ItemLoader also takes as an input argument a Scrapy selector that
you'll use to extract the content that you're interested in.
You'll then specify a mapping within this loader for
every field that you want populated.
For example,
I want the company_name to be populated using the XPath specification here.
The XPath that you specify for each individual field has to be relative
to the selector object associated with the ItemLoader.
Our selector here is an individual row corresponding to a particular company,
and the company name information is available in
the cell at index 2 in this row.
For each field that you want populated within your item,
you need to add an explicit mapping to your ItemLoader.
We've specified this mapping using XPath selectors.
You can also do this using CSS selectors.
After you iterate over the rows for each company and instantiate an ItemLoader,
yield item_loader.load_item.
As long as your XPath specifications and your selectors are correct,
Scrapy will take care of the rest.
Save your spider definition,
and let's switch over to the terminal window to run your spider.
Invoke the scrapy crawl command and specify the
company_details_itemloader spider.
The extracted content is printed out to screen.
But if you examine these details, you'll see that something is a little off.
We have the keys to the dictionary that are set up correctly,
but observe that all of the values are in the form of list objects.
For each bit of content that we're interested in,
it was extracted using a list format rather than as a plain string.
This is something that we'll need to fix, and we'll do that in just a bit.
Observe the values for the company_symbol_link field in each item.
The relative URLs that were originally extracted have been
converted to absolute URLs using the input processor that we
have specified for this field.
This part is all good.
What we need to do now is to fix this issue where
every value is in a list format.
And we do that using a default output processor called
TakeFirst that is available in Scrapy.
As the name implies, TakeFirst helps you extract the first item in a list.
We'll specify this TakeFirst as an output processor.
Output processing is performed on your data after the data
has been extracted from the scraped page.
If you want the same processing to be applied to all fields within an item,
you can specify a default processor for the entire ItemLoader.
Here, TakeFirst will apply to all scraped fields in an item.
Let's switch back to our terminal window after saving the
changes that we've made to our spider.
I'll now use the scrapy crawl command to run this spider once again.
And observe the content that was extracted using the spider.
Observe that every value is now in the form of a string rather than a list.
And if you want to save the scraped content in the form of a CSV file,
we can specify the same command line arguments as before,
-o, -t.
Once the spider has run through, you can run an ls command,
and you'll find a new CSV file in your current working directory.
And the CSV file will contain the contents extracted from the Yahoo!
Finance technology page.