In this demo,
we'll see how we can use pipelines in Scrapy to specify a
sequence of transformations that you can apply to your data once
they've been scraped from a web page.
I'm going to start a new project here called PipelineSpider.
Once again, as is usual, I'll cd into the subfolder,
which contains all of the files that have been generated via Scrapy.
We'll specify our Scrapy pipeline to transform data in the pipelines.py file.
But before that,
let's write out the other parts of the code that we are familiar with.
Let's start off in the items.py file where we define our Scrapy item.
This is the CompanyDetailsItem as before.
And I'm going to extract just two fields into this item.
In order to keep things simple,
we'll extract just the company name and its intraday price.
Let's now open up the pipelines.py code in order to
specify our processing pipeline.
I'm going to instantiate two classes here to transform our scraped data.
The first data transformation I'll specify within a
class called Above100PriceCheck.
Each class that you use to transform your data should
define a method called process_item.
This takes as its input arguments a reference to the item that we'll process,
as well as the spider object.
You can, of course, choose to specify your own custom transformation.
Here I'm extracting the intraday price from the item object,
converting it to a float object.
I then check to see whether the price of this stock is over $100.
If it is,
I update the company_price_intraday field to be equal
to the string greater than 100.
Pipelines can be used to perform any kind of
transformation you want on your scraped data.
Pipelines are often used to clean up the content
that you'd extract from a web page.
Return this transformed item from the process_item function.
You can specify as many transformations as you want
within this pipelines.py file.
Here is a second transformation,
which performs a price check to see whether the
intraday price for a stock is under $50.
If it is, then we update the price to be equal to the string less than 50.
Whatever transformation you apply,
return the updated item from the process_item function.
In order to enable Scrapy pipelines for this project and to
perform transformations on your scraped content,
you'll need to edit the settings.py file.
This file holds the configuration settings for your current project.
Scroll down until you find the setting for ITEM_PIPELINES.
Uncomment this line so you can configure the ITEM_PIPELINE that we set up.
You can use this dictionary here to specify all of the objects that
we'll use to apply transformations on our scraped data.
The key here is the name of the class that will perform the transformation,
and the value here is the priority of each transform.
These priorities determine the order in which the transforms will be applied.
A lower number indicates a higher priority.
The Above100PriceCheck transform will be applied
first and then the Below50PriceCheck.
Save the edited contents of the settings.py file,
and we can now move on to our spider definition.
The spider specification is very straightforward.
We'll use exactly the same spider with the ItemLoader
that we used in our previous demo.
Within the parse function,
we'll use an XPath selector as we did before to select the
individual rows associated with the companies.
We'll run a for loop to iterate over each row selected,
and we'll use an ItemLoader to scrape the contents.
The item loader will load the CompanyDetailsItem object.
We'll use the TakeFirst default output processor to get the
result in the form of a string rather than a list.
The next step is to specify the XPath to the ItemLoader,
which will extract the company name and its intraday price.
Once we have these specifications, it can yield details_loader.load_item.
Save the spider file as well, and switch over to your terminal window,
and invoke the scrapy crawl command for this spider.
If you take a look at the extracted contents printed out to screen,
you'll see that our pipeline transformations have been applied.
Any company, which has its intraday price over $100,
holds the string greater than 100 in the company_price_intraday field.
For a stock whose price was between $50 and $100,
we did not perform any transformation.
That is why the original price is available in this field.
And for those stocks whose prices were under $50,
we updated this field to hold the string less than 50.