[Autogenerated] So far,
we have done a limited set of operations on the scrape data by using
input and output processors and also item loaders.
But what happens if we want to do a series of operations one after the other?
That's where item pipelines come in.
IT, um, pipelines are an easy way to change together various transformations,
operations and checks on scraped raw data.
To understand this better, let's see some use cases where this might be useful.
IT, um, pipelines may come in handy for several use cases,
such as cleaning raw scraped data from the web,
things specifically about formatting results and removing unneeded characters.
Also, they are quite handy when validating scrape data,
making sure prices are numbers.
Also, that URL is are valid and, for instance,
validating in something that looks like an email address is
actually valid or properly formatted.
Another use cases checking that items contain values and are not empty.
Checking and removing duplicates is also a common scenario where item
pipelines are useful and storing items and files or databases is also
a typical use case of item pipelines.
So let's have a look at an example here we have an item pipeline.
Example.
We first import drop item from scrapie dot exceptions as well.
Use this to remove scraped items that we don't need to keep.
Then, within the price pipeline class,
we declare the value of sales tax that will need to
multiply to the price of the product.
We are scraping within process item.
We get the value of the price field from the item object,
and if the price does not include the sales tax,
then this is multiplied to the value of the price.
If a product does not contain a price field, then the complete item is dropped.
Remove from the scraped results, which is what drop-down does.
So as you have seen,
it's quite handy to use item pipelines for operations like these.