[Autogenerated] now let's talk about processors and what they are and why
they are useful processors allow the pre and post processing of raw
scraped data so it can be formatted and clean so what data for many and
cleaning steps are involved by default.
The spider gives us raw scraped data which has been
directly extracted from the html markup.
The data can be processed by input processors which can process
the extracted data as soon as has been received.
There are also output processors.
After collecting all the data and populating the item object.
The alpha processor is called with the data previously
collected and process using the input processor.
So what type of built in processors exist First we have the identity processor,
this is the simplest processor which doesn't do anything,
it just returns the original values unchanged.
Then we have to take first processor,
it is the first non null or non empty value from the values received.
So it's typically used as an output processor to single valued fields.
Then we have the joint processor which returns the values joined by a separator.
The composed processor works with the composition of
functions the map composed processor,
which is the most common one is quite similar to the composed processor.
The difference is that the input value of the processor is iterated
and the first function is applied to each element,
the select James processor queries the values using the
JSON path provided to the constructor and returns the
output to understand this better.
Let's look at an example here we have input and output
processors being used on the items dot py file to clean up
some raw extracted data for books.
First we import the required modules such as Scrapy and also the
processors we need such as join map composed.
And take first notice how we also import,
remove tags which as its name implies, is used for removing tags from the markup.
Then we have the filter price function that checks is the value of the
price is actually a number within the book class.
We use the map composed processor to remove all the markup tags from the raw
scraped data and call the joint function to return a single string which is
assigned to the title field for the price field.
The process is the same with the exception that besides the removed tags being
called to clean the raw data from the mark up the filter prices also called the
output processor in this case is done by invoking Take first well later in the
demo explore this with further details