[Autogenerated] We've covered quite a bit of ground,
and it's now time for a demo we'll put into practice.
All the concept we've explored in this module and will implement
a scraping pipeline sounds really exciting.
So let's jump right in.
We're going to be scraping the books dot to scrape dot com website,
which is also a side provided by the makers of the scrapie framework.
As you can see,
it's a site that contains books grouped into different categories.
So with the S code open, let's generate a new project with the following command.
Notice how it uses the crawl scrapie template.
Let's select the Project folder that has been created.
Let's also remove this code from book crawler dot pie and items dot pie.
Then let's other fields.
We are interested in scraping going back to book crawler DOM API.
Let's import the Codex Library,
which we will initially use to write data to disk.
Let's change the link extractor rule to point to the sub URL is we
want our spider to crawl in this case catalog,
Ford slash category and also change the callback from parts item to parse page.
Within parse page,
we create an instance of books item and also define the XPath
selector to get the markup data for each book.
We can extract the title field by using the selector and the price feel with
this one and the image URL we can extract it with the selector and upend that
result to the base URL of the site and the book URL could be extracted in
almost the same way as the image URL.
The difference is that one points to the web page of the book
and the other to the image use for the book.
Then, just like we did previously.
Using code x dot open, we write each field value to a text file.
There's one important bit we forgot to do this For all the books found.
We have to scope this code under this for loop.
I prefer to use raise instead of response.
It's just shorter to type, so I'll change that as well.
So let's run the spider using this command.
As you can see, it's scraping the site and books dot txt has been created.
Let's see the results we can see.
Each book has the title, a price and two links,
one for the image and another for the book page itself.
What we have just done is to create a cross spider by using a standard approach.
Let's now use processors,
item loaders and pipelines to make the most out of the scrapie framework.
So let's remove all this code an import item loader.
Then let's create an instance of it.
We pass it the books item and the current book as its selector.
Then, for each field,
we invoke book loader dot AD DS XPath and pass it the
corresponding Mark apps electors, which are the same ones we used recently.
After that, we invoke yield book loader dot load items.
This returns every scraped book.
After that, we need to modify islands dot pie and add some processors.
To do that, we first need to import map composed.
Then we define an ad link function that Wilken Katyn eight and
return the right URL for image and book pages.
We assign that processor to the Image URL Field,
which calls the ad link function, and we do the same for the book URL Fields.
What this does is to format the U.
R L for the image URL and book URL fields before
their values are assigned to them.
Let's also add this closing character and let's remove the
instance of books items as we no longer needed.
And this is being taken care of by the item loader.
No, let's run the spider again.
Notice how the results are being returned through the console.
We just implemented input processors and added an item loader to our spider.
This means we are making use of the built in functionality of the
framework rather than relying on custom code to get results.
Now let's modify the code so we cannot a pipeline.
The goal is to focus on books that have a price of £50 or less.
The ones with the price above that value will be considered expensive.
Let's add to take first processor and assign that to
our spider default output processor.
Let's add this line so we can see the results better on the console output.
When the spider runs, let's go to pipelines dot pie and,
at some logic, to check for book prices.
Let's import drop item,
which we will use later to remove books that are expensive.
We can remove the pound symbol and check if the price is above £50.
If so, the value expensive is assigned to the price fields.
Otherwise the price value is kept.
Then we define the checked as viable class,
which checks if the book price doesn't contain the word expensive.
And if so, the book details are printed To enable pipelines,
we need to go into the settings, not Python ill,
and add those pipeline classes as follows books.
Price has a value of 100 which means that it executes.
First Check Is Viable.
Has a value of 200 which means that it executes after
the spider has executed books price.
So books price sets the value as expensive, if applicable, and check is viable.
Determines if the book can be printed out to the console.
So let's run the spider and see the results.
Notice how this book was flagged as expensive and this other one is well in.
The next module will learn how to take this a step further and remove these
expensive books from the results and also use the file writing mechanism
built into scrapie to write the results to disk