[Autogenerated] We've covered quite a bit of ground and
talked a lot about different scrapie spiders.
So now it's time to have a demo and put what we've learned
into practice in this demo will be focusing on how to
implement a spider using the scrape it out.
Spider Class sounds really exciting,
so let's start right in with the S code open.
Let's modify the items dot pai file here we will define
the attributes that will be extracted.
We are interested in getting the quote itself.
The author and tags associated with each quote each are
defined as scrapie field objects.
So let's modify the quote Spider DOM Python file,
which will contain all the scraping logic.
Let's import and references all the libraries we need.
Then we need to import the items properties we just
declared within the items that pi file.
Now let's add some fields are spider class is going to
need txt is going to be the file extension that will be
used to save the scraped results.
All will be used to indicate later if we want to have the spider scrape,
the whole site or just part of it F and indicates the file that
will be used to save the scraped results,
which also coincides with the URL of the site we will scrape.
De en is the sites domain name.
First Page represents the URL of the Sites First page, which will be scraped.
This is how this page looks.
If we open the browser, this will be the Spiders entry point or start page.
Scope refers to the other pages from the same site that will be
scraped when the value of all is set to falsy.
Then we need to assign allowed domains and start you are
else to the end to avoid repeating strings.
Let's define a method that will delete the scrape results file
from disk so that each time the spider runs,
the results can be fresh.
So if the file already exist on this, we simply deleted this way.
The Spider Concrete's a new one when it runs,
then let's define the start request method.
This will call the Dell File method and loop through all the
pages defined within the scraping scope.
Then, for each page, a scraping request is triggered.
The parts method will be used to extract the data from each page to
extract the data we need to create an instance of quotes.
Item as a place holder of the data we're going to extract,
Let's select what we're interested in and inspect the element
to understand what details we need to get.
The main element is this TIFF with a quote class to do this.
For all the quotes on the page,
we need to loop through all the DIV quotes returned by the scrapie response.
So going back to are code we need to get the value of the Div Quote
from the scraping response object using a CSS elector.
The quote itself is contained within the span.
Do text element and we need to access its text property
and call the extract first method.
Let's also clean the text result and remove any unnecessary
characters using a Unicode range of valid characters.
And as it is a quote, let's add the quote characters.
Now let's inspect the author and how we can get it.
We need to look for the small dot author selector and get its
text property and then call extract first.
No, we need to get the tags.
We can do this by checking the div do ETags selector and
then within get each a tag for each attack.
We need to get the text property to get all the ties.
We use the extract method as we will get all the tags.
For that quote, we need to split each word and then join them as one string.
Then the values of the quote instantiate are written to a file by
calling right txt right y-axis calls code X dot Open passing the file
name and extension with the append option and by specifying that the
content will support utf eight encoding.
Then each of the values of quote,
author and tags are written to disk for each quote.
All right, we now have a spider.
Let's change the all value to falls and correct the small typo.
So let's run the scrapie crawl,
quote Spider Command and see how the spider executes.
And then let's check the scraped results.
Look what we have.
Isn't that cool?
Let's compare these results to what is on the actual website.
Let's browse to Page one.
See how the quote on the web page matched the quote that the spider scraped.
Let's check Page two.
Notice how the quotes on page two are actually
found at the bottom of our results.
You would have thought that they would just come
after the results from Page one.
This is because scrapie executed scraping of the pages a
synchronously and also concurrently.
So the scraping of Page three might finish first than the scraping of Page two,
even though they started more or less at the same time.
Which is what actually happened.
If we go to Page three we can,
see that the scrape results are before the ones of Page two.
And if we check the ones from Page four,
there's somewhere in the middle between the ones from Page three and Page two.
So this is something really cool about scrapie and how it is able to
scrape different pages at the same time as we have seen.
We've just scraped a limited subset of the whole site,
so actually four pages out of the 10 available.
This is because we said the old value to falsy and also have
defined a limited scope of pages to crawl.
To be able to scrape all pages, we need to be able to follow the next button,
so let's get its CSS selector we can.
See, it's defined by the next selector,
and from there we need to get the href attributes of its corresponding a tag.
We add this logic to parse so it gets called if the all value is set to truthy,
which we can get by calling response do CSS.
So now we have a way to get the next link for each page that follows.
If there is a next page,
we execute the scrapie that request method to crawl the page that follows.
So let's try this out.
Let's set the value of all too true again.
Let's manually check how many pages we have.
As you can see, we have 10 pages.
So let's run the scrapie crawl command again and see how
all the 10 pages have been crawled.
Let's inspect the results.
Awesome.
All the quotes are there.
We can change the scope to two pages only and give it another try.
Cool.
So now we only have two pages crawled and scraped