[Autogenerated] and the last generic spider type available
within scrapie is the site mint spider.
Let's have a look at some of his properties.
The site map URL rows Property indicates a list of URL is pointing
to the site map whose URL is we want to crawl.
Then we have the site Map Rules Property,
which is a list of topples, a regular expression and a callback.
The regular expression is used to match.
URL is extracted from site maps,
and it could be either a string or a compiled rags object.
The callback is used for processing the U.
R L that matched the regular expression.
The callback could be a string indicating the name of the spider method.
Next, we have the site map.
Follow property,
which is a list of site map regular expressions that should be followed.
The site map Alternate the links property specifies,
if alternate, the links for when URL should be followed.
The site map filter property is a filter method that could be overridden
to select site map entries based on their attributes.
Let's have a look at the following example to understand this better as usual,
let's import the scrapie module and that's also
import the site map Spider module.
Then we define the site map URL property.
This will be pointing to the URL that contains the website
site map file for the site map rules.
We define the product URL that is going to be scraped by the
parts product method and the product category.
URL Path is going to be scraped by the parts prod category method,
and here we define the parts product method,
which will be responsible for scraping each product and the
parts broad category method is going to be responsible for
scraping each product category.