In this demo,
we'll see how we can use Scrapy to select elements on
a web page using XPath selectors.
We start here on our terminal window.
I am going to re-start my Scrapy shell with an
updated version of the webnots page.
This will cause Scrapy to download the contents once
again and cache it on our local machine.
Re-starting the Scrapy shell in this manner will retrieve the
HTML contents afresh from this website.
Here we are on the webnots page, I have Chrome browser tools open,
and I have the title tag for this page selected.
With this title element selected, I am going to right-click,
choose the Copy option, and copy the XPath to this title element.
I am now going to use this XPath information to select this
element using response.xpath within my Scrapy shell.
XPath allows you to walk down the HTML hierarchy of your web page.
Observe that this XPath starts with the forward slash.
A forward slash indicates an absolute path to the title
text starting at the root of the document.
The absolute XPath to our title element is /html/head/title,
and then we extract the text content.
We select the text of the title element and invoke
the get function on this selector,
and this will return the title text for this document.
On the webnots page, here is the first paragraph of text within this HTML page.
Right-click, select inspect, and you'll see that this is a p element,
this is the first paragraph.
We'll now use XPath to extract the text of this first paragraph.
Response.xpath is the function invocation,
and we specify the XPath using the double forward slash.
The double forward slash indicates that this is not an absolute path.
This selects all of the descendants which match this path
at or below the root of the document.
Because we've used the get function,
this will select only the first paragraph and return the
text associated with the first paragraph.
If you want to access the text in all of the
paragraph elements within this document,
use the //p/text and invoke the extract function.
This will return the text contained within all of elements of this page.
Here is the text in the first paragraph,
and if you scroll down below,
here is the text from the last two paragraph elements on this page.
Another way to access the text in the first element is
to use the extract_first function.
And this gives us the first paragraph text.
Let's go back to our webnots page here.
Right here in the middle of the page, we have an image.
I am going to right-click and inspect this image element.
Here is the image element that is selected within Chrome browser tools.
Let's see how we can use XPath to extract the source URL for this image element.
Use the response.xpath function as before.
The double forward slash indicates that we'll match all
image elements at and under the root.
And to extract the value in a particular attribute using XPath,
you use the at sign.
This will give you the values for all of the source
attributes for all of the images on this page.
We're repeating the same selections that we performed using CSS before,
this time we do it using XPath.
Here are the basics of a static website.
This is an ordered list as we've seen before.
Remember, there is an easy way to get the XPath for any element within the page.
I've selected the ol element, I right-click,
choose the Copy option, and select Copy XPath.
And this is the XPath that I choose here in the response.xpath function.
The double forward slash and the asterisk means to select
any node with these characteristics.
The asterisk serves as a wild card selector in XPath.
Under any node, look for a node with the ID post-149.
If you remember from our earlier demo,
this is the ID of a parent element for our ordered list.
The XPath that we copied over indicates which particular div
we want to access under the hierarchy.
The number specified in square brackets is the index position of that div.
Our XPath selector gives us the text elements of all of
the list items within the ordered list.
Back to our webnots page where I scroll down to the very bottom where we have
the number of likes or hearts for this particular page.
If you remember, I'd clicked on it by mistake earlier, and it now shows 34.
My like has been added here as well.
I right-click and choose Inspect,
and this will select the HTML element that corresponds to this content.
I am going to right-click and copy the XPath for this
element over so that I can use it to select this
particular element using response.xpath.
We look for an element with the ID like-149.
Under that we find a span element and extract the text of that span.
And this time around we get 34.
This is because we re-started a Scrapy shell.
We Scraped and downloaded the contents of the live site once again,
so our cached version matches the live version.
Back to the webnots page for the static website.
And here is the title for this image.
Let's inspect this, and you see that it has the class wp-caption-text applied.
Let's scroll down below and take a look at the title for the second image here.
If you hit Inspect, you'll see that it has the same class, wp-caption-text.
Let's use this information in our XPath to extract the
text content of these image titles.
We use the asterisk sign to indicate that this is
an attribute of an HTML element, class=wp-caption-text.
This selects both image titles and extracts the text content of both.
Scrapy allows you to select elements in more powerful ways as well.
Here we want to extract all of the text within our web
page that contains the word static.
We use XPath with wildcards to match any node that contains the text static.
Executing this command will return selectors for all of the
text that contains the word static within it.
If you don't want the selectors,
but instead you want to access the actual text content,
any text that contains the word static,
simply invoke the extract function on your selector list.
And here is all of the text that contains the word static.