In this demo,
we'll see how we can use CSS selectors and Scrapy to
extract content from our web page.
The web page that we'll parse using Scrapy is this
webnots.com /basics-of-a-static-website page.
We can download and load the HTML contents of this page
into a Scrapy shell by invoking the scrapy shell command
and specifying the webnots URL.
Under the hood,
Scrapy will use a default spider to crawl the contents of this web page and
load these contents into the Scrapy shell ready for us to use.
Let's head back to the site that we just downloaded.
This is the webnots.com site.
Right-click and choose Inspect.
We'll inspect this page using Chrome browser tools and
see what content we want to extract.
For any web page, the Browser tab displays the title of that web page,
the text specified within the title HTML element.
Chrome browser tools, which has opened up on the right hand side of my browser,
allows me to navigate and explore the contents of this website.
I can expand the header element, and within the header element,
I can select the title tag.
Selecting a particular HTML element within developer tools will
highlight that element in your main browser window.
If you want to know the path to a particular element,
you can always right-click on an element within browser tools,
go to Copy and say Copy selector.
This will select the CSS path to this particular title element.
Observe that the same menu offers you options to copy
the XPath to this particular element,
as well as the JavaScript path to this element as well.
When you want to extract the contents of a specific HTML element,
this is an easy way for you to get the CSS or the XPath to that element.
Let's switch back to our Scrapy shell terminal window here.
If you want to access the contents of the title tag,
you can use response.css,
and this will give you a selector object for the title HTML element.
The CSS selector allows you to specify elements by tags.
And here is a selector to the title of our webnots speech.
Scrapy selectors select certain portions of the HTML page
using either CSS or XPath selectors.
If you want access to the underlying HTML elements specified by the selector,
simply call the get function on the selector object.
The get function on a selector extracts the underlying
HTML element in the form of a string.
If you want to extract the text content within an HTML element,
specify the CSS selector for that element followed by a double colon text.
This will extract only the text content of that element.
Observe that the title HTML tags are not present within this text.
Let's go back to the website that we are interested in,
and let's see how we can extract specific content from this website.
This is the content that I want to extract.
I've selected it, I'm going to right-click and hit Inspect.
When you select a portion of the web page and hit Inspect,
Inspect will take you directly to that HTML element,
which you have highlighted within the main browser window.
Here, Inspect takes me directly to that paragraph tag,
which contains the text that I'm interested in.
It so happens that this is the first paragraph element within this page.
Let's switch back to our Scrapy shell terminal window.
And let's extract the text of this first paragraph,
response.css ('p::text') .get will give us the text in the first p element.
When you use the get function on a selector,
it returns the contents of just the first element that was selected.
If you switch back to the webnots page,
you'll see that there are multiple paragraph elements.
For example,
this copyright information at the bottom is also a paragraph element.
I am now going to scroll to the very top and click on this search icon here.
This will bring up another bit of text.
And if I inspect this text,
you'll find that this text is also within a paragraph element.
Clearly, there are many more paragraph elements within this HTML page.
If you want to be able to access the content of all of these paragraph elements,
you need to invoke the getall function on your Scrapy selector.
Response.css ('p::text') selects all of the text within all paragraph elements.
Getall will return all of this text.
You can see that getall returns a list of strings here.
The first string here is the text from the first paragraph element,
and the last two strings are the text from the last two
paragraph elements on that HTML page.
Scrapy selectors also support the extract function.
The extract function is the equivalent of getall that we've seen earlier.
This CSS selector will match all of the text within paragraph elements,
and extract will extract all of this text information in the form of a list.
If you want to extract just the first text content that matches the selector,
you can use the extract_first function,
and this will extract the text from the first paragraph element that matched.
Let's go back to our webnots site once again.
And right here in the middle of the page is an image.
I am going to right-click and choose the Inspect option.
And this will select the HTML element that displays this image.
You can view the contents of this image element and see the different
attributes that it has including the source of the image.
Let's go back to our Scrapy shell here,
and let's extract the source attribute of all of the
images present within our website.
I've used the CSS selector to specify the image tag
:: followed by attr and then src.
This will extract the source links of all of the
images present on this web page.
We used our browser developer tools to inspect just a single image,
but here we have the source links of all of the images present here,
including the logo that you see here at the very top.
Let's right-click and inspect this logo.
And you'll see that it corresponds to an image tag as well,
which was extracted by our image selector.
We've seen that the extract function returns a list of all of the HTML content
that was selected using our CSS or even our XPath selectors.
You can access the individual elements of this list
as well using square brackets.
The image element at index 1 happens to be the
header logo that we just inspected.
Here is another technique you can use within Scrapy to extract
a particular attribute for an HTML element.
Here we've selected the image element by tag name,
the .attrib property serves as a dictionary,
and you can look up attributes by name.
Here is the source URL for the first image element selected on our web page.