Let's head back to this webnots site and continue exploring.
At the center of this page, we have a list,
which gives us the basics of a static website.
Select the contents of this list, right-click, and choose Inspect.
You can see that this is an ordered list with six list items.
Let's examine the HTML hierarchy that leads to this list.
You can see that at the top,
we have an article HTML element with the ID post-149.
Within this outer element, we have four nested divs.
And within the innermost nested div, we have this ordered list.
When you're working at extracting content from web pages,
you need to understand the structure of the web page so you can
specify what portion of the web page you want to select.
I'll switch back to the Scrapy shell here,
and this time,
I am going to use the CSS selector and select an HTML element by ID.
I use the hash here to specify the element with the ID post-149.
Within this outer element, I walk down four nested divs.
Within the last nested div, I access the ordered list,
that is the ol element,
then the individual list items specified by the li elements.
I then extract the text content of these list items,
and this gives me the six list items that we were
interested in in the webnots page.
If you want just the text for the first list item,
you can use the same CSS selector as before,
but invoke the extract_first function and this will
give you the text from the first item.
Let's head back to this webnots page here.
And I am going to scroll down here to the very bottom where I
have the number of likes for this page.
I am going to right-click on this and inspect this element.
When I inspect this element, the original value of 33 shows up at 34.
Well,
this is because I clicked on the element by mistake while I was right-clicking.
So while recording I accidentally added a Like to this page.
But this will allow me to demonstrate something
interesting about how this Scrapy shell works.
We'll come to that in a bit.
Observe that the span for this 34 element is contained
within an outer div with the Id like-149.
This outer div also has a class assigned called like-holder.
Keep both of these in mind as we switch over to the Scrapy shell and specify a
CSS selector for the text contained within the span element.
I select the element with ID like-149.
The hash specifies that it's an id selector.
This will select the div element with this ID.
Under that I look for the span element and extract the text.
What do you think this will return?
34 or 33?
Well, if you guessed 33, you are absolutely right.
Even though the text on the live page at this point in time is 34,
thanks to my clicking that like element,
why do we get 33?
This is because this Scrapy shell has downloaded the contents of the original
page as it existed when we started this Scrapy shell up.
All of the operations that we performed on this Scrapy
shell here work on the cached content.
If there are updates on the live site,
those will not be available within the Scrapy shell
until you re-start the shell.
This proves to you that the Scrapy shell works on the cached content.
The value of this text is still 33,
even though it has changed to 34 on the live site.
We'll use the CSS selector once again,
but this time we'll specify the class name for the outer div.
The dot prefix for the CSS selector indicates that it's a class name.
And this returns 33 as well since it gets the
information from the cached HTML content.
I am going to head back to the webnots page,
and somewhere here in the middle of this page,
there is this content which says Viewing Static Site in Chrome Browser.
I am going to right-click and hit Inspect,
so that I can see the HTML element that corresponds to this content.
This element here happens to be the title of the image displayed just above it.
Observe that this is a paragraph element with the ID
caption-attachment-2247 and the class wp-caption-text.
I'll now switch back to our Scrapy shell and use the CSS selector to
select this element by ID #caption-attachment-2247::text extracts
the text corresponding to this element.
Let's go back and inspect our webnots page once again.
Here is another image, and we have a title just below that image.
I am going to inspect this title,
and you can see that it has the same class name applied.
The ID for this image title is different, but it has the class wp-caption-text.
I'll switch back to this Scrapy shell once again and use a CSS selector
to select all elements with the class wp-caption-text,
and I invoke the getall function.
And this selects both of the image titles that we examined earlier
and extracts the text information for those titles.