In this demo,
we'll see how we can use Beautiful Soup to extract a list of links from a page.
This is a little more of a real world problem,
and you have sufficient experience with Beautiful Soup so
that we can work with a real website rather than just use an
HTML page on our local machine.
Here is the URL to the website that we'll scrape for links.
It's at mashable.com,
and it talks about how you can build your own static website.
So what is the structure of this web page?
Well,
if you're using the Chrome web browser you can right-click and choose
the Inspect option to bring up Chrome developer tools.
Chrome developer tools give you quick access to the HTML content of your page,
it allows you to debug JavaScript,
everything that you need to build and work with websites.
What you see here onscreen is the HTML structure of this Mashable site.
Even very simple websites tend to have complex HTML structures,
which is why it's useful to have something like Beautiful
Soup to help you parse the structure.
Let's get out of this tab,
and let's see how we can extract a list of links from this Mashable page.
Since we are working on a real website,
we'll use the requests library to make GET requests to
get the HTML contents off the site,
and then use Beautiful Soup to parse those contents.
Here is the URL that we are going to make a GET request to.
This is the Mashable page that we just looked at.
The Requests library makes it very easy for us to make a GET request
to this URL to get access to the HTML contents,
requests.get will have the HTML stored within response.text.
Let's parse this using Beautiful Soup,
and this will create a soup object that we can work with.
We've already seen how we can search for and filter specific
elements of interest from HTML content, and that's exactly what we'll use here.
We'll use the findAll function.
Observe that Beautiful Soup supports findAll using find_all as well.
So how do we identify links in our page?
We want to find all a tags or a HTML elements which have href attributes,
and we can specify that these href attributes match a
particular regular expression as well.
For example, here I've specified that I'm only looking for secure links,
links that start with HTTPS.
And here are all the HTTPS links extracted from the
Mashable site that we just saw.
Observe that this is a list of a tag elements.
All of these a tags have href links, and all href links start with HTTPS.
If you're interested only in the links leading out of this page,
and not in the a tags, you can run a little for loop,
find_all a tags with href links that start with HTTPS,
and print out only the href link.
Once we have the a tags,
we can simply look up the href attribute using square brackets for that a tag.
This gives us a cleaner,
more readable result containing only those links that lead
out of the Mashable page that we just saw.
If you're interested not just in the secure links,
but in all links that start with, say, HTTP,
you can specify your regular expression to match just http://.
This will give you the non-secured links,
the ones which are not encrypted using SSL,
and here is the list from that Mashable page.
If you want to extract all links, HTTP as well as HTTPS links,
all you have to do is edit your regular expression.
Look for all links that start with HTTP or HTTPS,
and you'll get the entire list.
Well, if you know something of regular expressions,
you'll know that the way we specified this regular
expression was arbitrarily complicated.
You can just look for all links that start with HTTP,
and that'll give you both HTTP, as well as HTTPS links,
as you see here.
With this particular regular expression,
we only extracted absolute links from our Mashable page.
We didn't extract any relative links, links that point within the Mashable site.
These are all links to external pages which is why they've
been specified using absolute HTTP links.
If you want to be able to extract all of the links that exist on this page,
you need to do a slightly different operation.
Observe that I've used soup.findAll here to find all a
elements which have the href attribute, href=True.
I've not looked for any patterns within the href attribute.
I only want the href attribute to be set,
and this will give me all links that lead out of this page.
Observe that these are relative links.
These are links to other pages within mashable.com static website generator.
That is the URL that we passed and can be specified using
relative links that start with a forward slash.
Right now this list has relative links,
as well as absolute links to other web pages.
In order to get the complete URL from relative links,
you might want to prefix it with the URL that we just scraped,
and that's exactly what we'll do here.
I invoke soup.findAll and find all a tags where hrefs are present,
that is href=True.
I'm doing a fairly simple check here,
I check to see whether the href starts with HTTP.
If it does not, I know it's a relative link,
so I prefix the current URL that I parsed.
The current URL is the static website generator page
that we viewed on mashable.com.
I also strip the leading forward slash because my
current URL has a trailing forward slash.
If the link starts with HTTP, I know it's an absolute link,
and I can use it as is.
Once I've updated the link string, I can print the link out to screen.
Now all of the links that we've extracted from this page and
printed out to screen are absolute links.
They either are part of the mashable.com static website
generator site or they're links to other pages.