In this demo,
we'll see how we can use filters in Beautiful Soup to extract
all of the links present in an HTML page.
Instead of using a simple HTML page from our local machine,
we'll use the request library in order to make a
get-request to download contents from the web.
We'll download contents from the pluralsight.com site.
The request.get function will make a GET request to get
the contents from the Pluralsight URL,
and we'll extract the text contents and construct a BeautifulSoup object.
The soup object contains this HTML,
which we'll now use to extract links from the Pluralsight page.
If you explore this page,
you'll find that there are different kinds of links available.
There are relative links within the Pluralsight site,
and there are absolute links which point to other web pages as well.
Let's specify a simple filter which will extract all of the links
from the anchor tags present on this HTML page.
We'll extract just those links that begin with HTTPS, that is, the secure links.
Soup.find_all will return all of the html tags that
contain reference to secure links.
You can scroll down and view the results of this find_all operation,
and you can see that all of the href links point to HTTPS sites.
When you're extracting the links present within an HTML site,
you're not interested in the html tags that contain the link.
What you're interested in is the link itself.
You can use square brackets to access all of the href
properties of the tags that you've accessed.
This will extract all of the secure links present on this pluralsight.com site.
You can see that many of these links lead to other pages on Pluralsight,
and here at the bottom there are links leading to other pages as well,
Twitter, LinkedIn, Facebook, etc.
What if you want to extract non-secure links as well,
those that begin with HTTP and are not secure using SSL?
You can simply specify http:// in your regular expression,
and this will give you the non-secure links.
There's exactly one here.
Once you know how to apply regular expressions,
you can tweak it to access both the secure,
as well as non-secure links.
The pipe operator performs an or operation.
It'll extract HTTP links, as well as HTTPS links.
Here are all of the secure links on the main Pluralsight page.
There is exactly one link which begins just with HTTP.
That is the help page.
And instead of using the or operator,
you can just keep your regular expression simple.
You want all HTTP links, all links that start with HTTP.
So far we've accessed and extracted all of the
absolute links present on this page.
You might want access to the relative links as well.
I'm going to filter all of the anchor tags which have
some value for the href property, href=True.
Href properties can point to relative links, as well as absolute links.
You can see that many of the links here are relative
links within the same website.
There are other links at the bottom which are absolute links to other websites.
If you want to ensure that all of your links are
absolute links which can be used directly,
you can perform a very simple prefix addition.
Here is a for loop that finds all a tags with href=True,
and if the link does not start with HTTP,
prefix the Pluralsight URL to that link.
This prefix will convert relative links to absolute links.
If the href attribute starts with HTTP, we know that it's an absolute link,
and we can use it as is.
Let's print out all of the links here.
We now have just absolute links.
All of the relative links have been prepended with the Pluralsight URL prefix.
In addition to anchor tags, you can have HTML elements with the link tag as well,
which usually point to CSS styles and other HTML pages.
You can simply tweak your filter a little bit to search for both a tags,
as well as link tags within your HTML.
We extract the values of the href attributes in the same manner.
Now you can have image tags which point to links as well.
Let's find all of the image tags and access the source attribute.
This will be links to the images on the pluralsight.com page.
They can be relative links as you can see here.
They can also be absolute links.
And you already know how you can convert the relative links to
absolute links by prefixing the Pluralsight URL.