In this demo,
we'll see how you can parse and extract the content from a
real web page using regular expressions.
Here we are on a brand-new Jupyter notebook.
We'll use the RE module in Python to work with regular expressions,
and we'll use requests to download the HTML content of a page.
Here is an example page with randomly generated addresses.
This is the URL.
I'm going to open up this web page within my browser
to show you what it looks like.
Call webbrowser.open,
and here is a page full of sample addresses for our parsing enjoyment.
You can see that every address is associated with a name and
is present in the form of a bulleted list.
Before you parse and extract content from a page,
you'll need to have an idea of what the HTML structure of the page looks like.
Right-click and hit Inspect,
and this will bring up Chrome browser tools that will allow you
to navigate and explore the HTML content.
The HTML structure here is interactive.
You can select, click on, and expand nodes.
Here I've expanded the header node, and within that,
I can see the title of the page.
This page has two headers under the body node,
h1 and h2, and there are a number of bulleted list items.
You can hover over and select specific HTML
elements within Chrome browser tools,
and the corresponding element on that page will be selected.
Here I just select and expand the individual li elements,
which correspond to the list items and see the structure within each list item.
Each of these li items contain the name an individual,
followed by a break, followed by a P.O.
Box number, followed by another break, then an address,
followed by a break, and, finally, the phone number.
Not all web pages in the real world will be so well-structured,
but this is a great way to get started using regular expressions with web pages.
Let's switch back to our Jupyter notebook.
The first thing we need to do is to download the content of the page,
which we'll do by invoking the GET request using the request API.
The HTML content of the page will be present in the response.text property.
Extract this content and print it out to screen.
This HTML content is now available within your program as a string.
And with this string, you can use regular expressions to perform pattern matches.
Here is a regular expression that allows you to extract
the title element within this HTML page.
I use re.findall,
and the regular expression pattern that I've
specified includes the title element, a dot,
and an asterisk.
The dot represents that you want to match any character except a new line,
and the asterisk represents that you want to match zero
or more repetitions of that character.
This regular expression matched the title element in our HTML,
and we have this extracted as a string within a list.
What if you don't want the title tags,
and you're interested only in the text content within the title element?
Well, you simply need to change your regular expression a little bit.
The entire pattern here represents what you want to match within your
string; the brackets that you specify indicates what token you want to
extract after the pattern match has been found.
The matcher gives us all of the content contained within the title element.
I'll now use a similar regular expression,
this time to match and extract all of the contents
within the h1 element in our page.
We've extracted just the content,
and here is the string which makes up header 1 on the web page.
Our page contains more than one header.
What if you're interested in the content of all headers on this page?
Here is a regular expression that can help you do that.
H.
indicates that you want a pattern that starts with the angular brackets and h.
The dot indicates it can be any character after the h,
whether it's 1, 2, 3, anything other than a new line.
This matches the content of all headers within our web page.
There are two headers.
Both of the strings are now available.
As you can see here,
you can combine the different building blocks available within
regular expressions to set up your pattern matches.
Here I want to match list item elements as specified by the li tags.
I want to match elements starting with li, which end with a break.
Once a pattern has been matched,
I extract the contents of the element which are made up of alphabets,
A to Z in uppercase and lowercase, separated by a space.
The plus special character indicates that whatever you match within
square brackets should be repeated one or more times.
This particular pattern matches all the first names and last names of
individuals present within a randomly generated page.
Let's say you want to perform the same first name/last name match,
but you want to extract all of the first names
separately and the last names separately.
You can control what and how you want to extract within
your pattern match using the parentheses.
I have specified parentheses around the first name,
as well as the last name,
and each of these have been extracted individually here in my result.
Every element within this list is a tuple.
The first name and last name are fields within each tuple.
Here is a pattern that matches all bullet items where the
first character after the bullet is a C.
The \w that follows the C indicates that we're
looking for alphanumeric characters.
And the plus sign indicates that we're looking for one or more
repetitions of the alphanumeric character.
And the parentheses indicate that you want to
extract the content of the list item.
This gives us all first names, starting with the uppercase C.
Here is a slight tweak to this previous regular expression.
Give me all first names that start with the uppercase C,
but along with the first names, I also want the last names of these individuals.
That's why we have the space and the square brackets and the plus indicating
that we want to match zero or more alphabets in the last name.
Here are all the individuals where their first name starts with an uppercase C,
along with their last names.
Let's see a pattern that you can use to extract phone numbers from this page.
Phone numbers are made up of digits from 0 through 9.
Here I specified that I want to do digit matches.
I want the first set of digits to have exactly three digits,
then three digits, and then four.
After the first three digits, we have a space.
Between the second and third set of digits, we have a dash.
When you want an exact match for the number of characters,
you specify that number within curly braces in regular expressions.
And here are all of the phone numbers scraped and extracted from that web page.
One last bit of practice with regular expressions
before we move on to Beautiful Soup.
Extract the addresses from this web page.
Addresses have alphabet characters, so we have alphabet pattern matches for city,
followed by a space, then the state, and,
finally, we have an exact digit match for the five-digit zip code.
And here are all of the addresses successfully extracted from our site.