In this section, we are going to code out what we discussed before.
As usual,
we start off by extracting all the relevant packages and adding
the required options for the Chrome driver.
Nothing you haven't seen before.
Next, we navigate to the Premier League's website.
We then use the find_element_by_link_text to click on the Players section.
Now, there's a small waiting period before the search bar appears,
which we take care of by using this explicit wait.
We then find the search box element, type in Wayne Rooney, and hit Return.
There is another small waiting period.
In the interest of showing all the sites, we use implicit wait at this juncture.
The difference between implicit and explicit waits is that implicit waits
tell the web driver to wait for a certain amount of time,
no matter what,
and explicit waits tells the web driver to wait until
a certain condition is satisfied.
We now click on the image of the player, which will take us to his profile page.
Let's run this code block and see if everything happens like we expect it to.
(Loading) Yes it does.
The page source information is then assigned to the
page_source_overview variable.
It is now time for Beautiful Soup to enter the picture.
We import BeautifulSoup and then load the page source
information while parsing it with the LXML parser.
Now, we need to locate the titles.
If we go back to the site, inspect the news headline elements,
we can see that it is located in the span tag under the class title.
So we use the find_all function in Beautiful Soup to find everything
that is under the span tag and under the title class.
And this is where it outputs.
You can see we are nearly there,
and all we need to do now is to get the text from these titles.
That is exactly what this piece of code does.
And as you can see, we get the latest headlines about Wayne Rooney.
Next, we need to locate the stats button.
On inspection, we can see it is under the href attribute.
We find element by the XPath method and we click on it.
And as you can see in the browser, we are now in the stats page.
We get the page source information and load it into Beautiful Soup.
If we inspect any of the elements we need to extract,
we can see that they're all part of the allStatContainer class.
Let's try and get them all with the find_all function.
And there you go.
Here we can see that the stat titles are stored in the data-stat
attributes and the stat numbers themselves are stored as text.
To extract both of them out, we use the code shown here.
We've come of the end of the demo for this module,
but the thing to notice,
that whatever we've done from here can be extended to a much more higher level.
All the players have such pages in this site,
from where you'll be able to extract their stats.
You could then maybe put these stats, along with the headlines,
into a Pandas DataFrame and use sentiment analysis and other
machine learning techniques to probably predict some of these
stats for an upcoming match.
There is very little Wayne Rooney specific code here,
and whatever there is, can be easily modified to make it clear agnostic.
That was just an idea off the top of my head.
But the point is, web scraping like this can lead to so much more,
and I encourage you to develop some ideas and try them out.
The comments section on this course is open all the time,
to interact with me and amongst yourselves regarding these
ideas and any other queries you may have.