In this demo,
we're going to create an account and website using Python and Selenium.
We're going to be using a dummy site called automationpractice.com.
And the registration page that is this particular page contains the
data through which we're going to go through.
We're not going to be scraping anything here,
so why did I choose this particular example?
I chose this example because this page contains all of the things we just
mentioned at the start of this section of the module,
radio buttons, checkboxes, and drop‑down menus.
And what we're trying to do at this point in the course is
learning how to deal with them when they come up during your
standard scraping procedures.
This might not be the type of example you had in mind when
starting this section of the module,
but the concepts you learn here can be applied across almost any
website We'll be doing this using Selenium.
Now what is Selenium?
I'm guessing some of you watching this course might already be familiar with it.
If you're not, there's absolutely no need to worry.
You'll get the hang of it in the actual demo.
But before that, let's look at it a bit more closely.
Selenium is a set of tools and libraries that automate web browser actions.
In a nutshell, that is all Selenium does.
It automates web browsers, and what you do with it is entirely up to you.
Selenium uses a web driver package that can take control of the browser and
emulate user‑oriented actions to trigger desired events,
events such as clicking buttons or links and even scrolling.
It is free and open source and works across Windows, macOS, and Linux.
It also supports plenty of language bindings such as Python,
Java, Ruby, Scala, C#,
so you can write scripts in whatever language you're comfortable with.
Let's now get back to the actual demo.
As usual, the first order of business is importing all the required libraries.
Then, we initialize the Chrome web driver.
We do that by specifying the location of the downloaded file on our machine.
You can pretty much ignore what is written over here.
This is just so that the web page opens up in full screen.
And then, we use driver.get to navigate to the required page.
When we press Shift+Enter,
you can see a full page Chrome window opens up displaying the page in question.
Now this is different to what I showed you before.
To create an account on this site,
we first need to enter the email address we're going to use.
Now this is a WebDriverWait.
Did you notice the wait time when the window opened up and the page loaded?
This code takes care of that.
The field in which we need to enter our email
address is not immediately accessible,
and this bit of code adds a wait time so that we proceed
further only once it becomes accessible.
How did I find the element for the WebDriverWait though?
It'll be clear within the next minute.
What you need to do next is to right‑click on the
email address field and click Inspect.
You can then see this section where they refer to this element by ID or by name.
You can then use either of that to locate the element.
Here, we use the find_element_by_name function.
This code here clears the field if there's anything already
written on it and then types in the email address that we
need to register for the site.
Then, we'll need to click the button labelled Create an account.
So like before, we inspect it and see that the ID of the element is SubmitCreate.
We use another function called find_element_by_xpath,
which is similar to what we did earlier in the module.
We use the ID of the button to locate it, and with this code, we click on it.
By the way,
if you'd like to know more about the different functions in
Selenium and how to scrape dynamic websites using it,
you can always go through my course Scraping Dynamic
Web Pages with Python and Selenium.
When we run that,
we can see that we've gone past the first page and
onto the page I showed you earlier.
This is where our main work takes place.
First, we deal with the radio button.
Let's right‑click on both Mr.
and Mrs.
and click Inspect.
To locate this button,
we need to use find_element_by_id because if you look closely,
you can see both the buttons have the same name.
That's what we do here, and then we use this code here to click on Mr.
We inspect each of these fields, use find_elements_by_id to locate them and
use the send_keys function to input what is required,
which is what we've done here.
Once you run this code block here,
you can go back to the browser and see that they're all filled in.
Now we deal with date of birth, which is a drop‑down menu with three fields,
day, month, and year.
If you go back up top, you can see we've imported something called Select.
We'll use this to deal with the drop‑down menu.
Let's inspect this field, and we see that the id is days.
We'll use the Select class in combination with the
find_element_by_id. And then, we'll use the select_by_value
function to input some valid value, let's say 10. We'll do the same
thing with the second field whose ID is months.
Here, we use the function select_by_visible_text.
There's a slight difference between the two.
Select_by_visible_text makes use of selection of drop‑down items based on the
text present in the drop‑down. This has to be the exact text. Select_by_value
will select based on the value attribute of the select tag. This value is
different for different visible text.
So here, we need to find the text present in the drop‑down. Let's say we want
the month to be May. Go here, copy this text, and paste it here. And the
exact same procedure is done with the years field.
Let's run that and go check out browser. And yes, the Date of Birth
field looks exactly like we want it to. And finally, we'll deal with
checkbox. Let's inspect the sign up for our newsletter checkbox. We
can see that the ID is newsletter.
We used this code to mark that.
There you go.
We've dealt with drop‑down menus,
checkboxes, and radio buttons. The code to fill up the
remaining parts of the page is right here, but I implore
you to fill it up without looking.
There are a couple of drop‑down lists.
Everything else looks straightforward. After registering, in case you do
want to scrape something from the site, the method to do so is to
transfer the page source from the web driver to Beautiful Soup and then
proceed normally from there. Like I mentioned before, this site doesn't
really require us to be registered in order to access its data, but I
can think of several sites that do, and this whole exercise was geared
towards teaching you how to navigate button input values on such sites.