Selenium was originally released in 2004,
and it's a very powerful tool for automating web browsers.
Let's take a look.
You may be wondering which is best, requests‑html or Selenium?
Well, both are great, but there are some differences.
Requests‑html is based on pyppeteer, which uses Google Chrome.
Selenium, on the other hand, supports all major browsers,
Firefox, Safari, Opera, Chrome, or Edge,
even Internet Explorer with some extra configuration.
Both options have powerful XPath and CSS selectors, so no difference there.
But to me, requests‑html is Python only and a bit easier to learn,
though still quite powerful.
Selenium supports several languages, and its API is more complicated,
but it is very powerful.
You can't go wrong with either option.
Selenium has wonderful documentation, including code examples in many languages,
and, yes, Python is one of the options.
Visit selenium.dev/documentation.
And Selenium is easy to install to.
Pipenv install selenium will do the trick.
The rest of the project setup is the same that we used for requests‑html.
You can even reuse that environment, except there's one other thing to install,
a web driver.
The web driver is specific to the browser you want to automate.
Chrome has a web driver,
but I thought you might like to see Firefox for a change.
Its web driver is named geckodriver,
as Gecko is the name of the Firefox browser engine.
You have to install a web driver first before Selenium will work.
On a Mac, it's easy, brew install geckodriver.
That's all it takes.
For a PC, you can download the web driver, unzip it, and add it to your path.
Let's take a look at the code.
We've got some imports from Selenium and still have to specify the start_url.
You may not be that familiar with Python's with‑as syntax,
so let me give you an overview.
Of course, you may have already seen this used to open and write files.
Here's how it works.
An expression follows the with keyword.
In this case, it starts the webdriver.firefox object,
and from then on, references it as driver.
After the colon, we're in the with block and can put the driver to work.
We imported WebDriverWait,
and I'll tell you more about its capabilities in a minute.
For now,
just know this code is essentially the same code as we used for requests‑html.
Driver.get goes and downloads the initial HTML.
Then we sleep for 10 seconds while the browser renders all the JavaScript.
We used 5 seconds before, but Selenium and Firefox seem to take somewhat longer.
Then we can use find_by_css_selector to get the Teslas and
look inside a Tesla to find a specific model.
You've seen this so many times, it's probably boring now.
But wait, I promised to tell you about wait.
See,
you can wait for an element to render. Many times, JavaScript makes
substantial changes to the initial HTML page.
In these cases,
you can use Selenium's wait.until functionality
instead of sleeping for a fixed delay.
Inside the until statement,
you'd use XPath or CSS selectors to check for an element
or check for some other condition.
_____ cars did not really change the structure of
the page, only the data changed, so I had to fall back to the sleep delay.
When you can,
it's best to use the condition as scraping will be faster and more reliable.
Now you're introduced to advanced web scraping, you
understand how JavaScript can be a problem,
and I've shown you two libraries that can overcome the problem.
Requests‑html is a souped‑up version of requests.
You saw a code example and learned how to specify a sleep
delay to give the JavaScript time to render.
Then we explored Selenium, you saw another code example, learned about
installing a web driver, and learned how you might use wait.until to check
if the page is rendered. You're dangerous now.