I did unleash my imagination.
Since I like cars, why not go scraping for a car?
I'll admit it, ever since Elon Musk launched his Roadster into space,
I've been thinking about driving a Tesla. Elon Musk and Tesla fascinate me.
If only I could get a Tesla at a good price.
The one in space is free, I suppose,
but the galactic delivery costs are way too high.
Why not try to scrape up a Tesla? There are car
buying websites right here on Earth.
What if we could automate a way to find the right car?
TrueCar is a good example.
There's pictures, prices, plus filter and search tools.
Nice.
Only it's really too much information to read.
And for some reason, those filters never seem to do what I really want.
Worse, I could waste a lot of time just looking around.
Now I'm one of those guys that likes a really good deal.
Websites like TrueCar indicate cars that are a great deal, but is
it really a great deal or just marketing hype?
By scraping multiple car websites, doing some analysis,
even machine learning, I can discover the very best deals.
I like a great deal so much, I don't really care where the car is located.
I love an adventure, too,
so I could fly to the car and drive it home as long as the
airfare expense doesn't ruin my great deal.
Fortunately, I'm a data scientist and Python programmer.
I know what to do.
First, scrape the car buying websites to find all the Teslas,
then evaluate the prices to find truly great deals, check the
airfares and adjust all the deals for airfare expenses, and send
myself an email digest of the top deals each day.
Building the full app is a bit too much for an overview
course like the one you're watching.
Still, I'll show you the basics so you can get started on your own ideas.
Fortunately, Python offers a rich set of libraries and tools for web scraping.
I'll show you how to use the Requests library to download web pages.
Then, you'll learn how to use Beautiful Soup to extract
just the data you want from the page.
For many projects, that's really all you'll need.
We'll also explore Scrapy,
which is a complete application framework for crawling
websites, web scraping, and more.
Scrapy provides all the capabilities of Requests and Beautiful Soup,
but is faster and even more powerful.
Finally,
there's Selenium. It lets you build an automated robot web
browser you can control with Python.
If a browser can do it, you can automate it with Selenium.
Each library has its place, and I'll show you all of them.
In summary, why scrape the web?
Because the web is full of profitable data that can lead to new
business models, use an API when you can, but scraping may be
the only way to unlock trapped data.
You can use scraped data to combine data in unique ways, like to buy a car.
Really, scraping is just automated web browsing with the benefit
that data is cleaned and stored for further processingAnd that's the
next module on the web scraping process.
Keep going.