I'm Clarke Bishop, a big data engineer.
And in this module, we'll be exploring the question, why scrape the web?
We'll be looking at how web scraping is useful,
some web scraping examples, and scraping for a car, yeah. Let's get started.
The essential question is, why scrape the web? Well,
there's profit in web scraping.
I'm going to show you a web scraping business model as an example. Really,
when you think about it,
every web page is data, only the data is locked up in all those web
pages. Google has identified well over 100 trillion web pages and 4
million new blog posts are added every day.
That's a lot of data.
If there's an API, that's usually the best choice.
If not, web scraping may be the only way to unleash that trapped data.
There's a lot of potential. With some imagination and web
scraping skills, it's all at your fingertips.
Even better,
companies are looking to hire developers with web scraping knowledge.
Have you ever heard of hiQ Labs? They're a data science
company informed by public data sources.
It says so right on their website.
Wait, I thought this was a course on web scraping with Python.
Well,
it turns out that hiQ Labs has been part of an important recent legal
battle about web scraping. More on the legalities later on.
For now,
I want you to understand how web scraping is key to hiQ's business model.
Almost every company has a big problem, how to attract and keep top people.
That's bad enough,
but companies often have employee talents they don't even
know about. HiQ had an interesting idea.
What if they could scrape LinkedIn to discover
skill‑related data on a company's people?
After all,
most professionals have a detailed LinkedIn profile, and
since the data is self‑maintained,
it's often better than the company's HR database.
Talent analytics is the new term for insights into the workforce, and
hiQ took this a step further by using machine learning to predict who
might be at risk of leaving the company.
Overall,
that's a nice business model, scrape LinkedIn for accurate data on
employee skills and to prevent employee turnover,
and it's working. Owler estimates that hiQ has almost 9
million in revenue and employees 88 people.
A profitable business model is a great reason to scrape the web,
but there's much more you can do