You've seen one example, but there are numerous web scraping use cases.
Let's look at some examples, starting with marketing.
First, there's competitive price and analysis.
Wouldn't you like to be able to monitor your competitor's pricing on
an automated basis? And lead generation. After all,
the web is a gigantic database and there are lots of sales opportunities.
Your scraper could scour the internet looking for people or businesses that
match your company's customer profile or keyword research.
After all,
Google scrapes the web continuously to help you find
the information you're seeking.
For Google to find your company,
you need to know the terms customers are typing into
Google. Scrape competitor websites and compare their
keywords to your company's website.
Many research projects have special data requirements.
Science, I've scraped web pages filled with public earthquake data that was
not available via an API. Scraping was the only option.
Product research, what are similar products or competitive products, and
what are all their product features and capabilities?
Finding or filling a job, that's two sides of the same coin. Job descriptions
and job postings are a rich source of data to find your next web scraping
dream job. And researching what the government's up to, as well as what
people are doing that might interest governments. There are plenty of
financial scraping applications as well.
An obvious one is stock analysis,
monitor prices via an API, then scrape for company news. Insurance and risk
management, life is risky, that's why we have insurance.
But data can reduce that risk.
Then finally, news gathering and news analysis.
We hear a lot about fake news.
Scraping can help us analyze the news and get closer to the truth.
I'm certain you can envision interesting ways to use web scraping,
and I hope some of these examples help you imagine new projects.