As you've just learned,
web scraping itself is often acceptable and even encouraged by website
owners who want search engines to find their content.
But is it legal?
There are relatively few cases,
and the legal precedents are not all that clear, but I'm going to give you
some guidelines. Legal risks are what will review next. Okay,
you probably won't go to jail for web scraping,
but you could get sued, and getting sued could cost you
hundreds of thousands in legal fees.
Jail might be better.
The content on the website may be covered by copyright laws too.
If you scrape an entire website, even part of a website,
then upload it to your site, you can be sued for copyright infringement.
Remember,
the legal risk often depends mainly on what you do with the data and
content after you finish web scraping. Watch out too for private
websites, sites where you have to log in and agree with certain terms
and conditions to use the site.
The terms and conditions may add additional constraints on what you can do.
Still,
website owners may try to impose unreasonable conditions to limit
scraping or prevent you from using data. The practical rule is that for
personal projects, like scraping up a Tesla,
you're usually safe.
The worst that normally happens is you get a nasty letter from
the website owner demanding that you stop,
and even that's unusual. However,
if you're using the data for business or commercial purposes,
plan for some legal expenses, even for a lawsuit.
In what situations is it relatively safe to scrape? Public
government websites. I said public. If you break into a secure
military site and go to jail for espionage,
don't blame me. Scraping for personal use.
I've got a colleague who scrapes eBay looking for good watch prices.
That's really no different than manually searching the site. Or
aggregated data projects for various types of research.
And finally, when the terms and conditions specifically say it's okay,
you're typically safe.
Some situations where there's more legal risk include scraping for personal use,
even though the terms and conditions say it's prohibited, scraping
data you don't own while logged into a website,
or scraping on a large scale to publish news reports that are widely promoted.
There was a recent case where some reporters scraped data
regarding donations to Donald Trump then wrote an article
that was widely promoted.
The data was public, but their work was still controversial.
As long as you can take the heat, go for it.
Hire an attorney first,
or maybe even a team of attorneys any time you're scraping on a
large scale to make money, when you're scraping to create a
commercial product, or if you're scraping large company sites like
Amazon, LinkedIn, or Facebook,
especially if you're doing it for profit, when creating
derivative works, a derivative work is a new original product
that includes aspects of a preexisting, already copyrighted work,
and any time scraping personally identifiable data,
you better watch out.