What about the legalities of web scraping?
The legal status of web scraping can be murky,
and in this module, we'll review some of the legal issues.
I'm Clarke Bishop, I'm a big data engineer, but I'm not an attorney.
Still, I'll show you what I've learned.
Specifically,
you'll learn when web scraping is legal and whether it's ethical or not,
the legal risks that come up around web scraping,
and we'll explore a recent court case.
Let's get to it.
Many of you may have questions about the legalities or ethics of web scraping.
In this section, you'll start to learn about some of the issues.
As I mentioned, I'm not an attorney.
If you're scraping on a large scale to make money
and create a commercial product, definitely seek good legal advice.
When you do ask an attorney if web scraping is legal,
they're likely to say, that all depends.
It's different from one country or jurisdiction to another,
and the laws are often vague.
Worse, the legal risk depends on your exact web scraping application.
There are many acceptable web scraping uses, though.
So I'll give you an overview about the legal and ethical questions.
Let's start with a couple of easier questions.
Perhaps you've heard that web scraping isn't ethical.
Remember in an earlier module where we compared
human browsing with web scraping?
Web scraping is really just automated web browsing.
You could hire an assistant and have them copy and
paste data into a spreadsheet.
If the website is public,
then surely writing code to do the same thing is ethical.
There are plenty of examples of good bots.
The Electronic Freedom Foundation is a nonprofit
organization defending digital privacy, free speech, and innovation.
Regarding web scraping,
they mention good bot examples like allowing researchers to investigate
racial discrimination on Airbnb or other types of sites,
letting journalists reveal price disparities on Amazon, or
even search engines in general. Website owners usually want
Google to scrape their websites.
There's even an industry devoted to helping search engines scrape websites.
It's called Search Engine Optimization, or SEO.
Public website owners want visitors to find their content.
That's why they spend money to hire SEO consultants.
It's all to make it easier for Google to scrape their sites.
Web scraping itself is certainly ethical, but it is a powerful tool.
It's possible to do unethical things with content after it's scraped,
and that's where the trouble starts.
Maybe you're wondering, can I go to jail for web scraping?
That's unlikely.
As long as the website is public, you're just browsing in an automated way.
Now, if you hack into a private website,
you can get into serious trouble for hacking, only that's hacking,
not web scraping. And don't make this beginner mistake, don't spin up a powerful
server and bombard the website with page download requests.
It can take down the website and look like a denial of
service attack. That can get you into trouble,
too, but it's for violating any hacking laws and for disabling a server.
Web scraping alone is not going to land you in jail.