Web Scraping Is Vital

(Paywall-free popularization like this is what I do for a living. To support me, see the end of this post)

Seriously. Even if you have NO idea what it is. Keep reading:

Web Scraping Is Vital /img/web-scraping-is-not-a-crime.jpg

Web scraping means collecting data from websites automatically, by writing programs, sometimes very simple ones, that do all the job (I have written several tutorial myself on how to do it. It is what Google, and every other search engine, do every second, if they must provide any service. In general, it is hard to overestimate the importance of this activity for students, researchers, businesses, watchdogs and “fact checkers” of any kind.

Journalists, for example, have used scrapers to collect data that rooted out extremist cops, tracked lobbyists, and uncovered an underground market for adopted children.

Web scraping is essential for democracy, and human rights in general. This is why The Markup filed an amicus brief in a case before the U.S. Supreme Court this week that threatens to make scraping illegal.

The case itself is about other questions, not directly about scraping: someone was prosecuted under the american Computer Fraud and Abuse Act (CFAA), which prohibits unauthorized access to a computer network such as computer hacking. Since the accused was allowed to access the database for work, the question is whether the court will broadly define his troubling activities as “exceeding authorized access”_ to extract data, which is what would make it a crime under the CFAA.

The problem is that _“such a definition could also affect journalists. Or, as Justice Neil Gorsuch put it… lead in the direction of perhaps making a federal criminal of us all”.

If that line of thought enters jurisprudence, websites that don’t want to be scraped (even if when it would be in the public intereset, and perfectly legal until today) could then just “change the fine print on their terms of service to label the aggregation of that information unauthorized."

And the U.S. Supreme Court, depending on how it rules, could decide that violating those terms of service is a crime under the CFAA. Even when it is normal news gathering by journalists.

What sort of work is at risk?

To have an idea of how bad this could be, check the Markup article to know, quoting again, how web scraping has been successfully used, among other things, to “collect data that rooted out extremist cops, tracked lobbyists, and uncovered an underground market for adopted children”.

Who writes this, why, and how to help

I am Marco Fioretti, tech writer and aspiring polymath doing human-digital research and popularization.
I do it because YOUR civil rights and the quality of YOUR life depend every year more on how software is used AROUND you.

To this end, I have already shared more than a million words on this blog, without any paywall or user tracking, and am sharing the next million through a newsletter, also without any paywall.

The more direct support I get, the more I can continue to inform for free parents, teachers, decision makers, and everybody else who should know more stuff like this. You can support me with paid subscriptions to my newsletter, donations via PayPal (mfioretti@nexaima.net) or LiberaPay, or in any of the other ways listed here.THANKS for your support!