A study from the Reuters Institute for the Study of Journalism at the University of Oxford found that more news sites worldwide are blocking AI web crawlers
The study, authored by Dr. Richard Fletcher, Director of Research at the Reuters Institute for the Study of Journalism, found that nearly half (48%) of the most popular news sites worldwide are now inaccessible to OpenAI’s crawlers, with Google’s AI crawlers being blocked by 24% of sites.
New @risj_oxford factsheet by me that asks: How many news websites block generative AI like ChatGPT and Gemini from using their content to train their models?
It depends on the country. Very large differences in how many top news sites are blocking, and how soon they started. pic.twitter.com/CaebVc4gfZ
— Richard Fletcher (@richrdfletcher) February 22, 2024
AI crawlers are designed to comb the internet to collect data for AI models like ChatGPT and Gemini. This ensures a steady supply of up-to-date information, pivotal to keeping AI responses accurate and relevant.
Without fresh data, AI models will become locked in time and unable to adapt to the advancements of the real world. If models consume too much poor-quality, synthetic, and AI-generated data rather than new, high-quality, human-produced data, they could even face model collapse.
So, why are news sites blocking AI web crawlers? They’re primarily concerned about copyright and fair compensation, fears of spreading misinformation, and the potential loss of direct traffic to news sites.
The New York Times is suing OpenAI and Microsoft for copyright infringement, joining a host of authors, artists, and businesses who allege AI developers used their data unlawfully.
AI companies understand the problem. That’s why they’re striking licensing deals with media companies like OpenAI’s deal with Axel Springer last year.
Content behemoth Reddit is the latest company to tempt AI companies with multi-million dollar content licensing deals.
Key insights
Here are some key insights from the report:
- As of late 2023, 48% of prominent news platforms internationally had restricted access to OpenAI’s crawlers, with a lesser 24% doing the same for Google’s AI crawler.
- Notably, 97% of sites blocking Google’s AI were also found to block OpenAI’s crawlers.
- The likelihood of websites blocking AI crawlers varied significantly by country, with the highest rates observed in the USA (79%) and the lowest in Mexico and Poland (20%).
- Throughout 2023, no instances of websites reversing their decision to block AI crawlers were recorded.
- Larger news outlets demonstrated a slightly higher propensity to block AI crawlers than smaller ones.
- The tendency to block varies across different types of news organizations. Legacy print outlets (57%) lead in blocking, compared to digital-born outlets (31%)
News companies are evidently fortifying their defenses against AI web crawlers, and AI companies will probably need to deal their way out to keep their models convincingly updated.
The alternative is dire. AI model performance will improve, but knowledge will become slowly outdated to the point of unsatisfactory hallucination rates, inaccuracy, redundancy, and irrelevancy.
