A new report out today from edge cloud platform provider Fastly Inc. highlights how the explosive growth of artificial intelligence bots is reshaping web traffic, content access and infrastructure risks.

The company’s Q2 2025 Threat Insights Report focuses exclusively on AI bots, providing data on the scale of automated traffic used for training and powering large language models and results, arguably disturbing in their breadth: Today’s internet looks very different from even five years ago.

According to the report, AI bots now represent a rapidly growing share of global web traffic, with crawler activity driving nearly 80% of the volume. Commerce, media, entertainment and high-tech sectors are seeing the highest levels of scraping, reflecting the value of their frequently updated content for AI training.

In terms of which companies are doing what, Meta Platforms Inc. leads the crawler category at 52% of observed traffic, followed by Google LLC at 23% and OpenAI at 20%. Despite ranking third in request volume, OpenAI’s GPTBot covered the widest share of unique websites, indexing 95% of all domains observed in the study.

Fetcher bots, which retrieve live content during inference, make up the remaining 20% of AI traffic. Fastly found that OpenAI dominates this category as well, with ChatGPT and OAI-SearchBot generating 98% of fetcher requests.

The volume created by Fetcher bots, though accounting for only 20% of AI traffic, can cause serious impacts. In one case highlighted in a report, a fetcher peaked at 39,000 requests per minute to a single site, producing distributed-denial-of-service-like effects despite no malicious intent. By comparison, peak crawler loads were lower, though still disruptive to unprotected infrastructure.

There were also regional and vertical differences with AI bot traffic. North America experienced a heavy skew toward crawler traffic, while Europe, the Middle East and Africa received more than half of their AI bot activity from fetchers.

The impact differs across industries. In commerce, healthcare and the public sector, crawlers account for more than 90% of AI bot traffic. By contrast, education and media lean heavily toward fetchers, which drive most of their requests.

Fastly also points to concerns around bias in training data. Much of the material used for AI model development comes from North American websites, embedding the region’s cultural and political perspectives into the systems. Other players focus differently: Japanese groups such as NICT and SoftBank target more Asia-Pacific sources, while Diffbot gathers a larger share of European content. This uneven mix underscores how regional data choices can shape model behavior.

To help manage the growing pressure from AI bots, Fastly is recommending a layered approach. Website operators can use robots.txt rules, rate limits, CAPTCHAs and more advanced bot management tools to separate useful traffic from harmful activity. The report also notes that licensing platforms such as Tollbit give owners the option to channel AI bots toward approved use, turning what was once a drain on resources into a possible revenue stream.

Image: SiliconANGLE/Reve