A new report out today from edge cloud platform provider Fastly Inc. has found that bot traffic continues to reshape the modern web, as artificial intelligence crawlers, fetchers and increasingly sophisticated headless bots drive both operational strain and new business decisions across industries.

The findings, detailed in Fastly’s Q3 Threat Insights Report, found that while humans still account for the majority of web traffic, bots represented roughly 29% of all requests during the quarter. The report highlights how automated activity has now become a structural feature of the internet rather than a fringe security concern.

While bot traffic continues to grow, the AI crawler and fetcher traffic is primarily coming from a small number of large platforms.

Meta Platforms Inc. accounted for approximately 60% of all AI crawler traffic in the third quarter, while OpenAI Group PBC’s ChatGPT was responsible for 68% of AI fetcher traffic.

The bots play a critical role in powering AI search, summaries and conversational responses, but their scale is also forcing organizations to reassess how much automated access they are willing to allow. Only 4% of all wanted bot traffic was found to have been blocked during the quarter, particularly by companies in the media and entertainment and high technology sectors.

The report also highlights the growing threat posed by headless bots, automated programs that use full browser engines to interact with websites that closely mimic real human behavior. The bots can be used for benign tasks such as testing, as well as for malicious activities such as scraping, fraud and credential abuse.

Headless bots accounted for billions of requests in the third quarter, with common headless automation tools making up 94% of that activity. Nearly 89% of headless bot traffic targeted transaction-heavy industries, primarily financial services and commerce, with attackers exploiting automation to scrape data, test credentials or probe for fraud opportunities under the guise of legitimate browsing behavior.

The report notes that organizations now face a more nuanced challenge than simply blocking bots outright, as many of the same tools used by attackers are also used internally for testing, monitoring and performance validation. The problem that arises is that in applying overly aggressive controls, organizations can also disrupt development pipelines and business operations.

With the continued rise of AI-driven traffic, Fastly argues, security teams and business leaders will need more granular visibility and policy controls to balance innovation, content protection and operational resilience in an age of an increasingly automated web.

Image: Fastly