AI Crawlers Face Search Crawlers in Seven‑Night Census
openai perplexity
| Source: Dev.to | Original article
Over a week, 8,654 bot visits and 247 human visits were recorded, with Perplexity’s AI crawler outperforming OpenAI’s by a 9‑to‑1 margin and accounting for 65% of the activity.
A seven‑night crawl of web‑server logs has revealed that AI‑driven bots are already outpacing traditional search engines in sheer volume. Over the period, the site recorded 8,654 automated “knocks” from bots and just 247 visits from human users. When the traffic was split between AI crawlers and classic search‑engine crawlers, the data showed Perplexity’s bot making nine requests for every one from OpenAI’s, while 65 % of the AI‑bot hits came from agents that did not identify themselves.
The findings matter because AI crawlers—such as PerplexityBot, GPTBot, ClaudeBot and Google‑Extended—are not merely indexing pages for ranking. They harvest content to feed large language models, a use case that places different demands on site owners. Unlike Googlebot or Bingbot, which follow well‑documented robots.txt rules and serve a predictable SEO purpose, AI bots often operate under opaque policies, making it harder for webmasters to control what is scraped and how often. The surge in AI‑bot traffic can strain infrastructure, inflate bandwidth costs, and raise privacy concerns when proprietary or user‑generated content is ingested without clear consent.
Looking ahead, site operators will need to monitor bot activity more closely and consider granular controls beyond the standard robots.txt file. Industry observers expect AI‑bot operators to refine identification strings and possibly introduce rate‑limiting standards, while regulators may scrutinise the unchecked harvesting of web content for training data. Keeping an eye on updates from major AI providers and emerging SEO‑tool guidance will be essential for anyone whose site could become a target of the growing AI‑crawler tide.
Sources
Back to AIPULSEN