Markdown Delivered to AI Agents via Accept Headers
agents
| Source: HN | Original article
Developers can use HTTP Accept headers to serve a Markdown variant, enabling AI agents to bypass HTML navigation and scripts and read content directly.
A new wave of developer guidance is showing how to serve raw Markdown to large‑language‑model (LLM) agents via the HTTP Accept header, letting AI clients retrieve clean, token‑efficient content instead of full HTML pages. The approach, outlined in a series of recent blog posts and a community tutorial, leverages the standard content‑negotiation mechanism: when an AI agent requests a URL with Accept: text/markdown (or text/plain), the server returns the Markdown source directly, bypassing navigation, scripts and layout markup.
The technique promises a ten‑fold reduction in token consumption for LLMs that ingest documentation, because the model no longer has to parse and discard HTML boiler‑plate. Implementations range from simple server‑side checks that inspect the Accept header to more sophisticated edge‑computing setups using CloudFront Functions and the SST framework, which inject the correct Vary header and handle edge‑case routing. An experimental feature in ModPageSpeed 2.0 also offers automatic Markdown delivery and even generates an /llms.txt index from a site’s sitemap, though it remains license‑gated.
Why it matters is twofold. First, developers can lower API costs and latency when feeding documentation to AI agents, a growing use case as LLMs become assistants for codebases, knowledge bases and customer support. Second, the method aligns web standards with AI consumption patterns, reinforcing the role of HTTP content negotiation in a landscape increasingly dominated by machine clients.
Looking ahead, the community will be watching for broader adoption across CDNs and static‑site generators, as well as any emerging tooling that automates the dual‑format publishing workflow. If the token‑saving claims hold at scale, we may see a shift toward Markdown‑first APIs for AI‑driven services, prompting further refinements to server configurations and possibly new standards for AI‑specific content types.
Sources
Back to AIPULSEN