What is llms.txt and why does it matter for advertisers and publishers?

llms.txt is a plain-text file placed at the root of a website that tells AI language models what content is available on the site and how to retrieve it: the AI-era equivalent of robots.txt for search engines, designed to make a site's content legible to models rather than crawlers.

llms.txt: A plain-text or markdown file at the root of a domain (domain.com/llms.txt) that indexes a site's content for AI models, listing available pages, their purpose, and how they should be retrieved and interpreted.

AI indexing: The process by which AI language models build up knowledge of what content exists at a domain; sites without llms.txt rely on the model having encountered their content during training, which may be outdated or incomplete.

AEO (Answer Engine Optimisation): The practice of structuring content so that AI answer engines retrieve and cite it accurately; llms.txt is the technical infrastructure layer beneath AEO content strategy.

Publisher discoverability: Whether an AI buying agent can find and evaluate a publisher's inventory and deal terms programmatically; without appropriate indexing infrastructure, publishers are not discoverable to agents that navigate by structured content.

Why llms.txt was created

Standard robots.txt tells search engine crawlers which pages they can and cannot index. It was designed for bots that work by following links and downloading page content. AI language models work differently: they retrieve context from sources they have been trained on or can access during a conversation, and they need to understand the purpose and structure of content, not just its URL.

llms.txt provides a structured way to give AI models this context. A site's llms.txt file lists the most important content, groups it by topic or purpose, and can include brief descriptions of what each section contains and how it should be used. A model retrieving context about agentic advertising can find a publisher's llms.txt and immediately understand what is available, without having to infer it from page structure.

The format was proposed by fast.ai in 2024 and has been adopted by a growing number of technical documentation sites, standards bodies, and platforms. Several ad tech and protocol organisations now maintain llms.txt files.

What happens without llms.txt

Without llms.txt, an AI model's knowledge of a site depends on what it encountered during training. If the site has not been updated in the model's training data, or if its content was not well-structured enough to be absorbed accurately, the model's understanding may be outdated, incomplete, or wrong.

For publishers, this means that an AI buying agent evaluating inventory options may not be able to find or accurately evaluate their offering. The agent knows what it has been trained on; if the publisher's current deal terms, inventory capabilities, and contact information are not in a form the agent can retrieve, the publisher is effectively invisible to that agent.

For brands and advertisers, the same logic applies to product documentation. If a brand's product specifications, brand safety guidelines, and audience definitions are not retrievable by AI models in structured form, agents managing campaigns on that brand's behalf cannot operate with accurate information.

What llms.txt contains for an ad tech company

An ad tech publisher's llms.txt would typically list its key content in structured form: inventory descriptions and deal term documentation, technical integration guides, audience and contextual category definitions, contact and API access information, and a pointer to any machine-readable deal term documentation structured to AAMP standards.

A brand's llms.txt would list product documentation, brand guidelines, approved audience segments, restricted content categories, and campaign briefs in structured form. An agent reading this file before executing a campaign would have the context it needs to make accurate decisions without a human briefing it manually.

How to create a basic llms.txt file

A minimal llms.txt is a markdown file at the root of your domain. It starts with the site name and a one-paragraph description of what the site is. It then lists the most important pages grouped by topic, each with a brief description of its purpose. Optional: include a pointer to an llms-full.txt file with the full content of key pages pre-formatted for single-shot AI retrieval.

The file should be kept current. An outdated llms.txt that points to stale content is worse than no file at all, because it signals available content that no longer reflects current capability.

Frequently asked questions

Is llms.txt an official standard?

llms.txt is an emerging convention rather than a ratified standard. It was proposed by fast.ai and has been adopted by a growing number of organisations, including IAB Tech Lab working groups. It does not yet have the status of robots.txt, but adoption is increasing rapidly among AI-forward publishers and platforms.

Do AI buying agents read llms.txt directly?

AI buying agents can be configured to check llms.txt as part of their publisher research workflow. Whether a specific agent does so depends on how it has been built. As the format becomes more common, agent developers are increasingly building llms.txt retrieval into standard discovery routines.

Which ad tech companies currently have llms.txt files?

Several specification and standards bodies in ad tech maintain llms.txt files, including protocol documentation sites. Most DSPs, SSPs, and agencies do not yet have them. Early adoption represents a competitive advantage for publisher and advertiser discoverability in AI-powered buying workflows.