Common Crawl senior research engineer Malte Ostendorff published an analysis of 584,107 llms.txt files from the July 2026 web crawl. The headline findings are significant for any GEO practitioner who has invested effort in llms.txt as a mechanism for governing AI crawler access to their content. Sixty-eight per cent of all files were generated by a plugin or site builder; Wix alone accounts for 41 per cent of the total corpus. Twenty-two per cent of files contain zero links — they are structural shells with no substantive content. Most critically: the format "grants nothing and blocks nothing, and no crawler is obliged to read it." Of 1,570 files that explicitly attempted to restrict specific named AI crawlers, none enforced those same restrictions in their robots.txt — meaning the declared restrictions are entirely symbolic.

The practical distinction is between a format that expresses intent and a mechanism that enforces it. robots.txt works because the major AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and others — have adopted compliance as an operational norm. There is no equivalent norm for llms.txt. A site that declares in its llms.txt that ClaudeBot may not access certain sections has no assurance that ClaudeBot will check, read, or honour that declaration. The format was proposed by Jeremy Howard and adopted widely as a best practice for AI-era web governance. The Common Crawl data suggests that what was adopted was primarily a convention, not a control.

The implications for GEO practitioners are operational. The mechanisms that currently exert actual influence on what AI systems ingest and cite are: robots.txt and Crawl-Delay headers (compliance norm exists among major crawlers); structured data via schema.org markup (shapes how AI systems parse and interpret content at the extraction stage); canonical signals (influence training data deduplication); and content quality signals that affect inclusion decisions made by AI training pipelines rather than by any rule the site operator can specify. llms.txt is worth maintaining as a declaration of intent and for future-compatibility if compliance norms develop — it costs nothing to implement and represents a reasonable signal of governance preference. It should not be treated as a control, included in security or compliance documentation as an active protection, or relied upon to restrict AI access to sensitive content. For content that must not be accessible to AI training pipelines, the only currently enforceable mechanisms are robots.txt and authenticated access.

A companion piece in Search Engine Journal published the same day introduced the Brand Claim Audit as the positive GEO governance framework: systematically cataloguing every factual claim about your brand that AI systems can find and synthesise — across HTML pages, PDFs, product feeds, biographies, job listings, and press releases — and writing explicit bridge content that connects outdated terminology to current reality. The insight is that AI search risk is primarily an information-consistency problem, not a crawl-permission problem. AI answers look settled even when the underlying sources conflict; the conflict is flattened into apparent fact. Solving that requires auditing the corpus, not the robots file.