For years, the conversation around preparing websites for artificial intelligence has been dominated by a single, persistent question: should you create an llms.txt file? The answer, according to Google’s own search team, has been a consistent no, at least when it comes to ranking in traditional search results. Yet, in a move that has sent ripples through the SEO community, Google’s own Chrome Lighthouse tool has introduced a new audit category that explicitly checks for the presence of that very file. This development, buried in the experimental “Agentic Browsing” section of Lighthouse, creates a fascinating and nuanced tension between what Google says you do not need for search and what its own browser-based tools are now measuring for machine readability. The new audits signal a clear strategic shift: while llms.txt may not influence Google Search rankings today, it is increasingly becoming a signal for how well a website can be understood and used by AI agents and browser-based automation tools. This distinction between traditional search discovery and agentic functionality is at the heart of the latest conversation, and it demands a closer look at what exactly Lighthouse is now checking, why it matters, and how web publishers should interpret the apparent contradiction.
The New Lighthouse Agentic Browsing Category and Its Specific Checks
Chrome’s Lighthouse, the widely used open-source tool for auditing web page quality, has expanded its remit beyond performance, accessibility, and SEO. A new experimental category called “Agentic Browsing” is now live, and its purpose is to evaluate how well a site is constructed for machine interaction. This is not about human users navigating with a mouse and keyboard, but about software agents, AI-driven tools, and automated systems that interact with web pages programmatically. The documentation frames this as a set of deterministic audits, meaning they are objective pass-or-fail checks rather than subjective scores. Among the specific items being audited, four stand out as the core pillars of this new evaluation: WebMCP integration, accessibility tree integrity, layout stability through Cumulative Layout Shift (CLS), and the presence of an llms.txt file at the domain root.
The inclusion of llms.txt is the most immediately headline-grabbing component, but it is important to understand it in context. Lighthouse is not checking for the file’s quality or the accuracy of its contents. It is simply verifying its existence at the root of the domain. Google’s own documentation explains the rationale directly, stating that without an llms.txt file, agents may spend more time crawling the site to understand its high-level structure and primary content. This frames the file as a discoverability and efficiency signal for agents, not as a traditional crawling directive like robots.txt. The audit is designed to flag whether a site provides a machine-readable summary that allows an agent to quickly grasp the site’s architecture and main topics without having to parse every single page. The category as a whole does not produce a traditional Lighthouse score out of one hundred. Instead, Google surfaces a fractional pass ratio, along with individual pass or fail indicators for each audit check, providing a dashboard of agentic readiness signals rather than a single number.
The Apparent Tension With Google’s Official Search Guidance
This new development lands with considerable force because it comes less than a week after Google published a major update to its guidance on optimizing for generative AI search features like AI Overviews and AI Mode. In that guidance, Google included a specific mythbusting section that directly addressed llms.txt files. The official recommendation was clear and unequivocal: you do not need to create new machine-readable files, AI text files, markup, or Markdown to appear in generative AI search. The guidance further noted that while Google may discover, crawl, and index many kinds of files in addition to HTML, this does not mean that the file is treated in a special way for search purposes. For many SEO professionals who had been debating the value of creating an llms.txt file, this statement seemed to settle the matter. The file was not a ranking factor and was not required for visibility in Google’s AI-powered search features.
Now, with Chrome’s Lighthouse checking for the file’s presence, the landscape appears contradictory. On one hand, the search team says it is unnecessary. On the other hand, a browser-based auditing tool from the same company is treating its absence as a potential issue. The resolution to this tension lies in understanding the different domains these two pieces of guidance occupy. The search guidance applies specifically to Google Search ranking and the appearance of content in generative AI search features. The Lighthouse audits, by contrast, focus on AI agents and browser tools, not on search rankings. The audits are about how well a site can be used by software agents that may be performing tasks on behalf of a user, such as filling out forms, extracting data, or navigating a workflow. These are two different use cases, and the signals that matter for one do not automatically apply to the other. However, the psychological impact on the SEO community is undeniable. Seeing llms.txt explicitly listed in Chrome’s own readiness checks causes many to rethink earlier doubts, even if the technical justification for the audit is separate from search engine optimization.
John Mueller’s Nuanced Explanation of Discovery Versus Functionality
Perhaps the most illuminating commentary on this entire situation comes from Google’s John Mueller, who addressed the apparent irony head-on in a public exchange on Bluesky. Asked directly why Google itself publishes llms.txt files and Markdown pages for its own developer documentation while simultaneously telling publishers they are not needed for search, Mueller provided a detailed and nuanced response that cuts to the heart of the issue. His short answer was that it is not done for search, adding that there is more to websites than just SEO. This seemingly simple statement carries significant weight because it reframes the entire purpose of such technical files. They are not primarily about being found, but about being functional once a user or an agent has arrived.
Mueller elaborated by drawing a critical distinction between discovery, which means finding the website or pages with a global search engine, and functionality, which he described as helping someone best do the task they want to do once they have found the page. He compared this to a call-to-action button on a traditional page. You do not implement a CTA for SEO reasons to be found, but if you are responsible for a website overall, ensuring a high discovery rate together with a high conversion rate is useful to justify your work. The parallel is direct: an llms.txt file is not a tool for being discovered by a search engine, but it can be a tool for improving the efficiency of an AI agent that is already interacting with your site. Mueller pointed to the developers.google.com site as a prime example. AI coding has become immensely popular, and these coding systems can be more efficient and accurate with the code they produce if they can easily read and parse reference material. In those cases, providing a simplified version of a reference page in Markdown, or a context file like llms.txt, helps the agent understand the documentation it is looking at, saving tokens and processing time.
Mueller was careful to note that agents can read HTML just fine, so he views these files as more of a temporary crutch, perhaps useful for saving tokens in the current state of AI tooling. Crucially, he added a reality check for non-developer websites. He argued that for most sites, such as an e-commerce store selling shoes, creating a Markdown version of a product specification page is not going to increase sales. It primarily benefits competitors who can more easily scrape that structured data. He also cautioned against preparing for a future that may or may not arrive, stating bluntly that sites have much more important things to do for SEO than to prepare for a potential future situation of widespread agentic traffic. His advice was to prioritize current needs before speculative dreams, a perspective that grounds the conversation in practical reality.
Agentic Engine Optimization and the Broader Technical Context
The Lighthouse audits also align closely with a broader set of ideas that have been circulating under the banner of Agentic Engine Optimization, a concept articulated by Google Cloud AI engineering director Addy Osmani earlier in the year. Osmani’s framework directly addresses the limitations of current AI agents, particularly their constrained context windows. He pointed out that agents may cut off long pages or miss important information that is buried too deep within content. His recommendations for addressing these limitations read almost like a checklist for the new Lighthouse category: cleaner semantic structure, token-efficient content, Markdown delivery, llms.txt discovery layers, and capability signaling files like AGENTS.md. The overlap between Osmani’s recommendations and the Lighthouse audits is not coincidental. Both are responding to the same underlying reality: AI agents are becoming more prevalent, and they interact with web content differently than human users or traditional search engine crawlers.
This convergence suggests that the industry is moving toward a more structured, machine-readable web, not necessarily as a replacement for human-friendly design, but as an additional layer that coexists with it. The idea is that a site can serve both audiences simultaneously. A human visitor sees a visually rich, interactive page, while an agent reads a clean, token-efficient Markdown version or consults an llms.txt file for a high-level map of the site’s content. This dual-layer approach is what the Lighthouse audits are beginning to measure, and it represents a significant evolution in how web quality is defined. The audits are not just about checking for the existence of a file, but about evaluating whether the underlying structure of a page is amenable to programmatic interpretation.
What Agents Actually Rely On: Accessibility and Stability as Core Signals
Beyond the specific check for llms.txt, the new Lighthouse category places an unexpectedly strong emphasis on traditional web accessibility and visual stability. This is not an afterthought, but a central pillar of the evaluation. The documentation states explicitly that agents rely on the accessibility tree as their primary data model. This is a profound statement with major implications. The accessibility tree is the structured representation of a web page that assistive technologies, such as screen readers, use to navigate and interpret content. If that tree is broken, missing labels, or hiding interactive elements from assistive systems, an AI agent will face the same difficulties as a human using a screen reader. The audit specifically evaluates whether interactive elements have programmatic labels, whether the accessibility tree has a valid structure, and whether any interactive content is hidden from assistive systems. These checks ensure that the page is not only usable by people with disabilities, but also parsable by software agents that depend on the same underlying data model.
Layout stability, measured through Cumulative Layout Shift, is the other major factor. For a human, a page that jumps around while loading is annoying. For an agent, it can be catastrophic. If an agent is trying to click a button or extract data from a specific position on the page, and the layout shifts at the wrong moment, the agent’s interaction will fail. The audit therefore treats CLS as a signal of agentic readiness, ensuring that the page is stable enough for reliable programmatic interaction. Google also warns that dynamically registered WebMCP tools and large changes to the Document Object Model can negatively affect audit results. This creates a direct link between the technical implementation of a page and its suitability for agentic use. Publishers who invest in solid accessibility practices and stable layouts are not only serving human users better, but are also building a foundation that AI agents can navigate effectively.
Practical Implications for SEO Professionals and Web Publishers
So, what should a responsible web publisher do with all of this information? The first and most important takeaway is to avoid overreacting. Google’s official search guidance remains unchanged. You do not need an llms.txt file to rank in Google Search or to appear in AI Overviews. The Lighthouse audits are experimental and currently sit in the “Agentic Browsing” category, which is separate from the core performance and SEO categories that most site owners monitor. The fractional pass ratio is not a ranking signal, and it is unlikely to become one in the immediate future. However, ignoring the signal entirely would also be a mistake. The audits provide a clear window into how Google is thinking about the future of human-agent-web interaction. The fact that Chrome is now checking for these things means that the browser, and by extension the ecosystem of tools built on top of it, is beginning to value these properties.
For sites that are documentation-heavy, developer-facing, or content-rich in a way that AI agents might plausibly use, creating an llms.txt file is a low-effort, high-upside experiment. The file is simple to create and maintain, and its presence can only help an agent that encounters your site. For e-commerce sites, local business sites, or content that is primarily human-focused, the priority should remain on the fundamentals: solid HTML, clear semantic structure, strong accessibility, and a stable layout. Mueller’s advice to prioritize current needs over speculative future scenarios is sound. If your site has poor Core Web Vitals, broken accessibility, or slow loading times, those issues will harm both human users and any potential agent interactions far more than the absence of an llms.txt file. The new Lighthouse category reinforces the message that a well-built website is a well-built website for all users, whether human or machine. The specific file formats and signaling layers are secondary to the underlying quality of the code.
The broader strategic implication is that the line between human optimization and machine optimization is blurring. Accessibility, once considered purely a human-centric concern, is now recognized as a machine-reading enabler. Layout stability, once a user experience metric, is now an agentic reliability signal. As AI agents become more capable and more widely deployed, the sites that are easiest for them to parse and use will have a functional advantage, even if that advantage does not translate directly into a higher search ranking today. The smartest approach is to build for quality across all dimensions, knowing that the same investments that serve human visitors will also serve the agents of the future. The llms.txt check is merely the most visible tip of a much larger iceberg, one that is shifting the definition of a well-constructed website toward a model that seamlessly accommodates both organic users and programmatic agents.
