llms.txt is useful infrastructure, not an AI discovery strategy

This blog was auto-generated by a multi-agent eval loop, and optimised for SEO / AEO using DataforSEO MCP.

Are llms.txt files mostly pointless as an AI discovery tactic?

That was the question I posted on LinkedIn this week, because the format is starting to attract the kind of certainty that usually appears just before marketers over-invest in the wrong layer of a problem.

The short version is this: llms.txt can be useful infrastructure, but it is not a meaningful discovery strategy on its own.

At query time, an AI chatbot usually does not arrive at your homepage, inspect your website, open /llms.txt and patiently build an inventory of your best pages. Its retrieval layer normally identifies specific URLs through search results, partner indexes or existing retrieval systems, then fetches and evaluates those pages.

Unless that retrieval system has been designed to check /llms.txt, the file is largely invisible. It does not improve rankings, force inclusion or make weak content more likely to be cited.

That distinction matters for anyone trying to build answer engine visibility. The commercial problem is not whether a new text file exists on the server. The commercial problem is whether AI systems can find enough trustworthy evidence to include the brand in an answer.

What the keyword data says

I used DataForSEO MCP to pull a small keyword library around fractional CMO work, AI agentic marketing, forward deployed marketing, AEO and llms.txt.

The interesting result was not that llms.txt had volume. It did: DataForSEO returned UK search volume of 1,600 for llms.txt, compared with 170 for answer engine optimisation, 480 for ai marketing agents, 880 for fractional cmo and only 10 for forward deployed marketing.

The useful point is that the search demand sits across three different layers:

• people trying to understand the technical file;
• people trying to understand answer engine optimisation as a broader discipline;
• buyers looking for senior marketing help with AI systems, fractional leadership or agentic growth.

A good content loop should connect those layers without pretending they are the same query.

For me, llms.txt is a useful entry point because it lets me talk about a live technical question, then move the argument towards the more commercially important work: making a person, company or offer easier for AI systems to identify, understand and cite.

Where llms.txt is genuinely useful

There is a legitimate use case.

Once an agent is already working with a website, a concise map of authoritative pages can help it navigate technical documentation, find clean Markdown versions and avoid wasting context on irrelevant material. That is why the format makes sense for developer documentation, coding agents, controlled RAG systems and agent workflows where the site is already known to the agent.

It is also cheap to add. I would usually include it as part of the basic technical setup for an AI-visible website, alongside a clean sitemap, crawlable pages, schema, metadata and direct internal links to the pages that matter.

The mistake is treating it as the AI equivalent of a sitemap or a shortcut to being cited by ChatGPT, Perplexity, Claude or Gemini. The file simply does not carry that much weight.

The AEO loop that matters more

At a meet-up last week, the head of a Lisbon architecture practice told me they were seeding their business into ChatGPT.

You cannot meaningfully seed LLM training data for your brand. The scale is too vast, the datasets are opaque and the training cycles are outside your control.

What you can do is build an AEO loop around the systems that current AI products actually use.

The loop I am building for a client works like this:

1. Run a fixed set of commercially important prompts across ChatGPT, Claude, Gemini, Perplexity and Copilot.
2. Record which brands, pages and sources appear.
3. Analyse the cited content and identify where the client’s pages can be strengthened.
4. Monitor visibility across Google, Bing and Brave, because different AI products draw on different retrieval layers.
5. Update pages where the evidence, structure or answer quality is weak.
6. Wait for recrawling and run the tests again.

That is more useful than uploading more AI-generated posts and hoping a future model training run notices.

Over time, the client can see which messages, evidence, page structures and third-party references increase the probability of being cited. The objective is consistency across models, not a one-off screenshot where a chatbot happened to mention the brand.

Why search infrastructure still matters

This is where the debate around AEO often becomes too loose.

When an LLM needs current information, it relies on external retrieval. ChatGPT Search may draw on partners including Bing, Gemini relies heavily on Google, and Claude’s web search has been linked to Brave.

Brave is especially interesting because it is one of the few crypto-adjacent products that has crossed into mainstream use, and it is actively positioning its Search API for AI agents and chatbots.

That means the search index influencing your visibility changes according to the assistant your audience uses.

Winning on Google does not automatically mean winning on Bing or Brave. Each engine maintains its own index, ranking signals and coverage before the LLM applies another layer of selection and synthesis.

For a founder-led B2B company, this makes AEO less like a single optimisation trick and more like a monitored evidence system. The website, LinkedIn profile, technical files, third-party references, structured data and search coverage all need to reinforce the same entity.

What I would prioritise before llms.txt

If I were advising a founder, I would not start with a long debate about the file format.

I would start with the fundamentals:

• Can the website clearly explain who the company serves, what it does and why it should be trusted?
• Are the important pages indexable, internally linked and included in the sitemap?
• Does the site contain structured data for the person, organisation, articles, services and FAQs?
• Do key pages answer buyer questions directly enough for an AI system to extract a useful answer?
• Are claims supported by real proof: clients, case studies, methodology, product documentation, press, partner pages or credible external references?
• Does the same positioning appear consistently across the website, LinkedIn and other public profiles?
• Are prompts being tested regularly across the AI products that buyers actually use?

After that, add llms.txt. It is useful infrastructure, but it should sit behind stronger evidence, not in front of it.

The practical takeaway

The market will keep looking for simple AEO hacks because the underlying problem is uncomfortable. You cannot control what a foundation model learned, and you cannot force an AI answer engine to cite you.

You can systematically improve what current AI products find, trust and retrieve.

That means building a loop: prompt testing, citation analysis, search visibility checks, page improvements, technical accessibility and human-approved positioning changes.

This is where AI marketing agents become commercially useful. They can run the research, monitoring, drafting and QA continuously, while a human controls the judgement, proof and positioning.

Treat llms.txt as optional infrastructure for agents that already know you exist.

Treat AEO as the broader work of making your brand a clearer, better-supported answer.

That is the work that matters.