How Llms.txt And Robots.txt Affect AI Crawlers

If the only comparisons available are written by competitors and by review sites with incomplete information about you, that is the version being used. Publishing an honest one puts a source into circulation that at least contains your figures stated correctly, and honest treatment of where you lose makes the rest of the page more credible rather than less.

These pages are cited heavily and are frequently thin, because most are assembled purely to capture the search phrase. A genuinely useful one that says which alternative suits which situation, including cases where staying put is correct, will outperform a dozen keyword driven versions.

This is the least interesting subject in the discipline and the one that most often explains a total absence from generated answers. A brand can do everything else correctly and remain invisible because a line in a text file, or a setting nobody remembers enabling, is turning the relevant crawlers away.

What Honest Reporting Contains The prompt set, versioned and unchanged since last month. The raw answers, kept in full rather than summarised. Which competitors were named. Which sources were cited. What work was done. What moved, and the specific claim about which work caused it.

The Blocks Nobody Chose Most blocking discovered during audits was never a decision. A disallow copied from a template. A staging rule that survived a migration. A security plugin with an aggressive default. A content delivery network setting labelled bot protection with a switch nobody has looked at since launch.

The idea is reasonable and adoption is inconsistent. Support varies by provider and no major system currently treats it as required. Treat it as a cheap and speculative addition rather than a deliverable worth paying much for.

Category Costs Rise as Coverage Fills In Influencing the third party sources assistants cite is easiest while those sources are thin. A category with two mediocre comparison articles is inexpensive to influence. The same category in three years, once somebody has built the definitive resource that every assistant settles on quoting, is not.

Blocking these is therefore not one decision. Turning away a training crawler is a defensible editorial position. Turning away the agent that fetches pages at answer time removes you from answers entirely, and the two are frequently confused.

This entire area usually amounts to a day of work. It is routinely the difference between a brand that appears in answers and one that does not, and it is worth doing before anybody writes a single word of new content. llm seo

The Baseline Is Worth More the Earlier You Take It A baseline taken today lets you attribute change later. Without one, when something moves you will be reduced to guessing whether it was the assistants, a search update, a competitor's campaign, seasonality or your own site changes.

Assume the pitch is good. Everyone's pitch is good, and the vocabulary in this field is easy enough that a competent salesperson can hold a convincing conversation without anyone behind them who can do the work.

One reframing helps when presenting this internally. Report the channel as influence rather than acquisition. Acquisition framing invites a comparison against paid media on cost per lead, which this channel will lose on the reported numbers even where it is working, because most of its effect never appears as a referral. Influence framing invites the right question, which is whether more of your market arrives already knowing who you are.

Ask What They Will Not Do Good practitioners have a list. They will not guarantee a position in an answer, because nobody controls that. They will not fabricate reviews or seed forum threads under false identities, because it is detectable, damaging and increasingly enforced against.

Equally, do not publish a stripped alternate version of your site for crawlers. Serving different content to machines than to people is cloaking, it has been penalised for two decades, and there is no reason to expect a more forgiving treatment here.

What Not to Do in the Name of Legibility Hidden text intended only for machines fails on every axis. It is detectable, it violates most guidelines, and it produces exactly the uniform low quality signal you were trying to avoid.

Be wary of pricing tied to a proprietary visibility score, since the vendor controls both the number and the prompt set that produces it. Be equally wary of performance pricing tied to mentions, which sounds aligned and creates pressure to game the measurement rather than improve the business.

Ask What Happens in Month One A proposal that opens with content production has skipped the diagnosis. There is no way to know what to write before you know which questions matter, which assistants answer them badly and which sources they draw on.

What llms.txt Proposes It is a proposed convention: a file at your root offering a curated, plain text guide to your site for language model consumers, pointing at the documents you consider authoritative.