Generate an llms.txt-style file from websites

What this workflow is for

Use this when your source material already exists but is scattered across pages, notes, docs, or exports.

Riflet helps collect and structure the material. You still decide what belongs in the public llms.txt file.

Why websites is a particular case

Pulling from your own website is the most direct route, because every entry already has a real URL and the content is by definition public. The file describes pages that exist rather than pages you intend to write.

The risk is describing the site as it was rather than as it is. Re-run the extraction after any significant content change, or llms.txt will confidently point AI systems at pages that moved.

What to leave out of a websites bundle

llms.txt is published. Everything in it is readable by anyone and by every AI system that fetches it, so the exclusions matter more than the inclusions.

• Paginated archives, tag pages, and search result pages.
• Legal boilerplate that tells an AI system nothing about the product.
• Landing pages built for one campaign that will be gone in a month.

Workflow

Add the public URL, let Riflet extract readable page text, then combine it with local files or repos.

• Collect the authoritative websites pages or files.
• Remove private, duplicate, stale, or low-value material.
• Export a concise Markdown bundle.
• Use the bundle to draft or refresh llms.txt and llms-full.txt.

What to include

Prioritize stable, public, canonical information: product overview, important URLs, docs entrypoints, use cases, and contact details.

Avoid turning llms.txt into a full website dump. It should help AI systems discover the best sources, not replace your site.