llms.txt: the details
It’s a proposal from September 2024, not an official standard. The idea is simple: a website is full of menus, scripts and styles, and a model’s context is limited. The file offers a clean index with a title, a summary and lists of links with a one-line description each. Some sites also publish llms-full.txt, with the full content in a single document.
It pays to keep expectations clear. No major search engine has confirmed using it for ranking or for deciding whom to cite, and it replaces neither robots.txt nor the sitemap. Where it does help is when a person or an agent asks an assistant to look at a specific site: the file gives it the map without crawling page by page. It costs little to maintain if it’s generated from the same content as the website.