llms.txt Generator
Paste a URL, fetch content via CORS proxy, generate llms.txt or llms-full.txt
What is llms.txt?
llms.txt is a Markdown text file placed in the root directory of a website (similar to robots.txt), specifically designed for large language models (LLMs) and AI crawlers. It summarizes the core information and page links of the website in a structured and readable way, helping AI systems like ChatGPT, Claude, and Perplexity quickly understand the content structure and purpose of the website.
This specification was proposed by Jeremy Howard of Answer.AI in 2024, aiming to provide a standardized 'website manual' for the AI era.
- Contains website name and one-line description
- Lists links and summaries of all important pages
- Small size, AI can read it completely within the context window
- Suitable for daily use on most websites
- Includes full Markdown body content for each page
- AI can obtain all information without accessing the original page
- Large in size, suitable for websites with less content or documentation sites
- Suitable for scenarios where AI needs to deeply understand every page detail
What is it for?
🔍 Enhance AI Search Visibility
When users ask questions in AI search engines like ChatGPT or Perplexity, llms.txt helps AI more accurately understand and cite your website content.
⚡ Reduce Crawling Costs
AI crawlers can directly read structured Markdown without parsing HTML page by page, reducing server load and improving indexing efficiency.
📋 Unified Content Entry
Provides a standardized content entry point for AI agents and RAG systems, making it easy to integrate into various AI workflows.
Usage Tip:After generation, upload the llms.txt file to your website root directory (e.g., https://yoursite.com/llms.txt) and add Sitemap: /llms.txt to your robots.txt to improve discoverability. It is recommended to update this file when your website content is updated.
