內容簡介
CrawlBrief for LLMs.txt 是一款 WordPress 外掛，專為生成結構化的 /llms.txt 文件及可選的 Markdown 版本而設計，旨在為 AI 助手、大型語言模型提供清晰的網站內容概覽，並結合自動生成與編輯控制。
【主要功能】
• 自動生成 /llms.txt 檔案，友善於 LLM 使用
• 支援 llms.txt v2 提案，包含 rel="describedby"
• 可選擇自動或手動控制重要頁面
• 搜尋選擇 WordPress 資源，簡化操作
• 可編輯的 llms.txt 描述，增強內容控制
• 結構化的 llms.txt 區段，提升可讀性
外掛標籤
開發者團隊
原文外掛簡介
CrawlBrief for LLMs.txt is a WordPress llms.txt generator that creates a structured /llms.txt file and optional Markdown versions of public WordPress content for AI assistants, AI agents and large language models.
The plugin supports the llms.txt v2 proposal, including rel="describedby" discovery for /llms.txt, rel="alternate" type="text/markdown" links for eligible WordPress content, and matching HTTP Link metadata on Markdown responses.
CrawlBrief combines automatic generation with editorial control. Instead of publishing an uncontrolled list of every URL, you can choose the pages that matter, organize important resources, edit descriptions, feature or exclude content, and provide clean Markdown representations that compatible AI tools can retrieve when needed.
WordPress llms.txt Generator
CrawlBrief automatically publishes:
/llms.txt – a structured, LLM-friendly overview of your WordPress website.
/index.html.md – a Markdown representation of the homepage.
/example-post/index.html.md – clean Markdown versions of eligible posts, pages and supported public post types.
The endpoints are served dynamically through WordPress. CrawlBrief does not create or overwrite physical llms.txt or Markdown files on your server.
llms.txt v2 Support
CrawlBrief includes discovery features from the current llms.txt v2 proposal:
Adds rel="describedby" so public pages can identify the applicable /llms.txt file.
Adds rel="alternate" type="text/markdown" for pages with a Markdown representation.
Sends matching discovery information through HTTP Link headers on Markdown responses.
Keeps the v2-compatible index.html.md URL format for WordPress URLs without filenames.
Links directly to clean Markdown resources from generated llms.txt sections when available.
Automatic and Manual Page Selection
Choose how CrawlBrief builds the important page sections of your llms.txt file:
Automatic – detect important WordPress pages automatically.
Automatic + Custom – use automatic detection and add your own selected WordPress resources.
Manual – control the selected resources yourself.
CrawlBrief can detect common site pages such as:
Homepage
About page
Contact page
Privacy Policy
Terms page
WordPress posts page
WooCommerce shop page when available
Automatically detected resources can be excluded individually.
Searchable WordPress Resource Selection
Add important WordPress pages using searchable selectors instead of manually copying URLs.
Selected resources are stored by WordPress post ID, so changing a page slug or permalink later does not break the corresponding llms.txt entry.
You can create separate selections for:
Start Here resources
Optional resources
Additional external URLs, archives and documentation hosted outside WordPress can still be added through the Advanced additional-links section.
Editable llms.txt Descriptions
CrawlBrief lets you control how important resources are described to AI systems.
You can:
Override descriptions for automatically detected pages.
Add description overrides to selected Start Here resources.
Add description overrides to selected Optional resources.
Set a dedicated llms.txt description on individual posts or pages.
If no override is supplied, CrawlBrief can fall back to the available WordPress description, excerpt, content-derived text or built-in role description.
Editorial Control
Each supported WordPress resource can use CrawlBrief’s editorial controls:
Automatic – let CrawlBrief include the resource normally.
Featured – prioritize the resource in a dedicated Featured Resources section.
Excluded – prevent the resource from appearing in generated CrawlBrief sections.
This makes CrawlBrief useful for curated llms.txt files rather than simple URL dumps.
Structured llms.txt Sections
Depending on your settings and content, CrawlBrief can generate sections for:
Start Here
Featured Resources
WordPress categories
Public post types
Pages
Site Indexes
Optional resources
You can configure the number of links per section, choose supported public post types, exclude selected category slugs, include the WordPress sitemap, and control optional sections.
CrawlBrief also prevents curated Optional resources from being duplicated in general page sections.
Clean Markdown for AI Assistants
Optional Markdown representations remove much of the surrounding theme and navigation markup and expose useful page content in a simpler text format.
Markdown output can contain:
Page or post title
Short description
Canonical URL
Author
Last-updated date
Featured image
Headings
Paragraphs
Lists
Links
Main page content
If a static WordPress homepage has no stored page body, CrawlBrief can use the configured site summary and detailed context so the homepage Markdown endpoint is still useful.
Caching and Automatic Regeneration
CrawlBrief includes an optional generated-output cache to reduce repeated database work.
It can:
Cache generated llms.txt output.
Regenerate cached content on a daily schedule.
Clear generated output when relevant WordPress content changes.
Invalidate stale output when CrawlBrief is upgraded.
Clear cached output when settings are saved.
Integrate with common WordPress caching plugins when available.
Privacy and Lightweight Operation
CrawlBrief performs generation inside your WordPress installation.
No external API is required.
No account is required.
No visitor analytics are collected by CrawlBrief.
Site content is not automatically sent to LLMs.txt Lab or another third-party service.
No public footer credit or backlink is added.
No physical llms.txt or Markdown files need to be written to the server.
Who Is CrawlBrief For?
CrawlBrief is useful for WordPress sites that want a maintained llms.txt file and clean Markdown content for AI assistants, AI agents, retrieval tools and other systems that support the llms.txt convention.
It can be used on:
Blogs
Educational websites
Documentation sites
Business websites
News and publishing sites
WooCommerce stores
Knowledge bases
Membership and learning websites
Sites using public custom post types
What Is llms.txt?
llms.txt is an emerging Markdown-based convention for giving AI systems a concise overview of a website and links to important machine-readable resources.
It is complementary to existing web standards such as robots.txt and XML sitemaps. CrawlBrief focuses on helping compatible AI assistants and agents discover and retrieve useful site content; it does not promise higher Google rankings or guaranteed inclusion in AI-generated answers.
Works With AI Assistants and Agents
CrawlBrief produces standard web and Markdown resources that can be accessed by compatible AI assistants, agents and retrieval tools, including tools associated with ecosystems such as ChatGPT, Claude, Perplexity and other large language model applications.
Actual use of llms.txt depends on the individual AI system or crawler.
Optional LLMs.txt Lab Tools
CrawlBrief generates and manages llms.txt inside WordPress. For additional checks or manual workflows, LLMs.txt Lab provides optional browser-based tools:
Validate llms.txt – check an existing llms.txt file or URL.
Generate llms.txt – manually create an llms.txt file when needed.
LLMs.txt Lab – access additional llms.txt tools and resources.
These tools are optional. CrawlBrief does not automatically send site content or generated llms.txt data to LLMs.txt Lab.
How to Verify the Plugin
After activation, open Settings > CrawlBrief for LLMs.txt. The settings page shows verification URLs for your website.
To check the generated llms.txt file, visit:
https://example.com/llms.txt
To check the homepage Markdown endpoint, visit:
https://example.com/index.html.md
To check a post or page Markdown endpoint, use the existing CrawlBrief pattern:
https://example.com/example-post/index.html.md
The /llms.txt URL should load as plain text. A post or page Markdown URL should load clean Markdown content with the title, canonical URL, last-updated date, author when available, and page body. The homepage Markdown endpoint contains the available homepage or configured site information.
For llms.txt v2 discovery, view the source of a public HTML page. CrawlBrief outputs:
For an eligible post or page with Markdown representations enabled, it also outputs:
Markdown endpoint responses also advertise the covering llms.txt file and canonical HTML page through the HTTP Link header.
LLMs.txt v2 Compatibility
CrawlBrief 1.2.5 follows the llms.txt v2 discovery model while retaining backward compatibility with existing CrawlBrief installations.
/llms.txt remains at the site root and covers the site’s public URLs.
Existing /index.html.md and /post/index.html.md representations are retained. llms.txt v2 explicitly permits index.html.md for URLs without file names.
Agents can discover Markdown representations through rel=”alternate” type=”text/markdown” rather than guessing URL transformations.
Public HTML pages point to the covering llms.txt with rel=”describedby”.
Markdown responses expose the same relationship through an HTTP Link header.
The Optional section remains available as a convention for secondary resources that agents may skip when a shorter context is needed; it is not treated as a special processing instruction.
Category and custom-post-type archive links remain normal HTML links unless CrawlBrief can actually serve a Markdown representation for the URL.
The llms.txt proposal remains an evolving community specification. See llmstxt.org for the current proposal.
Cache Compatibility
The plugin clears its internal transient cache when configured content changes. It also attempts to purge common caching plugins when available, including LiteSpeed Cache, WP Rocket, W3 Total Cache, WP Super Cache and SiteGround Optimizer.
Other caching plugins can hook into:
crwbrf_txt_cache_purged
Privacy
This plugin does not collect analytics, track visitors or automatically send generated llms.txt content or site data to a third-party service. All generation happens inside WordPress using public site content and settings saved by the site administrator. Links to optional external tools, including LLMs.txt Lab, are opened only when an administrator chooses to visit them. CrawlBrief does not automatically transmit site content or generated llms.txt data through those links.
