內容簡介
PufferSights 是一款專為監控 AI 爬蟲與代理而設計的外掛，支援追蹤超過 135 種已知的 AI 爬蟲，並提供詳細的流量分析與報告，幫助網站管理者了解爬蟲行為及其對網站內容的影響。
【主要功能】
• 監控 135+ 種 AI 爬蟲與代理
• 提供 HTTP 流量與爬蟲目的分析
• 自動生成 llms.txt 內容地圖
• 支援 robots.txt 審核與政策設定
• 追蹤人類推薦來源與爬蟲比例
• 儲存所有分析數據於 WordPress 資料庫
外掛標籤
開發者團隊
② 後台搜尋「PufferSights – AI Crawler Insights」→ 直接安裝（推薦）
原文外掛簡介
PufferSights monitors 135+ known AI crawlers and agents, hashes IP addresses, groups traffic by bot, provider, crawl purpose, content type, and response status, tracks human referrals from AI surfaces, and can publish a dynamic llms.txt content map for public site content.
The dashboard summarizes:
HTTP traffic by bot.
Crawl purpose.
Content type.
Response status.
AI referrals and crawl-to-refer ratio.
Top crawled content.
Tracked agent count.
Dynamic llms.txt content map.
robots.txt audit and policy snippets.
The crawler registry is based on current public operator documentation and industry references for OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta, ByteDance, Microsoft, Amazon, and related AI crawler operators.
The plugin does not contact any external service. All analytics data is stored in your own WordPress database.
robots.txt publishing is off by default. The plugin can generate and optionally publish policies for:
Monitor only.
Block training crawlers.
Allow AI search/user-action bots while blocking training crawlers.
Block all known AI bots.
llms.txt publishing is on by default and can be disabled in the PufferSights settings. The generated /llms.txt file lists selected published public pages and posts in Markdown so AI assistants can find the site’s main public content more easily.
Important Notes
Crawler detection is not bot verification. User agents and signature headers can be spoofed. Known browser agents are recognized by documented signing identities together with Signature and Signature-Input headers. Both a single quoted identity and a single dictionary entry are supported. The plugin does not authenticate signatures or fetch public keys. Header matches are labeled as unverified; unsigned requests and custom user agents may be unrecognized. Raw IP addresses are not stored; the plugin stores a salted hash for rough uniqueness.
robots.txt is voluntary. Use a WAF, CDN, or server-level controls when technical enforcement is required.
Google-Extended and Applebot-Extended are robots.txt control tokens, so they appear in robots.txt audits and policy snippets but are excluded from traffic detection and the tracked-agent count. Blocking Google-Extended restricts both Gemini training and grounding, but does not affect Google Search inclusion. The training policy includes this restriction. The former search-friendly training choice is retained as a compatible saved value but is no longer a separate setting.
ChatGPT Cloud browser has no documented robots.txt token. PufferSights does not generate a rule for it or treat a wildcard robots.txt rule as a supported control for that identity. Use server or CDN controls to restrict its access.
The registry was reviewed on October 3, 2026 and includes 135 trackable identities plus two robots.txt-only controls. Additions include Kimi’s training, search, and user-action crawlers; Kernel Search and Kernel Agent; Meta Muse; Grok Agent; RyeBot; and other documented AI crawlers and browser agents. OnirocoCrawler retains ChathiveCrawler as a legacy alias. Cloudflare AI Search External has separate attribution; older requests remain under Cloudflare AI Search. Known Amazon AgentCore regional signing identities share one entry. ADP research is not classified as AI model training. Documentation and identification limits are available in the dashboard Help tab.
Reflectionbot, CloudflareBrowserRenderingCrawler, and Cloudflare-AI-Search are recognized. Amazonbot is included in training restrictions; Amzn-SearchBot and Amzn-User remain separate. Googlebot, Applebot, and Bingbot are labeled Training & Search, while other ambiguous crawlers retain Mixed Purpose. Google-GeminiNotebook includes Google-NotebookLM as a legacy alias. Exa uses ExaSearchBot; the unconfirmed ExaBot match has been removed. Older ExaBot records retain their original data and may contain inaccurate attribution. Historical charts group by stable crawler IDs and show current names. Legacy Anthropic and Omgili identifiers are labeled in the dashboard Help tab and bot summaries.
Bing currently uses a page-level NOARCHIVE directive for AI training opt-out. This plugin does not add it. NOARCHIVE can also exclude content from Bing AI answers; an existing NOCACHE directive can change its effect. Bing robots.txt training opt-out support is planned for early 2027 and must not be assumed to work today.
The settings offer Monitor only, Request no AI training, and Restrict AI crawlers. The latter targets known AI-specific controls while preserving ordinary search crawlers. When enabled, Content Signals also publish a general ai-train=no preference without adding global Allow or Disallow restrictions. Content Signals are an extension, and crawler support varies. Experimental Content Signals use values and the IETF Content-Usage drafts are not enabled automatically.
AI referrals count recorded requests, not unique visitors or verified humans. Ordinary Bing search and OpenAI corporate-site referrals are excluded from AI referral reports, including older records; the stored data is preserved. The crawl-to-referral ratio compares HTML crawler requests with HTML AI referral requests. It is an aggregate across tracked crawlers, including shared search crawlers, and is not a per-platform attribution metric.
Only requests that execute WordPress can be captured. Static files, requests answered entirely by a page cache or CDN, and requests blocked before WordPress loads are outside these analytics. The plugin does not record every visitor to estimate an AI share of all site traffic.
llms.txt is a content map, not an access-control policy. It does not replace robots.txt and does not force AI systems to use or cite your content.
Privacy
PufferSights stores local analytics for public, logged-out requests only. It does not track wp-admin pages, logged-in users, AJAX requests, or WP-Cron requests.
The plugin stores:
Request time and date.
Event type, such as AI crawler request or AI referral.
HTTP method.
Request path without query string.
HTTP response status.
MIME/content group.
Matched crawler or AI referral provider.
User-agent string and user-agent hash.
Salted one-way hash of the request IP address.
Referrer origin only, such as https://chatgpt.com, without referrer path or query string.
The plugin does not store raw IP addresses, cookies, browser local storage, or complete referrer URLs. It does not send analytics, telemetry, crawler records, or site data to third-party services.
If llms.txt publishing is enabled, the plugin serves a Markdown overview of selected published public posts and pages at /llms.txt. Drafts, private posts, and password-protected posts are not included.
Administrators can disable tracking, clear captured events, and configure retention from the PufferSights admin page. The default retention period is 90 days. On uninstall, the plugin removes its custom analytics table, saved options, and scheduled cleanup hook.
The plugin also adds suggested disclosure text to WordPress’ Privacy Policy Guide.
