內容簡介
WPG AI Crawler Manager 是一款全面控制 AI 爬蟲的外掛，讓使用者能夠管理爬蟲訪問，生成 llms.txt 檔案，並根據每個爬蟲的需求自訂 robots.txt 規則，實時監控訪問統計，所有數據均儲存在本地伺服器上，無需第三方服務。
【主要功能】
• 生成 llms.txt 檔案，提供網站資訊
• 追蹤 19 種已知 AI 爬蟲
• 自訂爬蟲規則，允許或阻擋特定爬蟲
• 提供本地訪問統計數據
• 友好的控制面板，顯示統計資訊
外掛標籤
開發者團隊
原文外掛簡介
Stop guessing which AI crawlers visit your WordPress site. WPG AI Crawler Manager is the all-in-one control panel for AI bots: serve a clean llms.txt file, block ChatGPT / GPTBot / Claude / Perplexity / Google-Extended with per-bot robots.txt rules (or allow the crawlers you want), and watch live hit statistics — all stored locally on your own server. No third-party services, no accounts, no data ever leaves your site.
Whether you want to opt out of AI training, welcome AI search traffic, or practice generative engine optimization (GEO), this plugin gives you the switchboard — with a per-bot “last seen” log so you know exactly which AI engines are actually crawling you.
Features
llms.txt generator — publish a machine-readable /llms.txt file with your site name, a one-line description, an about paragraph and up to 20 of your best pages, so AI models and LLMs understand what your site is about.
19 known AI crawlers, tracked — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Applebot-Extended, Bytespider, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, cohere-ai, Diffbot, ImagesiftBot and PanguBot, each labelled by organization and purpose (Training / Search / User-triggered / Aggregator).
Block AI bots with robots.txt — a friendly segmented Default / Allow / Block control per bot, written straight into your robots.txt.
Custom crawler rules — block or allow any AI bot that is not in the known list by entering its user-agent token; it is tracked, counted and written to robots.txt just like the known crawlers.
Server-snippet helper — copy-paste Apache and nginx blocks for hosts whose robots.txt is not served by WordPress.
Local hit statistics — see all-time totals, a 7-day bar chart and a per-bot breakdown with “last seen” timestamps, logged entirely in your own database (no phone-home).
Polished dashboard — stat tiles, pure-CSS bar chart, status pills, and a reference guide explaining why you might block one crawler and allow another.
All rules are advisory (well-behaved bots obey them); bigger decisions like “training vs. search” are explained right in the Help screen so you can decide with confidence. Built and maintained by Rakesh Raja (WPGuru).
How it works
When enabled, the plugin intercepts /llms.txt as a virtual file, appends your per-bot rules to the robots.txt output, and watches the user-agent of every front-end visit to log recognized AI crawler hits locally. Every 5 minutes pending hits are merged into the stats, and old days are pruned automatically.
Who is this for?
Site owners who want to block AI training crawlers (GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot) from scraping their content.
Publishers who want AI search visibility and want to allow PerplexityBot, OAI-SearchBot and ChatGPT-User.
SEOs and agencies adding GEO / AI visibility to their service stack — the dashboard gives you client-ready reports of AI crawler activity.
