外掛標籤
開發者團隊
② 後台搜尋「FocusWeb AI Crawler Monitor – Track, Verify & Block AI Bots」→ 直接安裝（推薦）
原文外掛簡介
FocusWeb AI Crawler Monitor gives you the full loop for dealing with AI bots on your WordPress site: see which AI crawlers read your content, verify that each visit really comes from the vendor it claims (spoofed bot User-Agents are rampant), decide per bot what happens next — monitor, disallow via robots.txt, block, or block only the fakes — and measure how much real referral traffic AI answer engines (ChatGPT, Perplexity, and more) send you in return. Think of it as AI SEO / answer engine optimization (AEO / GEO) visibility and control for site owners.
Why verification matters
Anyone can send a request with “GPTBot” or “ClaudeBot” in the User-Agent header — and scrapers do exactly that to hide behind reputable names. Independent audits have found that on some sites, effectively all “ClaudeBot” traffic came from non-vendor IP addresses. This plugin checks every claimed bot visit against the vendor’s published IP ranges (refreshed daily) or documented reverse-DNS rules (processed hourly in the background, never in your visitors’ request path), and labels each visit Verified, Spoofed, or Unverifiable. Your analytics become trustworthy, and blocking becomes safe.
Headline feature: Block only fakes
The unique action no comparable plugin offers cleanly: per bot, refuse (403) only the visits that claim its User-Agent but fail verification. Impostor scrapers hiding behind “GPTBot” get blocked; the real GPTBot stays welcome. You protect your server and your content without giving up AI visibility.
Quick start:
Install and activate the plugin — no configuration required, logging and verification start immediately.
Wait for real traffic (or test with a curl request spoofing a bot User-Agent — it will show up flagged as Spoofed).
Check AI Crawlers > Dashboard for crawler visits with verification badges, and AI Crawlers > AI Referrals for human click-throughs from AI answer engines.
When you are ready to act, open AI Crawlers > Crawl Control and choose per bot: Monitor, Disallow (robots.txt), Block (403), or Block only fakes. Everything defaults to Monitor — nothing changes until you opt in.
Optionally generate a suggested llms.txt file from AI Crawlers > llms.txt Generator.
Features:
Verified Bot Detection: every logged visit is checked against vendor-published IP ranges (OpenAI, Perplexity) and reverse-DNS rules (Google, Apple, Amazon, ByteDance), with Verified / Spoofed / Unverifiable badges, summary cards, filters, and CSV export. Range data refreshes daily; reverse-DNS runs in an hourly background batch — zero added latency for visitors.
Crawl Control: per-bot actions with plain-language explanations and one-click presets (“Disallow all training crawlers”, “Block fakes everywhere”). Bots are categorized as AI training crawlers, AI search crawlers, or on-demand fetchers, so the referral-traffic tradeoff of each decision is clear — including hints computed from your own AI referral data.
Block only fakes: 403 spoofed impostors while the genuine crawler keeps its access.
robots.txt manager: Disallow rules served through the WordPress virtual robots.txt (robots_txt filter — no filesystem writes), with a live preview of the final merged file and conflict detection for physical robots.txt files and SEO plugins (Yoast SEO, Rank Math, All in One SEO).
robots.txt compliance flag: a disallowed bot that keeps visiting is labeled “Ignores robots.txt” — your log data becomes the evidence for escalating to a hard block.
Safety rails: everything defaults to Monitor; updates never change behavior until you opt in; blocking never applies to logged-in users; block responses send no-cache headers so page caches and CDNs never cache a 403 for humans.
AI Referral Traffic tracking (AEO / answer-engine-optimization visibility): detects human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, captures the source query when the platform exposes one, and shows a crawl-to-referral conversion insight alongside a per-day-per-source chart.
Dashboard with summary cards (including verified share, spoofed visits, and blocked requests), a Chart.js time-series chart, a most-crawled URL table, and a paginated raw log with CSV export.
Lightweight init-hook based detection, no server log file access required. Custom database tables with indexes for fast queries.
Configurable log retention with a daily cleanup cron job, optional IP anonymization (GDPR-friendly, enabled by default), and excludable URL path patterns.
Built-in llms.txt generator: produces suggested content from your site title, tagline, pages, and posts for manual placement at your site root (see FAQ for llms.txt’s actual adoption status).
Developer-friendly: extend the bot registry, verification rules, and block decisions via the aicm_known_bots, aicm_verification_config, and aicm_block_decision filters.
Fully functional on shared/managed hosting — no filesystem writes outside the database, no special server permissions needed.
How to use this data to optimize for AI:
A high number of crawler visits with few AI referrals is normal today — most AI crawling feeds training or indexing, not real-time answers. The Dashboard’s most-crawled-URL table shows which pages get the most attention; keep those accurate, current, and well-structured.
The AI Referrals page’s Crawl-to-Referral Gap panel puts a number on that relationship so you can track whether it improves.
Before blocking an AI search crawler, check the Crawl Control page’s per-bot referral hints — they show how many visitors that platform actually sent you in the last 30 days.
Check the Captured Queries table for the (best-effort) questions that led people to your site, and favor clear headings, concise direct answers, and FAQ-style content.
Roadmap (planned direction, not a commitment to specific versions or dates):
Correlating individual crawler visits with later referral visits to the same page.
Per-post/per-page llms.txt summaries, instead of one sitewide file.
Additional AI referral sources and bots as new platforms emerge, and additional vendor verification methods as they are published.
A combined “AI visibility” trend view comparing crawl volume and referral traffic over longer ranges.
Suggestions are welcome via the support forum.
