內容簡介
Crawler Record 外掛可記錄特定使用者代理（如搜尋引擎及 AI 聊天機器人）最後一次訪問內容的時間，幫助網站管理員了解爬蟲行為，並優化網站的可見性。
【主要功能】
• 記錄多種爬蟲的訪問時間與狀態
• 提供管理頁面查看爬蟲訪問紀錄
• 支援 robots.txt 檔案的檢查與評估
• 隱私友好，僅儲存爬蟲訪問時間與 URL
• 性能優化設計，減少資料庫負擔
外掛標籤
開發者團隊
原文外掛簡介
Crawler Record logs the last time specific user agents (for search and AI chat/LLMs) accessed your content including:
Google
Bing
ChatGPT (OpenAI)
Claude (Anthropic)
Perplexity
DuckDuckGo
Meta
Apple
Common Crawl
You can view this information:
Admin page: a grouped list of all crawlers, with last seen time, last page, and robots status for the crawler.
Admin Bar (front-end): quick-glance status for the current singular or URL context (no dropdowns in wp-admin).
Per Page/Post: From the edit screen on any page or post, you can see that last time each crawler visited that page.
Recent Pages: From the admin page you can select a crawlwer and get a list of the recent pages this UA has visited.
Robots-aware: The plugin checks your robots.txt and evaluates Allow/Disallow rules for a given path. If Settings → Reading → “Discourage search engines” is enabled, all agents are shown as blocked with a prominent warning.
Performance-friendly by design: Write-throttling (default 10 minutes) and an auxiliary “last post ID per agent” record avoids heavy admin queries on large sites.
Privacy-friendly: Saves only bot visit timestamps and last URLs crawled — no personal data.
Learn how to use this plugin.
Highlights
Supports common user-agent variants for Google, Bing, OpenAI (ChatGPT), Anthropic (Claude), Perplexity, Meta, Apple, DuckDuckGo, and Common Crawl.
Robots status computed from local robots.txt (physical or virtual) without outbound requests.
Clear UI with grouped sections, microsecond timestamps, and a small diagnostics toggle showing the matched robots group and rule.
Known limitations
Can only track crawlers from the time the plugin is installed- cannot look into the past.
robots.txt wildcards (*) and end-of-line marker ($) are not interpreted; matching is prefix-based only. Future versions may add full spec support.
To reduce database load and avoid slowing the website, bot tracking is throttled, so not every crawler visit will necessarily be recorded.
Privacy
This plugin stores:
– Timestamps of crawler visits (float, with microseconds)
– Last URL seen per crawler (per-URL records)
– Last post ID per crawler (for admin performance)
It does not collect or store personal data about site visitors. No data is transmitted to third parties.
License
GPLv2 or later. See LICENSE file.
