內容簡介
ParseLess 外掛將您的 WordPress 內容轉換為乾淨的 Markdown 格式，專為 AI 爬蟲和代理服務，減少不必要的佈景主題元素和資料庫查詢，提升效能與可讀性。
【主要功能】
• 將文章內容轉換為 Markdown 格式
• 減少 60-80% 的資料庫查詢
• 可選擇記錄請求以了解爬蟲來源
• 阻擋特定爬蟲並設定請求限制
• 支援 WordPress 6.9+ 的 Abilities API
外掛標籤
開發者團隊
原文外掛簡介
ParseLess serves your WordPress content as clean Markdown to AI crawlers and agents — same URL, same content, none of the theme chrome, navigation, widgets or page-builder scaffolding a bot discards anyway. A typical post goes from ~80 KB of HTML to ~4 KB of Markdown, with 60–80% fewer database queries per request.
Serve the good version. Your CDN may offer “Markdown for agents” now, but it converts the rendered page — menus, search buttons, cookie banners and all. ParseLess converts your post content, so an agent gets your article, not your layout. Agents can ask for it the standard way (Accept: text/markdown, a .md address, or ?format=md), and every page advertises where to find it. WooCommerce products come with price, stock and variations.
See who’s really reading. Optional request logging shows which AI companies fetch your content and why — model training, search indexing, live answers while someone chats with an assistant — with a per-bot “last seen” so you can tell when your host is quietly blocking a crawler at the edge. Everything stays in your database; IPs are only ever stored as salted hashes.
Cap what it costs you. Block specific crawlers from your Markdown, declare your terms in robots.txt (Content Signals, AI bot rules, RSL licence), and — if a bot ignores all that — throttle it to a sensible number of requests per minute. Every control is off by default: your stance on AI is yours, not ours.
Agent-native. On WordPress 6.9+ ParseLess registers Abilities (parseless/get-markdown, parseless/list-markdown-urls, parseless/get-stats) so MCP clients and AI agents can discover and fetch your Markdown through the WordPress Abilities API instead of guessing URLs.
It also publishes /llms.txt and /llms-full.txt for AI indexers, and an experimental Open Knowledge Format bundle.
Who is this for?
1. Developers using Claude Code, Cursor, Aider, or any CLI that feeds your own site content into an LLM.
If you’ve ever piped a blog post into Claude or ChatGPT for analysis, rewriting, or summarization, you’ve watched it burn through tokens parsing nav menus, footer markup, and CSS classes that have nothing to do with your content. A typical WordPress page measured live: ~19,800 tokens of HTML for ~975 tokens of actual content — a 20x reduction just by stripping the theme. Heavy page-builder sites (Elementor, Divi) routinely hit 100x or more.
That’s the difference between fitting a handful of pages in a context window and fitting dozens.
Just append ?format=md to any post URL and pipe it straight into your tool of choice:
curl https://yoursite.com/my-post/?format=md | claude "summarize this"
2. Site owners getting “high resource usage” warnings from their host because AI bots are hammering the site.
GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and a dozen others crawl WordPress sites constantly. Each request renders your full theme, runs widget queries, loads page-builder assets, and ships hundreds of KB of HTML per page — most of which the bot discards before extracting the actual text.
ParseLess intercepts these crawls and serves a tiny Markdown payload instead. You keep the AEO/SEO benefit of being indexed by AI search, without paying the server cost of rendering your full theme for every bot hit.
Typical impact on AI bot traffic (measured on real WordPress pages):
95%+ less bandwidth per crawled page (typically 15–30x smaller; e.g. a 79 KB page → 4 KB. Heavy page-builder sites see 100x+)
~60–80% fewer database queries per request
~60% lower peak PHP memory per request
~50–80% faster Time To First Byte (measured 54% on a content-heavy page, 67% on a short one)
On a site with 500 posts crawled monthly by major AI bots, that’s roughly gigabytes → tens of megabytes of monthly bandwidth and a fraction of the cumulative PHP execution time
Numbers vary by theme and content. Heavier setups (Elementor, Divi, Avada) see the biggest savings; lightweight themes see less dramatic but still meaningful gains. The conversion is cached as a transient, so repeated bot hits cost almost nothing.
How it works
AI crawlers are detected by User-Agent and served Markdown automatically — no configuration required.
Humans, search engines, and unknown bots receive your normal HTML output. ParseLess never affects what real visitors see.
The ?format=md query parameter works on any post URL for manual preview or CLI piping.
/llms.txt is published at your site root with a list of available content for AI indexers.
Conversion happens once per post and is cached. Subsequent requests serve a single transient read.
Features
Automatic Markdown for known AI bots (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, and more), plus Accept: text/markdown, .md addresses and ?format=md
Respects noindex flags from Yoast SEO, Rank Math, and Genesis; skips private, draft, password-protected and trashed posts; per-post “Exclude from AI” checkbox
Works with all public post types (configurable); WooCommerce products include price, SKU, stock, attributes and variations
Optional YAML frontmatter and schema.org Article block
Analytics: requests per bot, company and purpose; chat impressions; top URLs; unknown bot-like User-Agents with one-click “add to list”; per-bot reachability with last seen; CSV export. Optional site-wide crawler logging to see bots that get HTML too.
Controls: Blocked Bots (403), per-bot rate limit (429 + Retry-After), Content Signals, AI bot rules and RSL licence in robots.txt — all off by default
Abilities API registration for MCP clients on WordPress 6.9+
/llms.txt, /llms-full.txt, an AI sitemap advertised in robots.txt, and an experimental OKF bundle
Settings page at Tools → ParseLess; per-post meta box with Markdown preview, copy-to-clipboard and the exclude toggle
Zero dependencies, no build step, nothing phones home
For developers
The public API below is frozen for the 1.x series: names, arguments and return types will not change without a major version bump.
Filters
md4ai_should_serve_markdown( bool $serve ) — final say on whether the current request gets Markdown.
md4ai_bot_list( string[] $bots ) — User-Agent substrings treated as AI crawlers.
md4ai_blocked_bots( string[] $bots ) — User-Agent substrings refused with a 403.
md4ai_bot_taxonomy( array $map ) — bot_name => [ company, purpose ]; purposes are training, search, live, agent.
md4ai_rate_limit( int $per_minute, string $bot ) — per-bot request cap when rate limiting is on; return 0 to exempt a bot.
md4ai_unknown_ua_tokens( string[] $tokens ) — substrings that make an unlisted User-Agent “bot-like” for the unknown-UA log.
md4ai_supported_post_types( string[] $types ) — post types served as Markdown.
md4ai_excluded_post_types( string[] $types ) — public post types never served (page-builder templates and the like).
md4ai_excluded_post_ids( int[] $ids ) — individual posts never served on any endpoint.
md4ai_markdown_output( string $markdown, WP_Post $post ) — the finished Markdown document.
md4ai_product_details( string $table, WC_Product $product ) — the WooCommerce product details section.
md4ai_max_variations( int $max ) — cap on listed variations (default 25).
md4ai_cache_ttl( int $seconds, int $post_id ) — cache lifetime; $post_id is 0 for site-wide documents.
md4ai_cache_bypass( bool $bypass ) — skip the cache for this request (default: logged-in users).
md4ai_llmstxt_max_posts( int $max ), md4ai_llmsfull_max_bytes( int $bytes ), md4ai_sitemap_max_posts( int $max ), md4ai_okf_max_posts( int $max ) — endpoint size caps.
md4ai_okf_endpoint( string $slug ) — path segment of the OKF bundle.
md4ai_okf_concept( string $markdown, WP_Post $post ) — one OKF concept file.
md4ai_rsl_terms_url( string $url ), md4ai_rsl_xml( string $xml ) — the RSL licence document.
md4ai_log_request( bool $log, array $context ) — veto a log row before it is written.
Actions
md4ai_logged_request( int $row_id, array $context ) — after a request row is written.
Abilities (WordPress 6.9+, via wp_register_ability())
parseless/get-markdown — input post_id or url; returns post_id, url, title, markdown, tokens.
parseless/list-markdown-urls — input limit (capped by the llms.txt Max Posts setting); returns post_id, title, url, markdown_url, modified per post.
parseless/get-stats — input days (1–365); returns totals, cache hit rate, chat impressions, distinct crawlers and top bots. Requires manage_options.
Storage: one option, md4ai_settings; one table, {prefix}md4ai_requests (only when logging is on); post meta _md4ai_exclude; transients prefixed md4ai_. Uninstall removes all of them.
Privacy and data collection
When logging is enabled (off by default), ParseLess records the following for
each Markdown request:
Post ID, URL, and request timestamp
The full User-Agent header
A salted SHA-256 hash of the requester’s IP address (the raw IP is never stored)
Matched bot identifier, bytes served, and whether the response came from cache
What the request received: Markdown, the normal HTML page, a block (403) or
a throttle (429)
HTML page views are only recorded when “Log All AI Crawler Traffic” is also
enabled (off by default), and only for User-Agents on your bot list. Human
visitors are never logged.
IP hashes cannot be reversed from an email address, so the WordPress privacy
exporter/eraser tools will report no personal data on demand.
Logs are pruned daily according to the configured retention window
(7/30/90/365 days, default 30). Site owners can disable logging or click
“Delete all logged requests” at any time from Tools → ParseLess.
