[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$fDfULXbPY7myOMgHP1AuEKOwMqR2W9BUsi5v8eSf9L1g":3,"$fgQBXBzyYntnx-L5jNOhtHfTzgPuKiiSdUXpv_MKhFXY":129,"$fBHJcDzGcGPLke5agcl4LZIQL5S1nVlrJDZ6Hy5wOHoo":134},{"slug":4,"name":5,"version":6,"author":7,"author_profile":8,"description":9,"short_description":10,"active_installs":11,"downloaded":12,"rating":11,"num_ratings":11,"last_updated":13,"tested_up_to":14,"requires_at_least":15,"requires_php":16,"tags":17,"homepage":23,"download_link":24,"security_score":25,"vuln_count":11,"unpatched_count":11,"last_vuln_date":26,"fetched_at":27,"discovery_status":28,"vulnerabilities":29,"developer":30,"crawl_stats":26,"alternatives":39,"analysis":26,"fingerprints":26},"focusweb-ai-crawler-monitor","FocusWeb AI Crawler Monitor – Track, Verify & Block AI Bots","2.0.0","Uri N.","https:\u002F\u002Fprofiles.wordpress.org\u002Fupi\u002F","\u003Cp>FocusWeb AI Crawler Monitor gives you the full loop for dealing with AI bots on your WordPress site: \u003Cstrong>see\u003C\u002Fstrong> which AI crawlers read your content, \u003Cstrong>verify\u003C\u002Fstrong> that each visit really comes from the vendor it claims (spoofed bot User-Agents are rampant), \u003Cstrong>decide\u003C\u002Fstrong> per bot what happens next — monitor, disallow via robots.txt, block, or block only the fakes — and \u003Cstrong>measure\u003C\u002Fstrong> how much real referral traffic AI answer engines (ChatGPT, Perplexity, and more) send you in return. Think of it as AI SEO \u002F answer engine optimization (AEO \u002F GEO) visibility \u003Cem>and\u003C\u002Fem> control for site owners.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Why verification matters\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Anyone can send a request with “GPTBot” or “ClaudeBot” in the User-Agent header — and scrapers do exactly that to hide behind reputable names. Independent audits have found that on some sites, effectively all “ClaudeBot” traffic came from non-vendor IP addresses. This plugin checks every claimed bot visit against the vendor’s published IP ranges (refreshed daily) or documented reverse-DNS rules (processed hourly in the background, never in your visitors’ request path), and labels each visit \u003Cstrong>Verified\u003C\u002Fstrong>, \u003Cstrong>Spoofed\u003C\u002Fstrong>, or \u003Cstrong>Unverifiable\u003C\u002Fstrong>. Your analytics become trustworthy, and blocking becomes safe.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Headline feature: Block only fakes\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>The unique action no comparable plugin offers cleanly: per bot, refuse (403) only the visits that claim its User-Agent but fail verification. Impostor scrapers hiding behind “GPTBot” get blocked; the real GPTBot stays welcome. You protect your server and your content without giving up AI visibility.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Quick start:\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Col>\n\u003Cli>Install and activate the plugin — no configuration required, logging and verification start immediately.\u003C\u002Fli>\n\u003Cli>Wait for real traffic (or test with a \u003Ccode>curl\u003C\u002Fcode> request spoofing a bot User-Agent — it will show up flagged as Spoofed).\u003C\u002Fli>\n\u003Cli>Check \u003Cstrong>AI Crawlers > Dashboard\u003C\u002Fstrong> for crawler visits with verification badges, and \u003Cstrong>AI Crawlers > AI Referrals\u003C\u002Fstrong> for human click-throughs from AI answer engines.\u003C\u002Fli>\n\u003Cli>When you are ready to act, open \u003Cstrong>AI Crawlers > Crawl Control\u003C\u002Fstrong> and choose per bot: Monitor, Disallow (robots.txt), Block (403), or Block only fakes. Everything defaults to Monitor — nothing changes until you opt in.\u003C\u002Fli>\n\u003Cli>Optionally generate a suggested llms.txt file from \u003Cstrong>AI Crawlers > llms.txt Generator\u003C\u002Fstrong>.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>\u003Cstrong>Features:\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Verified Bot Detection\u003C\u002Fstrong>: every logged visit is checked against vendor-published IP ranges (OpenAI, Perplexity) and reverse-DNS rules (Google, Apple, Amazon, ByteDance), with Verified \u002F Spoofed \u002F Unverifiable badges, summary cards, filters, and CSV export. Range data refreshes daily; reverse-DNS runs in an hourly background batch — zero added latency for visitors.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Crawl Control\u003C\u002Fstrong>: per-bot actions with plain-language explanations and one-click presets (“Disallow all training crawlers”, “Block fakes everywhere”). Bots are categorized as AI training crawlers, AI search crawlers, or on-demand fetchers, so the referral-traffic tradeoff of each decision is clear — including hints computed from your own AI referral data.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Block only fakes\u003C\u002Fstrong>: 403 spoofed impostors while the genuine crawler keeps its access.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>robots.txt manager\u003C\u002Fstrong>: Disallow rules served through the WordPress virtual robots.txt (\u003Ccode>robots_txt\u003C\u002Fcode> filter — no filesystem writes), with a live preview of the final merged file and conflict detection for physical robots.txt files and SEO plugins (Yoast SEO, Rank Math, All in One SEO).\u003C\u002Fli>\n\u003Cli>\u003Cstrong>robots.txt compliance flag\u003C\u002Fstrong>: a disallowed bot that keeps visiting is labeled “Ignores robots.txt” — your log data becomes the evidence for escalating to a hard block.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Safety rails\u003C\u002Fstrong>: everything defaults to Monitor; updates never change behavior until you opt in; blocking never applies to logged-in users; block responses send no-cache headers so page caches and CDNs never cache a 403 for humans.\u003C\u002Fli>\n\u003Cli>AI Referral Traffic tracking (AEO \u002F answer-engine-optimization visibility): detects human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, captures the source query when the platform exposes one, and shows a crawl-to-referral conversion insight alongside a per-day-per-source chart.\u003C\u002Fli>\n\u003Cli>Dashboard with summary cards (including verified share, spoofed visits, and blocked requests), a Chart.js time-series chart, a most-crawled URL table, and a paginated raw log with CSV export.\u003C\u002Fli>\n\u003Cli>Lightweight \u003Ccode>init\u003C\u002Fcode>-hook based detection, no server log file access required. Custom database tables with indexes for fast queries.\u003C\u002Fli>\n\u003Cli>Configurable log retention with a daily cleanup cron job, optional IP anonymization (GDPR-friendly, enabled by default), and excludable URL path patterns.\u003C\u002Fli>\n\u003Cli>Built-in llms.txt generator: produces suggested content from your site title, tagline, pages, and posts for manual placement at your site root (see FAQ for llms.txt’s actual adoption status).\u003C\u002Fli>\n\u003Cli>Developer-friendly: extend the bot registry, verification rules, and block decisions via the \u003Ccode>aicm_known_bots\u003C\u002Fcode>, \u003Ccode>aicm_verification_config\u003C\u002Fcode>, and \u003Ccode>aicm_block_decision\u003C\u002Fcode> filters.\u003C\u002Fli>\n\u003Cli>Fully functional on shared\u002Fmanaged hosting — no filesystem writes outside the database, no special server permissions needed.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>How to use this data to optimize for AI:\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>A high number of crawler visits with few AI referrals is normal today — most AI crawling feeds training or indexing, not real-time answers. The Dashboard’s most-crawled-URL table shows which pages get the most attention; keep those accurate, current, and well-structured.\u003C\u002Fli>\n\u003Cli>The AI Referrals page’s Crawl-to-Referral Gap panel puts a number on that relationship so you can track whether it improves.\u003C\u002Fli>\n\u003Cli>Before blocking an AI search crawler, check the Crawl Control page’s per-bot referral hints — they show how many visitors that platform actually sent you in the last 30 days.\u003C\u002Fli>\n\u003Cli>Check the Captured Queries table for the (best-effort) questions that led people to your site, and favor clear headings, concise direct answers, and FAQ-style content.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>Roadmap (planned direction, not a commitment to specific versions or dates):\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Correlating individual crawler visits with later referral visits to the same page.\u003C\u002Fli>\n\u003Cli>Per-post\u002Fper-page llms.txt summaries, instead of one sitewide file.\u003C\u002Fli>\n\u003Cli>Additional AI referral sources and bots as new platforms emerge, and additional vendor verification methods as they are published.\u003C\u002Fli>\n\u003Cli>A combined “AI visibility” trend view comparing crawl volume and referral traffic over longer ranges.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Suggestions are welcome via the support forum.\u003C\u002Fp>\n","Track, verify and block AI crawlers (GPTBot, ClaudeBot, PerplexityBot). Spot spoofed bots, control robots.txt, measure AI referral traffic.",0,170,"2026-07-11T12:47:00.000Z","7.0.2","6.0","7.4",[18,19,20,21,22],"ai","bot","crawler","llms-txt","robots-txt","","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Ffocusweb-ai-crawler-monitor.2.0.0.zip",100,null,"2026-07-22T17:31:50.256Z","no_bundle",[],{"slug":31,"display_name":32,"profile_url":8,"plugin_count":33,"total_installs":34,"avg_security_score":35,"avg_patch_time_days":36,"trust_score":37,"computed_at":38},"upi","pressupinc",19,5160,90,30,87,"2026-08-29T16:52:11.764Z",[40,61,75,94,111],{"slug":41,"name":42,"version":43,"author":44,"author_profile":45,"description":46,"short_description":47,"active_installs":48,"downloaded":49,"rating":35,"num_ratings":50,"last_updated":51,"tested_up_to":14,"requires_at_least":52,"requires_php":16,"tags":53,"homepage":23,"download_link":57,"security_score":58,"vuln_count":59,"unpatched_count":11,"last_vuln_date":60,"fetched_at":27},"better-robots-txt","Better Robots.txt – AI-Ready Crawl Control & Bot Governance","3.1.2","Pagup","https:\u002F\u002Fprofiles.wordpress.org\u002Fpagup\u002F","\u003Cp>Better Robots.txt replaces the default WordPress robots.txt workflow with a smarter, structured version you can configure and preview before publishing.\u003C\u002Fp>\n\u003Cp>Instead of a blank textarea, you get a guided wizard with presets, plain-language explanations, and a final Review & Save step so you can inspect the generated robots.txt before it goes live.\u003C\u002Fp>\n\u003Cp>Built for beginners and advanced users alike, Better Robots.txt helps you control how search engines, AI crawlers, SEO tools, archive bots, bad bots, social preview bots, and other automated agents interact with your site.\u003C\u002Fp>\n\u003Cp>Trusted by thousands of WordPress sites, Better Robots.txt is designed for the AI era without resorting to hype, vague promises, or hidden rules.\u003C\u002Fp>\n\u003Cp>Better Robots.txt is available in Free, Pro, and Premium editions. The free plugin covers the guided workflow and essential crawl control features, while Pro and Premium unlock additional governance, protection, and AI-ready modules. Some screenshots on the plugin page show features from all three editions.\u003C\u002Fp>\n\u003Ch3>A quick overview\u003C\u002Fh3>\n\u003Cp>\u003Ciframe loading=\"lazy\" title=\"Better robots.txt Video — AI-Ready Crawl Control for WordPress\" src=\"https:\u002F\u002Fplayer.vimeo.com\u002Fvideo\u002F1169756981?dnt=1&app_id=122963\" width=\"750\" height=\"372\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\">\u003C\u002Fiframe>\u003C\u002Fp>\n\u003Ch3>Why Better Robots.txt is different\u003C\u002Fh3>\n\u003Cp>Most robots.txt plugins fall into one of three categories:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Simple text editor\u003C\u002Fli>\n\u003Cli>Virtual robots.txt manager\u003C\u002Fli>\n\u003Cli>Single-purpose AI or policy add-on\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Better Robots.txt goes further.\u003C\u002Fp>\n\u003Cp>It gives you a complete, guided crawl control workflow so you can:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Choose a preset that matches your goals\u003C\u002Fli>\n\u003Cli>Control major crawler categories without writing everything by hand\u003C\u002Fli>\n\u003Cli>Keep core WordPress protection rules visible and editable\u003C\u002Fli>\n\u003Cli>Clean up low-value crawl paths that waste crawl budget\u003C\u002Fli>\n\u003Cli>Generate a cleaner robots.txt output\u003C\u002Fli>\n\u003Cli>Preview the final result before saving\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>What you can control\u003C\u002Fh3>\n\u003Cp>Better Robots.txt helps you manage:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Search engine visibility\u003C\u002Fli>\n\u003Cli>AI and LLM crawler behavior\u003C\u002Fli>\n\u003Cli>AI usage signals such as search, ai-input, and ai-train preferences\u003C\u002Fli>\n\u003Cli>SEO tool crawlers\u003C\u002Fli>\n\u003Cli>Bad bots and abusive crawlers\u003C\u002Fli>\n\u003Cli>Archive and Wayback access\u003C\u002Fli>\n\u003Cli>Feed crawlers and crawl traps\u003C\u002Fli>\n\u003Cli>WooCommerce crawl cleanup\u003C\u002Fli>\n\u003Cli>CSS, JavaScript, and image crawling rules\u003C\u002Fli>\n\u003Cli>Social media preview crawlers\u003C\u002Fli>\n\u003Cli>ads.txt and app-ads.txt allowance\u003C\u002Fli>\n\u003Cli>llms.txt generation\u003C\u002Fli>\n\u003Cli>Advanced directives such as crawl-delay and custom rules\u003C\u002Fli>\n\u003Cli>Final review before publishing\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Editions\u003C\u002Fh3>\n\u003Cp>Better Robots.txt is available in three editions:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Free – Includes the guided setup, the Essential preset, core crawl control features, and the final Review & Save workflow.\u003C\u002Fli>\n\u003Cli>Pro – Adds more advanced governance and protection modules, including additional AI, crawler, and cleanup controls.\u003C\u002Fli>\n\u003Cli>Premium – Unlocks the most restrictive and advanced protection options, including the Fortress preset and additional high-control modules.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Some options shown in the interface are marked Free, Pro, or Premium so users can immediately understand which modules belong to each edition.\u003C\u002Fp>\n\u003Ch3>AI crawler controls by edition\u003C\u002Fh3>\n\u003Cp>The Free edition lets site owners choose a basic global policy for AI search and discovery crawlers: allow or block. AI training crawlers remain separated from AI search crawlers, so a site can block training use while still allowing AI-powered search and answer discovery.\u003C\u002Fp>\n\u003Cp>The Pro and Premium editions add advanced controls, including per-bot AI crawler rules, custom user agents, AI usage signals, llms.txt, and additional machine-readable governance files.\u003C\u002Fp>\n\u003Cp>Better Robots.txt separates AI-related agents into three practical families:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>AI training crawlers, such as GPTBot, Google-Extended, and ClaudeBot.\u003C\u002Fli>\n\u003Cli>AI search and discovery crawlers, such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot.\u003C\u002Fli>\n\u003Cli>User-action fetchers, such as ChatGPT-User, Perplexity-User, and Claude-User.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>User-action fetchers are triggered by user requests. Depending on the provider, robots.txt may not apply to those fetches in the same way as it applies to automated crawlers. For stronger enforcement, combine robots.txt with server logs, WAF rules, provider verification, or IP-based controls where available.\u003C\u002Fp>\n\u003Ch3>Presets\u003C\u002Fh3>\n\u003Cp>Setup starts with four modes:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Essential – A clean, practical configuration for most websites that want a better robots.txt without complexity.\u003C\u002Fli>\n\u003Cli>AI-First – For publishers and content sites that want AI-ready governance without shutting down discovery.\u003C\u002Fli>\n\u003Cli>Fortress – For websites that want stronger protection against scraping, archive capture, and unnecessary crawl activity.\u003C\u002Fli>\n\u003Cli>Custom – For users who prefer to configure each module manually.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>For many sites, one preset plus a quick review is enough.\u003C\u002Fp>\n\u003Ch3>Built for beginners and experts\u003C\u002Fh3>\n\u003Cp>Beginners get:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>A guided setup instead of a raw robots.txt box\u003C\u002Fli>\n\u003Cli>Preset-based configuration\u003C\u002Fli>\n\u003Cli>Plain-language explanations for important choices\u003C\u002Fli>\n\u003Cli>A safer workflow with a final preview step\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Advanced users get:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Editable core WordPress protection rules\u003C\u002Fli>\n\u003Cli>Fine-grained crawler controls by category\u003C\u002Fli>\n\u003Cli>WooCommerce-oriented cleanup options\u003C\u002Fli>\n\u003Cli>Consolidated output options\u003C\u002Fli>\n\u003Cli>Advanced directives and custom rules\u003C\u002Fli>\n\u003Cli>A final output they can inspect before publishing\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>AI-ready, without hype\u003C\u002Fh3>\n\u003Cp>Better Robots.txt includes features for modern AI-related crawl governance, including:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>AI crawler handling\u003C\u002Fli>\n\u003Cli>Optional llms.txt support\u003C\u002Fli>\n\u003Cli>AI usage signals for compliant systems\u003C\u002Fli>\n\u003Cli>Optional machine-readable governance signals for advanced use cases\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>These features help you express how you want automated systems to use your content.\u003C\u002Fp>\n\u003Cp>However, Better Robots.txt does not claim to control AI by force. Like robots.txt itself, these signals are most useful with compliant systems and good-faith crawlers.\u003C\u002Fp>\n\u003Ch3>What Better Robots.txt is\u003C\u002Fh3>\n\u003Cp>Better Robots.txt is:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>A robots.txt governance plugin for WordPress\u003C\u002Fli>\n\u003Cli>A guided configuration workflow instead of a raw text editor\u003C\u002Fli>\n\u003Cli>A crawl control layer to reduce wasteful crawling\u003C\u002Fli>\n\u003Cli>A practical bridge between SEO, crawl hygiene, and AI-era policy signaling\u003C\u002Fli>\n\u003Cli>A way to keep your crawl policy clearer for humans and machines\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Technical reference for advanced users: Better Robots.txt also maintains a public \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FGautierDorval\u002Fbetter-robots-txt\" rel=\"nofollow noopener noreferrer ugc\">GitHub repository\u003C\u002Fa> with product definition, governance notes, and machine-readable artefacts.\u003C\u002Fp>\n\u003Ch3>What Better Robots.txt is not\u003C\u002Fh3>\n\u003Cp>Better Robots.txt is not:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>A firewall or Web Application Firewall (WAF)\u003C\u002Fli>\n\u003Cli>An anti-scraping enforcement engine\u003C\u002Fli>\n\u003Cli>A legal compliance engine\u003C\u002Fli>\n\u003Cli>A guarantee that every bot will obey your rules\u003C\u002Fli>\n\u003Cli>A replacement for server-level security or access control\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>It helps you publish a clearer crawl policy.\u003C\u002Fp>\n\u003Cp>It does not replace infrastructure-level protection.\u003C\u002Fp>\n\u003Ch3>Typical use cases\u003C\u002Fh3>\n\u003Cp>Use Better Robots.txt if you want to:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Clean up a weak or noisy default robots.txt\u003C\u002Fli>\n\u003Cli>Reduce crawl waste on WordPress or WooCommerce\u003C\u002Fli>\n\u003Cli>Keep major search engines allowed while restricting other bots\u003C\u002Fli>\n\u003Cli>Control whether archive bots can snapshot your site\u003C\u002Fli>\n\u003Cli>Publish AI usage preferences more clearly\u003C\u002Fli>\n\u003Cli>Keep social preview bots allowed while limiting scrapers\u003C\u002Fli>\n\u003Cli>Review the final file before making it live\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Key Features\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Guided step-by-step wizard\u003C\u002Fli>\n\u003Cli>Preset-based setup: Essential, AI-First, Fortress, Custom\u003C\u002Fli>\n\u003Cli>Search engine visibility controls\u003C\u002Fli>\n\u003Cli>AI and LLM crawler governance\u003C\u002Fli>\n\u003Cli>AI usage signals support\u003C\u002Fli>\n\u003Cli>SEO tool crawler controls\u003C\u002Fli>\n\u003Cli>Bad bot and abusive crawler options\u003C\u002Fli>\n\u003Cli>Archive and Wayback access controls\u003C\u002Fli>\n\u003Cli>Spam, feed, and crawl trap cleanup\u003C\u002Fli>\n\u003Cli>WooCommerce crawl cleanup options\u003C\u002Fli>\n\u003Cli>CSS, JavaScript, and image crawling rules\u003C\u002Fli>\n\u003Cli>Social media preview crawler controls\u003C\u002Fli>\n\u003Cli>ads.txt and app-ads.txt allowance\u003C\u002Fli>\n\u003Cli>Optional llms.txt generation\u003C\u002Fli>\n\u003Cli>Consolidated output option\u003C\u002Fli>\n\u003Cli>Core WordPress protection rules remain visible and editable\u003C\u002Fli>\n\u003Cli>Final Review & Save preview screen\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>About the publisher\u003C\u002Fh4>\n\u003Cp>Better Robots.txt is developed and maintained by \u003Ca href=\"https:\u002F\u002Fpagup.com\u002F\" rel=\"nofollow ugc\">Pagup\u003C\u002Fa>, a digital readability firm based in Quebec, Canada. Pagup helps organizations become correctly understood by search engines, generative AI systems, and autonomous agents.\u003C\u002Fp>\n\u003Cp>The robots.txt file is the first surface that AI crawlers read when they discover a site. A well-structured robots.txt that references governance files such as llms.txt, ai-manifest.json, and interpretation policies helps AI systems understand your site faster and more accurately.\u003C\u002Fp>\n\u003Cp>Better Robots.txt is one component of a broader digital readability practice that includes \u003Ca href=\"https:\u002F\u002Fpagup.com\u002Fen\u002Fservices\u002Fsemantic-content-architecture\u002F\" rel=\"nofollow ugc\">semantic content architecture\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fpagup.com\u002Fen\u002Fservices\u002Fai-governance-and-machine-readability\u002F\" rel=\"nofollow ugc\">AI governance and machine readability\u003C\u002Fa>, and \u003Ca href=\"https:\u002F\u002Fpagup.com\u002Fen\u002Fglossary\u002Finterpretive-seo\u002F\" rel=\"nofollow ugc\">interpretive SEO\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch4>Part of the Pagup ecosystem\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fpagup.com\u002F\" rel=\"nofollow ugc\">pagup.com\u003C\u002Fa> — Digital readability firm. Diagnostic, semantic architecture, AI governance.\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fgautierdorval.com\u002F\" rel=\"nofollow ugc\">gautierdorval.com\u003C\u002Fa> — Doctrine, canonical definitions, interpretive governance research.\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Finterpretive-governance.org\u002F\" rel=\"nofollow ugc\">interpretive-governance.org\u003C\u002Fa> — Formal versioned standard for interpretive governance.\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fbetter-robots.com\u002F\" rel=\"nofollow ugc\">better-robots.com\u003C\u002Fa> — Documentation and resources for Better Robots.txt.\u003C\u002Fli>\n\u003C\u002Ful>\n","Replace the default WordPress robots.txt workflow with a smarter, structured version you can preview before publishing, with Free, Pro, and Premium ed …",5000,321566,102,"2026-07-04T14:07:00.000Z","5.0",[54,55,21,22,56],"ai-crawlers","bot-blocker","seo","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Fbetter-robots-txt.3.1.2.zip",99,2,"2023-02-14 00:00:00",{"slug":62,"name":63,"version":64,"author":65,"author_profile":66,"description":67,"short_description":68,"active_installs":11,"downloaded":69,"rating":11,"num_ratings":11,"last_updated":70,"tested_up_to":14,"requires_at_least":52,"requires_php":16,"tags":71,"homepage":73,"download_link":74,"security_score":25,"vuln_count":11,"unpatched_count":11,"last_vuln_date":26,"fetched_at":27},"miniorange-ai-visibility","AI Crawlers Visibility, Analytics & Control","1.2.0","miniOrange","https:\u002F\u002Fprofiles.wordpress.org\u002Fcyberlord92\u002F","\u003Cp>\u003Cstrong>AI Crawlers Visibility, Analytics & Control\u003C\u002Fstrong> helps WordPress site owners understand and manage how AI systems access their content. Monitor visits from \u003Cstrong>GPTBot, ClaudeBot, Google-Extended, PerplexityBot\u003C\u002Fstrong>, and 73+ known AI crawlers, then choose which bots to monitor, discourage through robots.txt, or block with a PHP 403 response.\u003C\u002Fp>\n\u003Cp>The plugin combines \u003Cstrong>AI crawler analytics\u003C\u002Fstrong>, \u003Cstrong>AI bot control\u003C\u002Fstrong>, and \u003Cstrong>AI discoverability tools\u003C\u002Fstrong> in one WordPress dashboard. It includes an AI Visibility Score, \u003Ccode>llms.txt\u003C\u002Fcode> and \u003Ccode>llms-full.txt\u003C\u002Fcode>, Markdown for Agents, JSON-LD, downloadable reports, email alerts, and an authenticated REST API.\u003C\u002Fp>\n\u003Cp>It works alongside SEO and security plugins. It does not replace traditional SEO, a firewall, or server-level bot protection.\u003C\u002Fp>\n\u003Ch4>AI SEO, GEO, and AEO for WordPress\u003C\u002Fh4>\n\u003Cp>Traditional SEO helps search engines index a site. \u003Cstrong>AI SEO\u003C\u002Fstrong>, \u003Cstrong>Generative Engine Optimization (GEO)\u003C\u002Fstrong>, and \u003Cstrong>Answer Engine Optimization (AEO)\u003C\u002Fstrong> focus on making content easier for AI crawlers and answer engines to discover and understand.\u003C\u002Fp>\n\u003Cp>This plugin supports the technical foundations of AI visibility by helping you:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Measure which AI crawlers access your content and which pages they request.\u003C\u002Fli>\n\u003Cli>Check discovery, structured-data, and content-readiness signals.\u003C\u002Fli>\n\u003Cli>Publish AI-readable discovery files and Markdown versions of public content.\u003C\u002Fli>\n\u003Cli>Provide machine-readable site identity and article information with JSON-LD.\u003C\u002Fli>\n\u003Cli>Control crawler access without changing the content human visitors see.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>No plugin can guarantee inclusion, citations, rankings, or traffic from ChatGPT, Claude, Gemini, Perplexity, or other AI services.\u003C\u002Fp>\n\u003Ch4>Why monitor AI crawlers on WordPress?\u003C\u002Fh4>\n\u003Cp>AI companies use automated \u003Cstrong>AI crawlers\u003C\u002Fstrong> to index, train on, or retrieve website content. Common examples include \u003Cstrong>GPTBot\u003C\u002Fstrong> (OpenAI), \u003Cstrong>ClaudeBot\u003C\u002Fstrong> (Anthropic), \u003Cstrong>Google-Extended\u003C\u002Fstrong> (Google), \u003Cstrong>Bytespider\u003C\u002Fstrong> (ByteDance), \u003Cstrong>CCBot\u003C\u002Fstrong> (Common Crawl), \u003Cstrong>PerplexityBot\u003C\u002Fstrong>, and many others.\u003C\u002Fp>\n\u003Cp>With this AI crawlers plugin you can:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>See which AI crawlers visit your site and which pages they request.\u003C\u002Fli>\n\u003Cli>Compare AI bot traffic across date ranges and spot trends over time.\u003C\u002Fli>\n\u003Cli>Block AI crawlers with robots.txt rules or PHP-level 403 responses.\u003C\u002Fli>\n\u003Cli>Keep logs for compliance, debugging, and content licensing decisions.\u003C\u002Fli>\n\u003Cli>See a 7-day crawler summary on the WordPress admin dashboard.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>How it works\u003C\u002Fh4>\n\u003Col>\n\u003Cli>Enable crawler tracking and choose how long activity logs should be retained.\u003C\u002Fli>\n\u003Cli>Review crawler trends, top bots, most-crawled pages, and recent requests.\u003C\u002Fli>\n\u003Cli>Use robots.txt or optional PHP blocking for crawlers you do not want accessing your content.\u003C\u002Fli>\n\u003Cli>Run an AI Visibility Score scan and enable the discoverability features that fit your site.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch4>AI crawler analytics dashboard\u003C\u002Fh4>\n\u003Cp>Understand which AI systems are interested in your site instead of relying on guesswork.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Dashboard with total requests, unique AI crawlers, pages crawled, and blocked attempts.\u003C\u002Fli>\n\u003Cli>Charts for AI crawler requests over time and traffic by bot category.\u003C\u002Fli>\n\u003Cli>Top AI crawlers and most-crawled pages tables with period comparison.\u003C\u002Fli>\n\u003Cli>Recent AI crawler activity log with filters for bot, HTTP status, and content type.\u003C\u002Fli>\n\u003Cli>Download CSV analytics reports for preset or custom date ranges.\u003C\u002Fli>\n\u003Cli>Blocked-attempt logging when PHP blocking is enabled.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>AI Visibility Score\u003C\u002Fh4>\n\u003Cp>Find technical signals that can limit AI discovery and get prioritized recommendations for improvement.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Analyze any page or your entire site for AI readiness.\u003C\u002Fli>\n\u003Cli>26 automated checks across discovery, structured data, content structure, and technical signals.\u003C\u002Fli>\n\u003Cli>0–100 score with letter grade and prioritized recommendations.\u003C\u002Fli>\n\u003Cli>Site-wide batch scan, scan history, comparison diff, and PDF export.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>AI discoverability: llms.txt, Markdown, and JSON-LD\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>Auto-generate \u003Ccode>\u002Fllms.txt\u003C\u002Fcode> and optional \u003Ccode>\u002Fllms-full.txt\u003C\u002Fcode> for AI systems.\u003C\u002Fli>\n\u003Cli>Choose public post types, set URL limits, and include or exclude individual content.\u003C\u002Fli>\n\u003Cli>Exclude content by URL pattern and automatically exclude noindex content.\u003C\u002Fli>\n\u003Cli>Respect noindex metadata from Yoast SEO, Rank Math, All in One SEO, SEOPress, and Genesis.\u003C\u002Fli>\n\u003Cli>Regenerate discovery files when content is published or on a schedule.\u003C\u002Fli>\n\u003Cli>Optionally add \u003Ccode>LLMs:\u003C\u002Fcode> and \u003Ccode>LLMs-full:\u003C\u002Fcode> references to robots.txt.\u003C\u002Fli>\n\u003Cli>Serve \u003Ccode>.md\u003C\u002Fcode> URL endpoints for public content and taxonomy archives.\u003C\u002Fli>\n\u003Cli>Respond to \u003Ccode>Accept: text\u002Fmarkdown\u003C\u002Fcode> requests from compatible AI agents.\u003C\u002Fli>\n\u003Cli>Output Organization, WebSite, and optional Article JSON-LD with AI discovery pointers.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>73+ built-in AI crawlers\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>Pre-configured detection for major AI, search, and scraper bots.\u003C\u002Fli>\n\u003Cli>Category labels (training, search, assistant, scraper, and more).\u003C\u002Fli>\n\u003Cli>Per-crawler info with user-agent pattern details.\u003C\u002Fli>\n\u003Cli>Enable or disable monitoring for individual AI crawlers.\u003C\u002Fli>\n\u003Cli>Includes crawlers associated with OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, Apple, ByteDance, Common Crawl, and others.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>Custom AI crawler detection\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>Add your own user-agent patterns for bots not in the built-in list.\u003C\u002Fli>\n\u003Cli>Assign company name and category for consistent reporting.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>robots.txt control for AI crawlers\u003C\u002Fh4>\n\u003Cp>Publish clear crawler preferences and identify bots that continue visiting after being disallowed.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Add Disallow rules for selected AI crawlers.\u003C\u002Fli>\n\u003Cli>Bulk templates to disallow or allow entire categories (training, scraper, and more).\u003C\u002Fli>\n\u003Cli>Compliance monitor shows top non-compliant crawlers that still visit.\u003C\u002Fli>\n\u003Cli>Rules are appended to your WordPress robots.txt output.\u003C\u002Fli>\n\u003Cli>Live preview before you publish changes.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>Block AI crawlers with PHP (403)\u003C\u002Fh4>\n\u003Cp>robots.txt communicates a preference; PHP blocking enforces access control at the WordPress application level.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Hard-block selected AI crawlers before WordPress serves a response.\u003C\u002Fli>\n\u003Cli>Block specific IP addresses with the same 403 PHP response.\u003C\u002Fli>\n\u003Cli>Customize the plain-text message returned to blocked bots (since 1.1.0).\u003C\u002Fli>\n\u003Cli>Blocked requests are logged in the AI crawler analytics dashboard.\u003C\u002Fli>\n\u003Cli>Works alongside robots.txt for stronger enforcement when needed.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>Data management\u003C\u002Fh4>\n\u003Cp>Crawler activity is stored in your WordPress database, giving you control over retention and deletion.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Configurable log retention (7 days to 1 year, or keep all data).\u003C\u002Fli>\n\u003Cli>Optional delete-all-data action from the admin.\u003C\u002Fli>\n\u003Cli>Delete plugin data on uninstall (optional setting).\u003C\u002Fli>\n\u003Cli>Multi-select crawler, category, HTTP status, and content-type activity filters.\u003C\u002Fli>\n\u003Cli>Download reports for preset or custom date ranges.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>Reports and integrations\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>Daily, weekly, or monthly HTML email reports with configurable recipients and alert types.\u003C\u002Fli>\n\u003Cli>Email summaries cover crawler activity, traffic changes, blocked attempts, new crawlers, and robots.txt non-compliance.\u003C\u002Fli>\n\u003Cli>Test email delivery and view the next scheduled report from the Reports page.\u003C\u002Fli>\n\u003Cli>Authenticated REST API for analytics summaries, recent activity, CSV exports, visibility scans, and blocked attempts.\u003C\u002Fli>\n\u003Cli>Professional report emails include key metrics, the top crawler, and most-crawled pages.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>Clean admin experience\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>WordPress admin menu: \u003Cstrong>AI Crawlers\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>Redesigned UI with grouped sidebar navigation and dark mode.\u003C\u002Fli>\n\u003Cli>Dashboard, Crawlers, Control, Visibility Score, Reports, and Settings in one place.\u003C\u002Fli>\n\u003Cli>Built for site administrators — no code required.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>Who is this plugin for?\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>\u003Cstrong>Bloggers and publishers\u003C\u002Fstrong> who want to know when AI crawlers access their articles.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Business and marketing sites\u003C\u002Fstrong> tracking AI bot traffic to key landing pages.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Developers and agencies\u003C\u002Fstrong> managing AI crawler policy across client WordPress sites.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Site owners\u003C\u002Fstrong> exploring robots.txt and PHP options for AI crawler control.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch4>Privacy and data ownership\u003C\u002Fh4>\n\u003Cul>\n\u003Cli>AI crawler activity logs are stored locally in your WordPress database.\u003C\u002Fli>\n\u003Cli>The analytics dashboard tracks detected crawler requests, not general human visitor analytics.\u003C\u002Fli>\n\u003Cli>Log retention and deletion are controlled from the plugin settings.\u003C\u002Fli>\n\u003Cli>Scheduled reports are sent through the standard WordPress \u003Ccode>wp_mail()\u003C\u002Fcode> function.\u003C\u002Fli>\n\u003Cli>Core crawler monitoring does not require a separate analytics account.\u003C\u002Fli>\n\u003C\u002Ful>\n","Monitor 73+ AI crawlers, analyze bot traffic, manage robots.txt, block unwanted bots, and improve AI visibility with llms.txt and readiness checks.",246,"2026-07-21T08:34:00.000Z",[54,72,55,21,22],"ai-seo","https:\u002F\u002Fplugins.miniorange.com\u002F","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Fminiorange-ai-visibility.zip",{"slug":76,"name":77,"version":78,"author":79,"author_profile":80,"description":81,"short_description":82,"active_installs":83,"downloaded":84,"rating":85,"num_ratings":86,"last_updated":87,"tested_up_to":14,"requires_at_least":88,"requires_php":89,"tags":90,"homepage":23,"download_link":93,"security_score":25,"vuln_count":11,"unpatched_count":11,"last_vuln_date":26,"fetched_at":27},"block-ai-crawlers","Block AI Crawlers","1.5.8","lastsplash (a11n)","https:\u002F\u002Fprofiles.wordpress.org\u002Flastsplash\u002F","\u003Cp>Protect Your Content from AI Scraping\u003C\u002Fp>\n\u003Cp>This plugin helps you prevent AI crawlers from using your content as training data for their products. By updating your site’s \u003Ccode>robots.txt\u003C\u002Fcode>, it blocks common AI crawlers and scrapers, aiming to protect your content from being used in the training of Large Language Models (LLMs).\u003C\u002Fp>\n\u003Ch3>Features\u003C\u002Fh3>\n\u003Ch3>Blocks AI Crawlers\u003C\u002Fh3>\n\u003Cp>Includes:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>OpenAI\u003C\u002Fstrong> – Blocks crawlers used for ChatGPT\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Google\u003C\u002Fstrong> – Blocks crawlers used by Google’s Gemini AI products\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Meta\u003C\u002Fstrong> – Blocks FacebookBot and Meta training crawlers\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Anthropic\u003C\u002Fstrong> – Blocks crawlers used by Claude\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Perplexity\u003C\u002Fstrong> – Blocks crawlers used by Perplexity\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Apple\u003C\u002Fstrong> – Blocks Applebot-Extended\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Amazon\u003C\u002Fstrong> – Blocks Amazonbot\u003C\u002Fli>\n\u003Cli>…and 150+ more via ai.robots.txt\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The blocked crawler list is generated at build time from \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fai-robots-txt\u002Fai.robots.txt\" rel=\"nofollow ugc\">ai.robots.txt\u003C\u002Fa> (\u003Ccode>robots.json\u003C\u002Fcode>, MIT). See the plugin’s \u003Ccode>THIRD-PARTY.md\u003C\u002Fcode> for license attribution.\u003C\u002Fp>\n\u003Ch3>Experimental Meta Tags\u003C\u002Fh3>\n\u003Cp>The plugin adds the “noai, noimageai” directive to your site’s meta tags, instructing AI bots not to use your content in their datasets. Please note that these tags are experimental and have not been standardized.\u003C\u002Fp>\n\u003Ch3>Custom robots.txt Rules\u003C\u002Fh3>\n\u003Cp>Have custom entries for your robots.txt file? You can now add them directly through the plugin!\u003C\u002Fp>\n\u003Ch3>Usage\u003C\u002Fh3>\n\u003Cp>After activation, the plugin will automatically update your \u003Ccode>robots.txt\u003C\u002Fcode> and add the necessary meta tags. No further configuration is required, but you can check the settings page for a full list of blocked crawlers.\u003C\u002Fp>\n\u003Ch3>Limitations\u003C\u002Fh3>\n\u003Cp>While this plugin aims to block specified crawlers, it cannot guarantee complete protection against all forms of scraping, as some bots may disregard \u003Ccode>robots.txt\u003C\u002Fcode> directives.\u003C\u002Fp>\n\u003Ch3>Support\u003C\u002Fh3>\n\u003Cp>For questions or support, \u003Ca href=\"https:\u002F\u002Fwordpress.org\u002Fsupport\u002Fplugin\u002Fblock-ai-crawlers\u002F\" rel=\"ugc\">please post on the forums\u003C\u002Fa>.\u003C\u002Fp>\n","Tell AI (Artificial Intelligence) companies not to scrape your site for their AI products.",1000,17841,92,7,"2026-07-18T12:43:00.000Z","6.9","8.2",[18,91,92,22],"chatgpt","crawlers","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Fblock-ai-crawlers.1.5.8.zip",{"slug":95,"name":96,"version":97,"author":98,"author_profile":99,"description":100,"short_description":101,"active_installs":102,"downloaded":103,"rating":25,"num_ratings":104,"last_updated":105,"tested_up_to":14,"requires_at_least":106,"requires_php":16,"tags":107,"homepage":109,"download_link":110,"security_score":25,"vuln_count":11,"unpatched_count":11,"last_vuln_date":26,"fetched_at":27},"ai-content-signals","AI Content Signals","1.3.0","Fernando Tellado","https:\u002F\u002Fprofiles.wordpress.org\u002Ffernandot\u002F","\u003Cp>AI Content Signals lets you declare, in a machine-readable way, how AI systems may use your content: for search indexing, real-time AI answers (RAG), or model training. It started with Cloudflare’s Content Signals in robots.txt and now expresses the same preferences across several surfaces, so more crawlers and tools can read them:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>robots.txt\u003C\u002Fstrong> — Cloudflare Content Signals (search \u002F ai-input \u002F ai-train)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>HTTP header\u003C\u002Fstrong> — \u003Ccode>X-Robots-Tag: noai, noimageai\u003C\u002Fcode>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>HTML meta\u003C\u002Fstrong> — robots \u003Ccode>noai, noimageai\u003C\u002Fcode>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>\u002F.well-known\u002Ftdmrep.json\u003C\u002Fstrong> — W3C Text and Data Mining Reservation Protocol\u003C\u002Fli>\n\u003Cli>\u003Cstrong>EU Directive 2019\u002F790\u003C\u002Fstrong> rights reservation\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Everything is opt-in and does not change your robots.txt output unless you enable it.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>The signals you control\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>You set three preferences, and AI Content Signals expresses each one in the right format on every surface you enable:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>search\u003C\u002Fstrong> — allow or deny search indexing and traditional search results (links and short snippets)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>ai-input\u003C\u002Fstrong> — allow or deny using your content for real-time AI answers (RAG, grounding, AI Overviews)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>ai-train\u003C\u002Fstrong> — allow or deny using your content to train or fine-tune AI models\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>These three come from Cloudflare’s Content Signals vocabulary, written to your robots.txt. When you deny AI training, the same opt-out is also emitted on any other surface you enable: \u003Ccode>noai, noimageai\u003C\u002Fcode> in an HTML meta tag and an \u003Ccode>X-Robots-Tag\u003C\u002Fcode> header, plus a \u003Ccode>tdm-reservation\u003C\u002Fcode> in your TDMRep manifest. Declaring the same preference in several places means a crawler that ignores one signal may still honor another.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Key Features\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Easy-to-use settings page in WordPress admin\u003C\u002Fli>\n\u003Cli>Set global defaults for all crawlers\u003C\u002Fli>\n\u003Cli>Configure specific settings for individual AI bots (GPTBot, ClaudeBot, PerplexityBot, etc.)\u003C\u002Fli>\n\u003Cli>Add custom bot User-Agents\u003C\u002Fli>\n\u003Cli>Supports both physical and virtual robots.txt files\u003C\u002Fli>\n\u003Cli>Option to create physical robots.txt with basic WordPress rules\u003C\u002Fli>\n\u003Cli>Preview generated Content Signals before applying\u003C\u002Fli>\n\u003Cli>Export and import settings as JSON for easy migration between sites\u003C\u002Fli>\n\u003Cli>Optional legal text with EU Directive reference\u003C\u002Fli>\n\u003Cli>Developer-friendly: filter hook to extend the predefined bots list\u003C\u002Fli>\n\u003Cli>Works with existing robots.txt from SEO plugins\u003C\u002Fli>\n\u003Cli>Automatic sitemap detection and inclusion\u003C\u002Fli>\n\u003Cli>Optional extra output surfaces: HTML robots meta tag (noai, noimageai), X-Robots-Tag header, and a W3C TDMRep file at \u002F.well-known\u002Ftdmrep.json\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>Supported Bots\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>The plugin includes predefined settings for 28 major AI crawlers:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>OpenAI GPTBot, OAI-SearchBot, and ChatGPT-User\u003C\u002Fli>\n\u003Cli>Anthropic ClaudeBot, Claude-Web, and anthropic-ai\u003C\u002Fli>\n\u003Cli>Perplexity Bot and Perplexity-User\u003C\u002Fli>\n\u003Cli>Google Extended (Gemini) and GoogleOther\u003C\u002Fli>\n\u003Cli>Amazon Bot\u003C\u002Fli>\n\u003Cli>Apple Extended\u003C\u002Fli>\n\u003Cli>Meta\u002FFacebook Bot and meta-externalagent\u003C\u002Fli>\n\u003Cli>DuckDuckGo DuckAssistBot\u003C\u002Fli>\n\u003Cli>Allen Institute AI2Bot\u003C\u002Fli>\n\u003Cli>Mistral AI\u003C\u002Fli>\n\u003Cli>ByteDance Bytespider\u003C\u002Fli>\n\u003Cli>DeepSeek AI\u003C\u002Fli>\n\u003Cli>xAI Grok\u003C\u002Fli>\n\u003Cli>Huawei Pangu\u003C\u002Fli>\n\u003Cli>Common Crawl, Cohere AI, Diffbot, You.com Bot, and more\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>Important Notice\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Content Signals is a declarative standard – it expresses your preferences but does not technically enforce them. AI companies are not legally required to respect these signals, though the plugin includes legal text referencing EU copyright directives.\u003C\u002Fp>\n\u003Cp>The IETF AI Preferences (AIPREF) Working Group is currently developing a formal standard based on similar concepts. This plugin implements the current Cloudflare Content Signals specification and will be updated as standards evolve.\u003C\u002Fp>\n\u003Cp>This plugin works best when combined with other protection measures like traditional robots.txt rules and server-level bot management.\u003C\u002Fp>\n\u003Ch3>Support\u003C\u002Fh3>\n\u003Cp>Need private support or custom development?\u003C\u002Fp>\n\u003Cp>Do you need one-on-one help, priority troubleshooting, or a custom feature, integration, or tweak built specifically for your site? I offer private support and custom development. Just \u003Ca href=\"mailto:ai-content-signals@ayudawp.com\" rel=\"nofollow ugc\">contact me\u003C\u002Fa> and tell me what you need.\u003C\u002Fp>\n\u003Cp>Need help or have suggestions?\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fservicios.ayudawp.com\" rel=\"nofollow ugc\">Official website\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwordpress.org\u002Fsupport\u002Fplugin\u002Fai-content-signals\u002F\" rel=\"ugc\">WordPress support forum\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwww.youtube.com\u002FAyudaWordPressES\" rel=\"nofollow ugc\">YouTube channel\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fayudawp.com\" rel=\"nofollow ugc\">Documentation and tutorials\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Love the plugin? Please leave us a \u003Ca href=\"https:\u002F\u002Fwordpress.org\u002Fsupport\u002Fplugin\u002Fai-content-signals\u002Freviews\u002F#new-post\" rel=\"ugc\">5-star review\u003C\u002Fa> and help spread the word!\u003C\u002Fp>\n\u003Ch3>About AyudaWP.com\u003C\u002Fh3>\n\u003Cp>We are specialists in WordPress security, SEO, and performance optimization plugins. We create tools that solve real problems for WordPress site owners while maintaining the highest coding standards and accessibility requirements.\u003C\u002Fp>\n","Control how AI uses your content — search, AI answers or training — via robots.txt Content Signals, HTTP headers, meta tags and TDMRep.",500,3221,3,"2026-06-16T17:57:00.000Z","5.6",[18,108,92,22,56],"cloudflare","https:\u002F\u002Fservicios.ayudawp.com","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Fai-content-signals.1.3.0.zip",{"slug":112,"name":113,"version":114,"author":115,"author_profile":116,"description":117,"short_description":118,"active_installs":119,"downloaded":120,"rating":25,"num_ratings":121,"last_updated":122,"tested_up_to":123,"requires_at_least":15,"requires_php":23,"tags":124,"homepage":127,"download_link":128,"security_score":25,"vuln_count":11,"unpatched_count":11,"last_vuln_date":26,"fetched_at":27},"ai-deny","AI Deny","0.1.1","Code For Broke","https:\u002F\u002Fprofiles.wordpress.org\u002Fcodeforbroke\u002F","\u003Cp>Blocking AI crawlers from your website is essential for maintaining both its performance and privacy. When left unchecked, these bots can strain server resources, causing slow load times for real users, and may harvest data beyond your intended audience. This creates potential risks, including the exposure of sensitive information and a compromised user experience.\u003C\u002Fp>\n\u003Cp>AI Deny offers an effective and straightforward solution to this problem. The plugin works by automatically configuring your website’s robots.txt file to disallow over 30 of the most common AI user-agents known for crawling websites. This ensures that unwanted bots are prevented from accessing your site, reducing server load and safeguarding privacy.\u003C\u002Fp>\n\u003Cp>With AI Deny, you can easily block all pre-configured user-agents or customize the list to suit your specific needs. By giving you control over which bots are allowed or denied access, AI Deny helps you protect your content, conserve bandwidth, and deliver a faster, more secure experience for your audience.\u003C\u002Fp>\n\u003Cp>AI Deny\u003C\u002Fp>\n","Easily block over 30 of the most common AI user-agents known for crawling websites, keeping unauthorized bots at bay.",40,1416,1,"2026-01-09T19:19:00.000Z","6.9.5",[18,125,20,126,22],"bots","robots","https:\u002F\u002Fgithub.com\u002Fcodeforbroke\u002Fai-deny","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Fai-deny.0.1.1.zip",{"error":130,"url":131,"statusCode":132,"statusMessage":133,"message":133},true,"http:\u002F\u002Flocalhost\u002Fapi\u002Fplugins\u002Ffocusweb-ai-crawler-monitor\u002Fbundle",404,"no bundle for this plugin yet",{"slug":4,"current_version":6,"total_versions":135,"versions":136},5,[137,143,150,157,164],{"version":6,"download_url":24,"svn_tag_url":138,"released_at":26,"has_diff":139,"diff_files_changed":140,"diff_lines":26,"trac_diff_url":141,"vulnerabilities":142,"is_current":130},"https:\u002F\u002Fplugins.svn.wordpress.org\u002Ffocusweb-ai-crawler-monitor\u002Ftags\u002F2.0.0\u002F",false,[],"https:\u002F\u002Fplugins.trac.wordpress.org\u002Fchangeset?old_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F1.4.0&new_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F2.0.0",[],{"version":144,"download_url":145,"svn_tag_url":146,"released_at":26,"has_diff":139,"diff_files_changed":147,"diff_lines":26,"trac_diff_url":148,"vulnerabilities":149,"is_current":139},"1.4.0","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Ffocusweb-ai-crawler-monitor.1.4.0.zip","https:\u002F\u002Fplugins.svn.wordpress.org\u002Ffocusweb-ai-crawler-monitor\u002Ftags\u002F1.4.0\u002F",[],"https:\u002F\u002Fplugins.trac.wordpress.org\u002Fchangeset?old_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F1.2.4&new_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F1.4.0",[],{"version":151,"download_url":152,"svn_tag_url":153,"released_at":26,"has_diff":139,"diff_files_changed":154,"diff_lines":26,"trac_diff_url":155,"vulnerabilities":156,"is_current":139},"1.2.4","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Ffocusweb-ai-crawler-monitor.1.2.4.zip","https:\u002F\u002Fplugins.svn.wordpress.org\u002Ffocusweb-ai-crawler-monitor\u002Ftags\u002F1.2.4\u002F",[],"https:\u002F\u002Fplugins.trac.wordpress.org\u002Fchangeset?old_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F1.2.3&new_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F1.2.4",[],{"version":158,"download_url":159,"svn_tag_url":160,"released_at":26,"has_diff":139,"diff_files_changed":161,"diff_lines":26,"trac_diff_url":162,"vulnerabilities":163,"is_current":139},"1.2.3","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Ffocusweb-ai-crawler-monitor.1.2.3.zip","https:\u002F\u002Fplugins.svn.wordpress.org\u002Ffocusweb-ai-crawler-monitor\u002Ftags\u002F1.2.3\u002F",[],"https:\u002F\u002Fplugins.trac.wordpress.org\u002Fchangeset?old_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F1.2.2&new_path=%2Ffocusweb-ai-crawler-monitor%2Ftags%2F1.2.3",[],{"version":165,"download_url":166,"svn_tag_url":167,"released_at":26,"has_diff":139,"diff_files_changed":168,"diff_lines":26,"trac_diff_url":26,"vulnerabilities":169,"is_current":139},"1.2.2","https:\u002F\u002Fdownloads.wordpress.org\u002Fplugin\u002Ffocusweb-ai-crawler-monitor.1.2.2.zip","https:\u002F\u002Fplugins.svn.wordpress.org\u002Ffocusweb-ai-crawler-monitor\u002Ftags\u002F1.2.2\u002F",[],[]]