{"id":31513,"date":"2026-10-26T20:58:48","date_gmt":"2026-10-26T19:58:48","guid":{"rendered":"https:\/\/wooacademy.sk\/produkt\/ai-crawler-policy\/"},"modified":"2026-10-26T21:15:11","modified_gmt":"2026-10-26T20:15:11","slug":"ai-crawler-policy","status":"publish","type":"product","link":"https:\/\/wooacademy.sk\/en\/produkt\/ai-crawler-policy\/","title":{"rendered":"AI Crawler Policy (robots.txt for GPTBot, ClaudeBot and Others)"},"content":{"rendered":"<div class=\"wa-seo\">\n<p class=\"wa-lead\"><strong>An AI crawler policy is a clear decision about which AI bots may read your website, set up correctly in robots.txt, your firewall and your CDN.<\/strong> Most sites either leave everything open without anyone deciding, or block every AI bot out of fear and then wonder why ChatGPT and Perplexity never cite them. Yet training and AI search are two different things. From \u20ac190.<\/p>\n<ul class=\"wa-facts\">\n<li>Basic: \u20ac190<\/li>\n<li>With access log verification: \u20ac390<\/li>\n<li>Training vs. AI search<\/li>\n<li>Robots.txt, firewall, CDN<\/li>\n<li>WordPress and other sites<\/li>\n<li>Done within 3 business days<\/li>\n<\/ul>\n<div class=\"wa-btns\"><a class=\"wa-btn\" href=\"https:\/\/wooacademy.sk\/kontakt\/\">Set up AI bot access<\/a><a class=\"wa-btn o\" href=\"tel:+421940634491\">+421 940 634 491<\/a><\/div>\n<h2>Why websites get AI bot blocking wrong<em>?<\/em><\/h2>\n<p><strong>AI crawlers aren&#8217;t one bot doing one thing.<\/strong> Some collect text to train models, others fetch sources for an answer the AI is writing for a user right now. Lump them together and you often block exactly the ones that bring you citations. This is what I see most often:<\/p>\n<div class=\"wa-3 wa-swipe\">\n<div class=\"wa-card\">\n<h3>Everything blocked<\/h3>\n<p>A robots.txt copied from some article also blocks AI search bots. The site then never shows up in ChatGPT or Perplexity answers, even where it belongs.<\/p>\n<\/div>\n<div class=\"wa-card\">\n<h3>Everything open, nobody decided<\/h3>\n<p>Nobody decided whether content may be used for model training. For paid content, courses or proprietary data, that&#8217;s a business question, not a technical one.<\/p>\n<\/div>\n<div class=\"wa-card\">\n<h3>The firewall blocks quietly<\/h3>\n<p>Robots.txt says yes, but Cloudflare or a security plugin cuts the AI bots off. Nobody knows until they look at the logs.<\/p>\n<\/div>\n<div class=\"wa-card\">\n<h3>Google-Extended as a bogeyman<\/h3>\n<p>Sites block Google-Extended thinking they&#8217;ll disappear from AI Overviews. They won&#8217;t. That token covers Gemini training, not how you appear in Search.<\/p>\n<\/div>\n<div class=\"wa-card\">\n<h3>Fake bots<\/h3>\n<p>Bots that only pretend to be GPTBot or ClaudeBot show up in the logs. Without checking IP addresses you can&#8217;t tell who&#8217;s who.<\/p>\n<\/div>\n<div class=\"wa-card\">\n<h3>Rules nobody checks<\/h3>\n<p>Robots.txt is voluntary. If you don&#8217;t check the access log to see whether bots follow it, you&#8217;re just hoping.<\/p>\n<\/div>\n<\/div>\n<p class=\"wa-mt\"><strong>What others get wrong:<\/strong> they paste a list of 40 AI bots from some GitHub repo into robots.txt with &#8220;Disallow: \/&#8221; for all of them. That also blocks citations in AI search, and the company has no idea. The right way is to decide first what you want (citations yes, training yes or no) and only then write the rules.<\/p>\n<h2>What I do for AI crawlers<\/h2>\n<div class=\"wa-4 wa-swipe\">\n<div class=\"wa-card c\"><span class=\"wa-n\">1<\/span><\/p>\n<h3>A 30-minute decision<\/h3>\n<p>We go through what you want: appearing in AI answers, allowing or blocking training, protecting paid content. The decision is yours; I explain the consequences.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">2<\/span><\/p>\n<h3>Training bots<\/h3>\n<p>GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended tokens, according to your decision.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">3<\/span><\/p>\n<h3>AI search bots<\/h3>\n<p>OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User stay open if you want to be cited.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">4<\/span><\/p>\n<h3>A new robots.txt<\/h3>\n<p>Clean rules with comments, without blocking Googlebot, CSS or JS. On WordPress via Rank Math or the file itself.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">5<\/span><\/p>\n<h3>Firewall and CDN<\/h3>\n<p>Checking Cloudflare, Wordfence and your hosting so they don&#8217;t block the bots you want to allow, and vice versa.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">6<\/span><\/p>\n<h3>Paid and protected content<\/h3>\n<p>I separate member areas, courses and files so they don&#8217;t end up where they shouldn&#8217;t.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">7<\/span><\/p>\n<h3>Access log verification<\/h3>\n<p>In the \u20ac390 package: which AI bots visit, which pages they read, whether they follow the rules and which ones are fake.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">8<\/span><\/p>\n<h3>Documentation<\/h3>\n<p>A short document on what&#8217;s set up and why, so your admin doesn&#8217;t break it with the next change.<\/p>\n<\/div>\n<\/div>\n<div class=\"wa-4 wa-sh wa-mt wa-swipe wa-swipe-s\">\n<div class=\"wa-v s\"><iframe width=\"315\" height=\"560\" loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/bhmVWiqSuxQ\" title=\"Did I outrank you on Google and in AI? Proof your SEO and GEO aren&#x27;t working\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<div class=\"wa-v s\"><iframe width=\"315\" height=\"560\" loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/iHMCSKgXenw\" title=\"Website speed as an SEO factor - how they lie\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<div class=\"wa-v s\"><iframe width=\"315\" height=\"560\" loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/ijqJnrY5MzA\" title=\"How to check your search rankings for free\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<div class=\"wa-v s\"><iframe width=\"315\" height=\"560\" loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/YBS-TqkM62I\" title=\"SEO CMS - which CMS is best for SEO? (Why WordPress wins)\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<\/div>\n<p class=\"wa-mt\">The videos are in Slovak; YouTube can show automatic English subtitles.<\/p>\n<h2>AI crawler setup pricing<\/h2>\n<table class=\"wa-tbl\">\n<thead>\n<tr>\n<th>Package<\/th>\n<th>Who it&#8217;s for<\/th>\n<th>Price<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"hl\">\n<td data-label=\"Package\"><strong>Basic<\/strong><\/td>\n<td data-label=\"Who it's for\">Decision, new robots.txt, firewall and CDN check, documentation. Most popular for company websites.<\/td>\n<td data-label=\"Price\">\u20ac190<\/td>\n<\/tr>\n<tr>\n<td data-label=\"Package\"><strong>With access log verification<\/strong><\/td>\n<td data-label=\"Who it's for\">Everything in Basic plus an access log analysis over recent weeks: actual AI bot visits, pages read, rule compliance and fake bots exposed.<\/td>\n<td data-label=\"Price\">\u20ac390<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p class=\"wa-mt\">Prices are final; we are not VAT registered. Done within 3 business days of receiving access. For access log verification I need at least 2 weeks of logs, ideally a month.<\/p>\n<div class=\"wa-tip\"><strong class=\"t\">\u2b50 What&#8217;s not included<\/strong>Changes that increase your chances of being cited by AI (content, structure, brand entity) are separate work. The llms.txt file is covered by <a href=\"https:\/\/wooacademy.sk\/produkt\/llms-txt-implementacia-pre-ai-seo-dohladatelnost\/\">llms.txt implementation<\/a>, and the bigger picture of how AI sees you by the article on <a href=\"https:\/\/wooacademy.sk\/geo-optimalizacia-ai-vyhladavanie\/\">GEO optimization<\/a> (both in Slovak). A detailed analysis of all bots, not just AI, is the <a href=\"https:\/\/wooacademy.sk\/en\/produkt\/log-file-analysis-seo\/\">log file analysis for SEO<\/a>.<\/div>\n<div class=\"wa-split wa-mt\">\n<div>\n<h2 class=\"wa-h2top\">How it works<\/h2>\n<p><strong>Short work, but with a clear decision at the start.<\/strong> I need access to your admin or robots.txt, to Cloudflare or your firewall, and for the larger package, access logs from your hosting.<\/p>\n<\/div>\n<div class=\"wa-v\"><iframe width=\"560\" height=\"315\" loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/gACg0OhSAjg\" title=\"What is GEO and why you still need SEO - Milan Fra\u0148o\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<\/div>\n<div class=\"wa-3 wa-steps\">\n<div class=\"wa-card\">\n<p><span class=\"wa-n\">1<\/span><\/p>\n<p><strong>Decision (day 1).<\/strong> A short call or email: what you want to allow, what to block and why. I send a recommendation with the consequences.<\/p>\n<\/div>\n<div class=\"wa-card\">\n<p><span class=\"wa-n\">2<\/span><\/p>\n<p><strong>Setup (day 2).<\/strong> Robots.txt, firewall, CDN, protected sections. Everything tested with a robots.txt testing tool.<\/p>\n<\/div>\n<div class=\"wa-card\">\n<p><span class=\"wa-n\">3<\/span><\/p>\n<p><strong>Check (day 3).<\/strong> Documentation, plus an access log report for the larger package. After 30 days I check that bots follow the rules.<\/p>\n<\/div>\n<\/div>\n<h2>What you get on top<\/h2>\n<div class=\"wa-3\">\n<div class=\"wa-card c\"><span class=\"wa-n\">1<\/span><\/p>\n<h3>Free robots.txt check<\/h3>\n<p>Send your website address and within one business day I&#8217;ll tell you which AI bots you block today and whether that&#8217;s what you intended.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">2<\/span><\/p>\n<h3>30-day follow-up<\/h3>\n<p>I check the logs or bot behavior again and verify the setup works.<\/p>\n<\/div>\n<div class=\"wa-card c\"><span class=\"wa-n\">3<\/span><\/p>\n<h3>Heads-up on new bots<\/h3>\n<p>When a major AI company announces a new bot, I&#8217;ll let you know within a year of setup whether your rules need updating.<\/p>\n<\/div>\n<\/div>\n<h2>Related services<\/h2>\n<table class=\"wa-tbl\">\n<thead>\n<tr>\n<th>Service<\/th>\n<th>When it fits<\/th>\n<th>Price<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"hl\">\n<td data-label=\"Service\"><strong>AI crawler policy<\/strong><\/td>\n<td data-label=\"When it fits\">You want to decide who may read your site<\/td>\n<td data-label=\"Price\">from \u20ac190<\/td>\n<\/tr>\n<tr>\n<td data-label=\"Service\"><a href=\"https:\/\/wooacademy.sk\/en\/produkt\/log-file-analysis-seo\/\">Log file analysis for SEO<\/a><\/td>\n<td data-label=\"When it fits\">You want to see all bots, not just AI<\/td>\n<td data-label=\"Price\">from \u20ac390<\/td>\n<\/tr>\n<tr>\n<td data-label=\"Service\"><a href=\"https:\/\/wooacademy.sk\/produkt\/llms-txt-implementacia-pre-ai-seo-dohladatelnost\/\">llms.txt implementation<\/a><\/td>\n<td data-label=\"When it fits\">You want to show AI what matters on your site (in Slovak)<\/td>\n<td data-label=\"Price\">\u20ac282<\/td>\n<\/tr>\n<tr>\n<td data-label=\"Service\"><a href=\"https:\/\/wooacademy.sk\/en\/produkt\/brand-entity-knowledge-panel\/\">Brand entity and Knowledge Panel<\/a><\/td>\n<td data-label=\"When it fits\">You want AI to know who you are<\/td>\n<td data-label=\"Price\">from \u20ac490<\/td>\n<\/tr>\n<tr>\n<td data-label=\"Service\"><a href=\"https:\/\/wooacademy.sk\/en\/produkt\/wordpress-maintenance\/\">WordPress maintenance<\/a><\/td>\n<td data-label=\"When it fits\">You want someone keeping an eye on it<\/td>\n<td data-label=\"Price\">from \u20ac49 \/ month<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Who does it<\/h2>\n<p><a href=\"https:\/\/wooacademy.sk\/o-nas\/seo-specialista-milan-frano\/\">RNDr. Milan Fra\u0148o, PhD.<\/a>, HEAD of SEO at WooAcademy. I read access logs regularly and write about how AI bots actually crawl websites in the article on <a href=\"https:\/\/wooacademy.sk\/en\/technical-seo-for-ai-search\/\">technical SEO for AI<\/a> (in Slovak). I don&#8217;t recommend that anyone block everything or let everything in blindly.<\/p>\n<div class=\"wa-btns\"><a class=\"wa-btn\" href=\"https:\/\/wooacademy.sk\/kontakt\/\">Free robots.txt check<\/a><a class=\"wa-btn o\" href=\"mailto:info@wooacademy.sk\">info@wooacademy.sk<\/a><\/div>\n<p class=\"wa-mt\">More short videos on SEO and GEO are on the <a href=\"https:\/\/www.youtube.com\/@wooacademy8573\" target=\"_blank\" rel=\"noopener\">WooAcademy YouTube channel<\/a>.<\/p>\n<h2>Frequently asked questions about AI crawlers<\/h2>\n<details>\n<summary>What are AI crawlers?<\/summary>\n<p>Bots from companies that build AI models and AI search. Some collect text for training (for example GPTBot, ClaudeBot, CCBot); others fetch sources for a specific answer (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User).<\/p>\n<\/details>\n<details>\n<summary>How much does an AI crawler setup cost?<\/summary>\n<p>Basic \u20ac190, with access log verification \u20ac390. Prices are final; we are not VAT registered.<\/p>\n<\/details>\n<details>\n<summary>If I block GPTBot, will I disappear from ChatGPT?<\/summary>\n<p>Not necessarily. GPTBot collects training data. Search in ChatGPT relies on OAI-SearchBot, and fetching a page at a user&#8217;s request on ChatGPT-User. So you can block training and keep the citations.<\/p>\n<\/details>\n<details>\n<summary>Will blocking Google-Extended remove me from AI Overviews?<\/summary>\n<p>No. Google-Extended is only a robots.txt token that says whether content may be used for Gemini training and grounding. AI Overviews are part of Google Search and are governed by regular Googlebot.<\/p>\n<\/details>\n<details>\n<summary>Do AI bots respect robots.txt?<\/summary>\n<p>The major companies say they do, but robots.txt is voluntary, and according to their documentation some bots triggered directly by users may not follow it. That&#8217;s why the \u20ac390 package verifies behavior in the access log.<\/p>\n<\/details>\n<details>\n<summary>Should I block AI training?<\/summary>\n<p>It depends on your content. If you sell courses, data or original content, it makes sense. If brand visibility is your main goal, you&#8217;ll often leave training open. The decision is yours; I explain the consequences.<\/p>\n<\/details>\n<details>\n<summary>Will this help me get cited in ChatGPT?<\/summary>\n<p>The setup on its own only ensures AI search can read you. Whether it cites you depends on your content and brand credibility. That&#8217;s covered in the <a href=\"https:\/\/wooacademy.sk\/en\/brand-mentions-ai-citations\/\">article on brand mentions in AI<\/a> (in Slovak).<\/p>\n<\/details>\n<p><em>AI crawler policy is set up by <a href=\"https:\/\/wooacademy.sk\/o-nas\/seo-specialista-milan-frano\/\">Milan Fra\u0148o<\/a>, WooAcademy, Bratislava. Prices valid from October 2026.<\/em><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p><strong>An AI crawler policy is a clear decision about which AI bots may read your website, set up correctly in robots.txt, your firewall and your CDN.<\/strong><\/p>\n<ul>\n<li>Basic: \u20ac190<\/li>\n<li>With access log verification: \u20ac390<\/li>\n<li>Training vs. AI search<\/li>\n<li>Robots.txt, firewall, CDN<\/li>\n<li>WordPress and other sites<\/li>\n<li>Done within 3 business days<\/li>\n<\/ul>\n<p>Send your website address and I&#8217;ll tell you for free which AI bots you block today. <a href=\"https:\/\/wooacademy.sk\/kontakt\/\">Get in touch<\/a> or +421 940 634 491.<\/p>\n","protected":false},"featured_media":16043,"comment_status":"open","ping_status":"closed","template":"","meta":[],"product_brand":[],"product_cat":[864,865],"product_tag":[],"class_list":["post-31513","product","type-product","status-publish","has-post-thumbnail","product_cat-marketingove-sluzby","product_cat-seo-sluzby","has-post-title","has-post-date","has-post-category","has-post-tag","has-post-comment","has-post-author","","first","instock","shipping-taxable","purchasable","product-type-variable","has-default-attributes"],"builder_content":"","_links":{"self":[{"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/product\/31513","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/product"}],"about":[{"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/types\/product"}],"replies":[{"embeddable":true,"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/comments?post=31513"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/media\/16043"}],"wp:attachment":[{"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/media?parent=31513"}],"wp:term":[{"taxonomy":"product_brand","embeddable":true,"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/product_brand?post=31513"},{"taxonomy":"product_cat","embeddable":true,"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/product_cat?post=31513"},{"taxonomy":"product_tag","embeddable":true,"href":"https:\/\/wooacademy.sk\/en\/wp-json\/wp\/v2\/product_tag?post=31513"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}