AI Crawler Policy (robots.txt for GPTBot, ClaudeBot and Others)
An AI crawler policy is a clear decision about which AI bots may read your website, set up correctly in robots.txt, your firewall and your CDN.
- Basic: €190
- With access log verification: €390
- Training vs. AI search
- Robots.txt, firewall, CDN
- WordPress and other sites
- Done within 3 business days
Send your website address and I’ll tell you for free which AI bots you block today. Get in touch or +421 940 634 491.
Price range: 190 € through 390 €
An AI crawler policy is a clear decision about which AI bots may read your website, set up correctly in robots.txt, your firewall and your CDN. Most sites either leave everything open without anyone deciding, or block every AI bot out of fear and then wonder why ChatGPT and Perplexity never cite them. Yet training and AI search are two different things. From €190.
- Basic: €190
- With access log verification: €390
- Training vs. AI search
- Robots.txt, firewall, CDN
- WordPress and other sites
- Done within 3 business days
Why websites get AI bot blocking wrong?
AI crawlers aren’t one bot doing one thing. Some collect text to train models, others fetch sources for an answer the AI is writing for a user right now. Lump them together and you often block exactly the ones that bring you citations. This is what I see most often:
Everything blocked
A robots.txt copied from some article also blocks AI search bots. The site then never shows up in ChatGPT or Perplexity answers, even where it belongs.
Everything open, nobody decided
Nobody decided whether content may be used for model training. For paid content, courses or proprietary data, that’s a business question, not a technical one.
The firewall blocks quietly
Robots.txt says yes, but Cloudflare or a security plugin cuts the AI bots off. Nobody knows until they look at the logs.
Google-Extended as a bogeyman
Sites block Google-Extended thinking they’ll disappear from AI Overviews. They won’t. That token covers Gemini training, not how you appear in Search.
Fake bots
Bots that only pretend to be GPTBot or ClaudeBot show up in the logs. Without checking IP addresses you can’t tell who’s who.
Rules nobody checks
Robots.txt is voluntary. If you don’t check the access log to see whether bots follow it, you’re just hoping.
What others get wrong: they paste a list of 40 AI bots from some GitHub repo into robots.txt with “Disallow: /” for all of them. That also blocks citations in AI search, and the company has no idea. The right way is to decide first what you want (citations yes, training yes or no) and only then write the rules.
What I do for AI crawlers
A 30-minute decision
We go through what you want: appearing in AI answers, allowing or blocking training, protecting paid content. The decision is yours; I explain the consequences.
Training bots
GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended tokens, according to your decision.
AI search bots
OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User stay open if you want to be cited.
A new robots.txt
Clean rules with comments, without blocking Googlebot, CSS or JS. On WordPress via Rank Math or the file itself.
Firewall and CDN
Checking Cloudflare, Wordfence and your hosting so they don’t block the bots you want to allow, and vice versa.
Paid and protected content
I separate member areas, courses and files so they don’t end up where they shouldn’t.
Access log verification
In the €390 package: which AI bots visit, which pages they read, whether they follow the rules and which ones are fake.
Documentation
A short document on what’s set up and why, so your admin doesn’t break it with the next change.
The videos are in Slovak; YouTube can show automatic English subtitles.
AI crawler setup pricing
| Package | Who it’s for | Price |
|---|---|---|
| Basic | Decision, new robots.txt, firewall and CDN check, documentation. Most popular for company websites. | €190 |
| With access log verification | Everything in Basic plus an access log analysis over recent weeks: actual AI bot visits, pages read, rule compliance and fake bots exposed. | €390 |
Prices are final; we are not VAT registered. Done within 3 business days of receiving access. For access log verification I need at least 2 weeks of logs, ideally a month.
How it works
Short work, but with a clear decision at the start. I need access to your admin or robots.txt, to Cloudflare or your firewall, and for the larger package, access logs from your hosting.
1
Decision (day 1). A short call or email: what you want to allow, what to block and why. I send a recommendation with the consequences.
2
Setup (day 2). Robots.txt, firewall, CDN, protected sections. Everything tested with a robots.txt testing tool.
3
Check (day 3). Documentation, plus an access log report for the larger package. After 30 days I check that bots follow the rules.
What you get on top
Free robots.txt check
Send your website address and within one business day I’ll tell you which AI bots you block today and whether that’s what you intended.
30-day follow-up
I check the logs or bot behavior again and verify the setup works.
Heads-up on new bots
When a major AI company announces a new bot, I’ll let you know within a year of setup whether your rules need updating.
Related services
| Service | When it fits | Price |
|---|---|---|
| AI crawler policy | You want to decide who may read your site | from €190 |
| Log file analysis for SEO | You want to see all bots, not just AI | from €390 |
| llms.txt implementation | You want to show AI what matters on your site (in Slovak) | €282 |
| Brand entity and Knowledge Panel | You want AI to know who you are | from €490 |
| WordPress maintenance | You want someone keeping an eye on it | from €49 / month |
Who does it
RNDr. Milan Fraňo, PhD., HEAD of SEO at WooAcademy. I read access logs regularly and write about how AI bots actually crawl websites in the article on technical SEO for AI (in Slovak). I don’t recommend that anyone block everything or let everything in blindly.
More short videos on SEO and GEO are on the WooAcademy YouTube channel.
Frequently asked questions about AI crawlers
What are AI crawlers?
Bots from companies that build AI models and AI search. Some collect text for training (for example GPTBot, ClaudeBot, CCBot); others fetch sources for a specific answer (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User).
How much does an AI crawler setup cost?
Basic €190, with access log verification €390. Prices are final; we are not VAT registered.
If I block GPTBot, will I disappear from ChatGPT?
Not necessarily. GPTBot collects training data. Search in ChatGPT relies on OAI-SearchBot, and fetching a page at a user’s request on ChatGPT-User. So you can block training and keep the citations.
Will blocking Google-Extended remove me from AI Overviews?
No. Google-Extended is only a robots.txt token that says whether content may be used for Gemini training and grounding. AI Overviews are part of Google Search and are governed by regular Googlebot.
Do AI bots respect robots.txt?
The major companies say they do, but robots.txt is voluntary, and according to their documentation some bots triggered directly by users may not follow it. That’s why the €390 package verifies behavior in the access log.
Should I block AI training?
It depends on your content. If you sell courses, data or original content, it makes sense. If brand visibility is your main goal, you’ll often leave training open. The decision is yours; I explain the consequences.
Will this help me get cited in ChatGPT?
The setup on its own only ensures AI search can read you. Whether it cites you depends on your content and brand credibility. That’s covered in the article on brand mentions in AI (in Slovak).
AI crawler policy is set up by Milan Fraňo, WooAcademy, Bratislava. Prices valid from October 2026.
Facebook hodnotenia.
Naplánuj si konzultáciu s našimi profesionálmi.
Google certifikovaná trénerka – Google ads kampane (PPC)
Facebook marketing partner – social marketing
HEAD of SEO a web editor – weby & SEO
Zanechajte nám kontaktné informácie a pripravíme cenovú ponuku na mieru ?
Marketingové služby na mieru, ozvite sa!