Robots.txt Generator
Generate a robots.txt for your hotel website. Toggle which AI crawlers to allow or block. Controls whether ChatGPT, Claude, Perplexity, and other AI engines can access and cite your content.
Quick Presets
Slows down aggressive crawlers. Useful for hotels on shared hosting where crawling impacts booking engine performance. Supported by Bing and Yandex.
AI Crawlers
Allow AI engines to crawl and cite your hotel, or block them to prevent AI training on your content. Bots marked "citation" fetch pages in real-time when users ask questions. Bots marked "training" crawl to build AI model training data.
BytespiderBytespidertraining
ByteDance crawler (TikTok AI training)
Search Engines
# Generated by HotelSignal robots.txt generator
# https://hotelsignal.com/tools/robots-txt-generator
User-agent: *
Disallow: /admin
Disallow: /api
Disallow: /private
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: CCBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: Bytespider
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: cohere-ai
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
User-agent: meta-externalfetcher
Allow: /
Test a URL path
Check if a specific URL on your hotel site would be allowed or blocked by these rules.
Upload this file to the root of your website (e.g., https://your-hotel.com/robots.txt).
Why block (or allow) AI bots for your hotel?
AI bots like GPTBot, ClaudeBot, and CCBot crawl the web to train Large Language Models. For hotels, the decision to allow or block is more nuanced than for other businesses:
AI citations drive direct bookings
When a traveler asks ChatGPT "best hotel in Marrakech for a honeymoon," the AI names 3 to 5 properties. If your hotel is cited, the traveler may visit your site directly, bypassing OTAs and their 15 to 25 percent commission. Allowing AI citation bots (like ChatGPT-User and Perplexity-User) enables this.
Training vs. citation: the key distinction
Training bots (GPTBot, ClaudeBot, CCBot) crawl to build model training data. Citation bots (ChatGPT-User, Perplexity-User, OAI-SearchBot) fetch pages in real-time when a user asks a question. Many hotels want citations but not training. Use the "AI Citations Only" preset for this.
Bandwidth and server load
AI crawlers can be aggressive, especially on hotel sites with many room pages, blog posts, and image galleries. If your site is on shared hosting, consider using crawl-delay or blocking the most aggressive training crawlers to protect booking engine performance.
Robots.txt vs Meta Robots Tag
robots.txt
- Controls crawling access: which bots can fetch pages
- Site-wide rules in a single file
- Cannot prevent indexing. Blocked pages can still appear in search results
- Cannot control link following (nofollow)
- Hotel example: block /admin/, /api/, /booking/checkout
Meta Robots Tag
- Controls indexing: noindex removes pages from search results
- Per-page control with nofollow, nosnippet, noimageindex
- Requires the bot to crawl the page first to read the tag
- Must be added to every page individually
- Hotel example: noindex seasonal promo pages, old event pages
Best practice: Use robots.txt to block crawling of entire sections (admin, staging, API endpoints). Use meta robots to control indexing of individual pages. For complete removal from search results, allow crawling (so the bot reads your noindex tag) but set<meta name="robots" content="noindex">on the page.
Common robots.txt mistakes for hotel websites
Blocking CSS/JS files
Disallowing /assets/ or /_next/ or /static/ prevents Google from rendering your page. Google needs these resources to understand your layout, booking widgets, and image galleries. Never block CSS/JS paths in robots.txt.
Using robots.txt to hide pages
Blocking a URL in robots.txt doesn't remove it from search results. Google can still index it based on links from other sites, OTAs, or travel blogs. Use the noindex meta tag instead for pages you want removed from search results.
Blocking AI citation bots accidentally
Many hotels block "all AI bots" thinking it only stops training. But this also blocks ChatGPT-User and Perplexity-User, which means AI engines cannot cite your hotel in real-time answers. Use the "AI Citations Only" preset to allow citations while blocking training.
Missing sitemap directive
Always include a Sitemap: line pointing to your XML sitemap. This is the simplest way to help search engines discover all your hotel pages, including room types, packages, and blog posts.
Frequently asked questions
Should my hotel allow ChatGPT to crawl our site?
How do I protect my booking engine from aggressive crawlers?
What's the ideal robots.txt for a hotel that wants AI citations but not training?
Can I allow Google AI Overviews but block ChatGPT?
What is the difference between GPTBot and ChatGPT-User?
Where do I put my robots.txt file?
What is Google-Extended?
Learn more
Robots.txt Validator
Check your existing robots.txt for syntax errors, AI signals, and verify bot access.
Meta Tag Checker
Check title, description, and Open Graph tags for your hotel pages. Preview how your hotel appears in search and social.
Schema Validator
Validate hotel structured data (Hotel, LodgingBusiness) for rich results in Google and AI search.