Generator
robots.txt generator with AI-ready policies
Generate a robots.txt with a clear AI bot policy and a safe baseline for sensitive paths.
Use case
Answer-bot and training-bot control
Output
robots.txt + header recommendation
Preset
Balanced
Configure your access policy
Choose the policy that matches your strategy and review the routes that should stay closed.
Policy
Generated robots.txt
Ready-to-publish snippet. Includes a Content-Signal recommendation as a header comment.
# robots.txt generated by Semantiz SEO Tools
# Recommended header: Content-Signal: search=yes, ai-input=yes, ai-train=no
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /internal/
Sitemap: https://semantiz.com/sitemap.xml
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /What to review before publishing
- Do not mix training control with private headers; each layer solves a different problem.
- Block transactional or private routes even if the public site stays open.
- If you change the policy, update your HTTP headers so the message stays consistent.
Once the technical layer is fixed, measure actual visibility
Use these utilities to fix technical blockers first, then validate the actual impact with Semantiz's full audit.
Recommended next tools
llms.txt Generator
0 LLM cost
Turn core pages into a publish-ready llms.txt in one pass.
Draft a clean llms.txt file with grouped pages and machine-readable descriptions.
Schema Validator
Instant score
Validate whether your structured data is useful for AI consumers.
Paste JSON-LD and get fast feedback on types, gaps, and recommended properties.
Meta Tags Generator
Head snippet
Ship a stronger page header without starting from a blank file.
Generate an HTML head block with canonical, OG, Twitter, and JSON-LD basics.