Robots.txt Generator
Build a robots.txt file visually with per-crawler rules
User-Agent Rules
Global Settings
Generated robots.txt
Build a standards-compliant robots.txt visually — no need to memorize the syntax. Add rules per crawler, block or allow specific paths, set a crawl delay, and point bots at your sitemap. Everything runs in your browser; nothing is uploaded.
How to Use
- Pick a preset: Start from Allow All or Block All, or begin with the default wildcard rule.
- Add user agents: Use
*for all crawlers, or pick named bots such as Googlebot or Bingbot from the suggestions. - List paths: Enter Disallow and Allow paths, one per line. Paths must start with
/. - Set global options: Optionally add a crawl delay (seconds) and your sitemap URL.
- Copy or download: Copy the generated file or download it as robots.txt and upload it to your site root.
Features
- Visual rule builder with unlimited user-agent groups
- Suggestions for well-known crawlers (Googlebot, Bingbot, GPTBot, and more)
- One-click Allow All and Block All presets
- Optional Crawl-delay and Sitemap directives
- Live validation for empty agents, duplicate rules, invalid paths, and malformed sitemap URLs
- 100% client-side — your configuration never leaves the browser
Understanding Robots.txt Syntax
A robots.txt file is made of groups. Each group starts with one User-agent line naming the crawler the rules apply to — * matches every bot — followed by Disallow and Allow lines. Paths are matched by prefix, so Disallow: /admin blocks /admin, /admin/users, and /administrator. An empty Disallow: means nothing is blocked, which is the standard way to allow full access. Comments start with # and are ignored by crawlers. Remember that robots.txt is a public file: anyone can read it, so never rely on it to hide sensitive URLs — use authentication for that.
Use Cases
- Launching a new site: Allow all crawlers and reference your sitemap for fast discovery.
- Staging environments: Block all bots so test sites never appear in search results.
- Saving crawl budget: Disallow faceted search, internal search results, or infinite calendar pages.
- Managing AI crawlers: Add dedicated rules for GPTBot or CCBot without affecting Googlebot.
- Throttling aggressive bots: Set a crawl delay for crawlers that hammer your server.
Related Tools
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a plain text file placed at the root of your website (for example https://example.com/robots.txt) that tells web crawlers which pages or sections they may access. It follows the Robots Exclusion Protocol and is the first file well-behaved bots like Googlebot request before crawling your site.
How do I block all crawlers from my site?
Use the Block All preset. It generates 「User-agent: *」 followed by 「Disallow: /」, which asks every crawler to stay away from every page. This is common for staging or development sites.
What is the difference between Allow and Disallow?
Disallow tells a crawler not to visit a path, while Allow explicitly permits it. Allow is mainly used to carve out exceptions inside a disallowed directory — for example, disallow /private/ but allow /private/public-notice.html. For Google, the most specific (longest) matching rule wins.
Is Crawl-delay supported by all search engines?
No. Crawl-delay is a non-standard directive honored by Bing, Yandex, and some other crawlers, but Google ignores it. To throttle Googlebot, use the crawl rate settings in Google Search Console instead.
Why should I include a Sitemap line?
The Sitemap directive points crawlers to your XML sitemap so they can discover all your important pages faster. It must be an absolute URL, such as https://example.com/sitemap.xml, and you may list more than one.
Does robots.txt keep my pages out of Google search results?
Not reliably. robots.txt only controls crawling, not indexing. A blocked page can still appear in search results (without a description) if other sites link to it. To prevent indexing, use a noindex meta tag or password protection instead.