Robots.txt Generator

Build a standards-compliant robots.txt file visually instead of memorizing the syntax. Add one rule group per crawler, block or allow specific paths, attach a crawl delay, and point bots at your XML sitemap. The generator validates as you type and runs entirely in your browser — the configuration is never uploaded, so you can draft rules for a private staging host without exposing the URLs.

How to Use

Start from a preset or the default wildcard group. Allow All produces User-agent: * with an empty Disallow:; Block All produces Disallow: /. Click Add agent to create additional crawler groups. The user-agent field offers suggestions for common bots including Googlebot, Bingbot, GPTBot, and CCBot, but you can type any token.

Each group has two textareas: one for Disallow paths and one for Allow paths, with one path per line. Paths must begin with a forward slash. Optional global settings sit below the groups: a crawl delay in whole seconds and a single sitemap URL. The generated output updates live, and validation flags empty user agents, duplicate groups, paths that do not start with /, a negative crawl delay, and sitemap values that are not valid http(s) URLs. When the output is clean, copy it or download it as robots.txt and upload it to your site root.

Worked Example: Blocking a Staging Site but Keeping a Sitemap

Suppose you want to keep a staging server out of every search engine while still advertising the sitemap. Add one wildcard group, put / in Disallow, leave Allow empty, and set the sitemap to https://staging.example.com/sitemap.xml. The output is:

User-agent: *
Disallow: /

Sitemap: https://staging.example.com/sitemap.xml

If instead you want to block everything except a public notice page, put / in Disallow and /public-notice.html in Allow. Because Allow is the longer, more specific match, that one page stays crawlable while the rest of the site is blocked.

How Rule Matching Works

Crawlers compare a requested URL against the rules for the matching user-agent group. Matching is by prefix, not by exact equality, so Disallow: /admin blocks /admin, /admin/users, and also /administrator — a frequent surprise. A trailing slash narrows the match: Disallow: /admin/ blocks the directory but leaves /administrator alone.

* matches any sequence of characters, and $ anchors the end of the URL, so Disallow: /*.pdf$ blocks PDFs anywhere on the site. The wildcard * as a user-agent matches every crawler that does not have its own named group. A crawler obeys only one group: the specific named group if it exists, otherwise the wildcard group. That is why adding a dedicated GPTBot group does not disturb your Googlebot rules.

Choosing Between Blocking, Noindex, and Authentication

The right control depends on what you are trying to achieve, and these tools are not interchangeable. The table below is the rule of thumb that prevents most search-visibility accidents.

Goal	Use	Why
Save crawl budget on faceted or internal search URLs	Disallow	The page stays out of the crawler’s queue while remaining indexable via links
Keep a page out of search results entirely	noindex meta tag or X-Robots-Tag	The bot must crawl the page to see the directive, so do not Disallow it
Protect private data	Authentication or a password gate	robots.txt is public and cannot enforce access
Stop a whole staging environment from being crawled	Disallow: / for *	Fastest way to keep a non-production host out of the index

A common pattern is to block faceted URLs in robots.txt while leaving the canonical product pages crawlable. If a page must be neither crawled nor indexed, Disallow is fine because you have decided it should never be seen at all.

Common Mistakes

• Treating robots.txt as access control. The file is public; anyone can read the paths you list. Never place admin panels, API keys, or private documents behind a Disallow rule expecting secrecy. Use authentication.
• Expecting noindex behavior. Disallowing a URL prevents crawling, so a crawler may never see the page’s noindex tag. Blocking and de-indexing are separate operations.
• Forgetting the leading slash. Disallow: admin is not a valid path. Every rule must start with /.
• Creating two groups for the same bot. Duplicate User-agent lines for one crawler are merged ambiguously. Keep one group per agent.
• Leaving Crawl-delay on a Google-focused file and assuming it works. Google ignores it; use Search Console’s crawl rate settings.
• Blocking CSS and JavaScript. If you Disallow your asset directories, crawlers cannot render pages properly and may misjudge mobile-friendliness and content.

Edge Cases and Gotchas

The Crawl-delay directive is written only into the first group, matching common crawler expectations; if your first group is a specific bot, the delay applies there. A missing robots.txt (HTTP 404) is treated as “allow everything”, while a server error (5xx) can make crawlers back off entirely, so keep the file reachable. Use UTF-8 with no byte-order mark; a stray BOM before User-agent can make the first line unreadable. Finally, user-agent matching is case-insensitive for the token itself, so Googlebot and googlebot refer to the same crawler.

An empty Disallow: and Disallow: / are opposites and easy to confuse when hand-editing. The first allows the whole site; the second blocks it. Comments beginning with # are ignored, so annotate freely, but keep them on their own line rather than trailing a rule, since some older parsers mishandle inline comments.

Validation in this tool catches structural problems before they reach production, but a few checks are still worth doing manually. Confirm the file returns HTTP 200 at https://yourdomain.com/robots.txt — a redirect there is followed by Google but ignored by some crawlers. Open the URL in a private window to make sure your server is not adding an HTML error page in front of the text. Then fetch https://yourdomain.com/robots.txt and confirm the first line is User-agent, not a stray character.

Once live, use the robots.txt report in Google Search Console and the equivalent in Bing Webmaster Tools to spot blocked resources you did not intend to block. If you manage many hosts, keep staging files strict and production files permissive, and review the file whenever you add a new top-level section such as /shop/ or /api/.

Related Tools

• Meta Tag Generator
• HTTP Status Reference
• URL Encoder/Decoder
• XML Formatter