Sitemaps and robots.txt Explained: How Search Engines Discover, Crawl, and Index Your Site

Every site owner eventually edits two files they don’t fully understand: robots.txt and sitemap.xml. They do opposite jobs — one tells crawlers where not to go, the other hands crawlers a map of everything that exists — and mixing them up produces sites that are either invisible to Google or letting crawlers waste budget on pages you meant to hide. This guide gives you a working mental model for both, plus the mistakes that trip up otherwise-solid sites.

One Sentence Each

robots.txt — a plain-text permission file: “crawler X, you may not fetch these URL prefixes.” It lives at the site root (https://example.com/robots.txt) and is the first thing many crawlers request.
sitemap.xml — an XML list of the pages you want discovered and how often they change. It exists to tell crawlers about new or updated pages they might never find by following links.

Both are signals, not hard rules. Search engines may ignore robots.txt lines, and a sitemap never forces a page to be indexed. But done right, they shape crawling and discovery more than almost anything else you control.

robots.txt: Format and Matching

A robots.txt file is made of rule groups, each starting with a User-agent: line, followed by the rules for that crawler:

User-agent: *
Disallow: /private/
Disallow: /tmp/*.pdf$
Allow: /private/public/

User-agent: * means “apply to all crawlers.” User-agent: Googlebot targets one crawler specifically.
Disallow: /private/ blocks the /private/ URL prefix (and everything under it).
Allow re-permits a sub-path that a broader Disallow would block. Google resolves conflicts by longest matching rule: Allow: /private/public/ wins over Disallow: /private/ because the path is longer.
Google also supports two wildcards: * matches any characters, $ matches the end of the URL (/tmp/*.pdf$ blocks PDFs under /tmp/).
An empty Disallow: allows everything; a file with no groups allows everything. A blocked prefix doesn’t mean “gone” — see below.

What robots.txt Does NOT Do

The most dangerous misunderstanding: robots.txt cannot hide a page. Blocking /private/ in robots.txt means crawlers won’t fetch it — but if other sites link to it, Google can still index the URL and show a result without a snippet (usually with the title and URL only).

If you want a page out of the index entirely, you need an explicit “no index” instruction:

<meta name="robots" content="noindex">

or the HTTP equivalent header X-Robots-Tag: noindex. The Meta Tag Generator writes valid robots and Open Graph tags, including noindex / nofollow combinations. The practical split: robots.txt is for blocking crawl (saving crawl budget), noindex is for removing from the index.

XML Sitemaps: The Map

A sitemap is XML with one <url> block per page:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-08-18</lastmod>
  </url>
  <url>
    <loc>https://example.com/guides/sitemap-robots-guide/</loc>
  </url>
</urlset>

<loc> is the only required child — the page’s canonical URL (one scheme + one host; never mix http/https or www/bare).
<lastmod> is an ISO 8601 date telling crawlers when the page last changed. It’s a hint; if you leave it stale or fake it, crawlers learn to ignore it.
<changefreq> and <priority> are legacy and largely ignored by Google — you can omit them.
Limits: at most 50,000 URLs or 50 MB (uncompressed) per sitemap. A compressed sitemap.xml.gz is fine and preferred.

Sitemaps don’t boost rankings. Their job is discovery: Google learns a URL exists and how fresh it is, so new and changed pages get crawled sooner instead of waiting for a link from somewhere. That’s why a big or fast-changing site publishes one — and why the sitemap must list canonical URLs, or you’ll teach crawlers about duplicate URLs instead of the real pages.

When One Sitemap Isn’t Enough

More than 50,000 URLs — or a site with clearly separated sections — calls for a sitemap index: an XML file listing other sitemaps:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-blog.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-tools.xml</loc></sitemap>
</sitemapindex>

Each <sitemap> entry can carry a <lastmod> too. You’d then submit the index URL to search engines rather than every individual sitemap.

How the Two Files Work Together

The files reinforce each other in one common setup:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

The Sitemap: directive is how many crawlers learn where your map lives (it can appear anywhere in the file). This is the pattern most sites want: let everything be crawled (so the sitemap’s pages can be reached), and point crawlers at the map.

Common Mistakes That Quietly Kill Discovery

Blocking the sitemap in robots.txt. If you Disallow: /sitemap.xml (or /sitemap*) while also declaring it via Sitemap:, crawlers may refuse to fetch the very file meant to help them. Keep sitemaps allowed.
Disallow: / on a site you want indexed. This is the “why is my site not in Google?” classic.
Inconsistent URLs. Sitemap lists http:// while the site redirects to https://, or mixes www and bare domains. The sitemap must use the canonical form.
Stale or wrong lastmod. A sitemap full of yesterday’s dates for untouched pages teaches crawlers to discount it.
Malformed robots.txt. A file that ends mid-rule or without a blank line between groups can make crawlers treat later rules differently. Keep it tiny and valid; most sites need fewer than ten lines.
Forgetting to submit. Even a perfect sitemap helps less if you never submit it in Google Search Console / Bing Webmaster Tools and only rely on the Sitemap: line.

Quick Reference

robots.txt = “don’t crawl these prefixes”; sitemap.xml = “here’s everything that exists.” One controls crawl, the other enables discovery.
Rules are User-agent: + Allow:/Disallow: lines; Google uses longest-match precedence and * / $ wildcards.
robots.txt cannot hide a page — use <meta name="robots" content="noindex"> (or X-Robots-Tag) to remove from the index.
A sitemap <url> needs <loc>; <lastmod> is a freshness hint; <changefreq>/<priority> are legacy.
Max 50,000 URLs / 50 MB per sitemap; use a sitemap index beyond that.
Put Sitemap: https://example.com/sitemap.xml in robots.txt and keep that path allowed.
Never block the sitemap, mix schemes/hosts, or ship stale lastmod dates.

To produce a correct file instead of hand-writing XML, the XML Sitemap Generator turns a URL list into a valid sitemap, and the Robots.txt Generator builds per-crawler rules visually. When crawlers do visit, HTTP status codes tell the story of what they found — 200 for pages to index, 404/410 to drop, 301 to consolidate.