Sitemap and robots.txt for a small SaaS

Google says a site of about 500 pages or fewer that is well linked internally may not need a sitemap, but that one helps when a site is new and has few external links (sitemaps overview). That describes most products at launch, so make one.

The sitemap should list the URLs you want to see in search results, in their canonical form, as fully-qualified absolute URLs (Google’s sitemap guide). Google treats every URL in a sitemap as a suggested canonical (source). So include:

• The homepage, pricing, feature pages, docs, blog posts and changelog.
• One URL per page: the one its canonical tag names.

And leave out:

• Anything behind login, and anything that redirects or returns an error.
• Pages with noindex. Listing a page and telling Google not to index it are opposite requests.
• Parameter, filter and sort variants of a page that is already listed.

The protocol requires all URLs in one sitemap to share the same protocol and host (sitemaps.org), so app.yoursite.com and yoursite.com need separate files. Submitting a sitemap is a hint. Google doesn’t promise to crawl or index everything in it.

Only urlset, url and loc are required. Everything else is optional (sitemaps.org).

Google’s documentation states that it ignores priority and changefreq values (source), and a 2023 post from its Search team repeats that Google doesn’t use either element at all (source). You can omit both.

lastmod is the one optional field worth getting right. Google uses it if it is consistently and verifiably accurate. The same post defines it as the date of the last significant modification: changes to the primary text, structured data or links count, and a tweak to the sidebar or footer does not. Two common ways to get it wrong are stamping every URL with the time the sitemap was generated, and bumping every date on each deploy. Use the date the content changed, in W3C Datetime format. A plain YYYY-MM-DD is valid. If you can’t supply a real date for a page, leave the field out for that page.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yoursite.com/</loc>
    <lastmod>2026-11-02</lastmod>
  </url>
  <url>
    <loc>https://yoursite.com/pricing</loc>
    <lastmod>2026-10-14</lastmod>
  </url>
  <url>
    <loc>https://yoursite.com/blog/import-from-csv</loc>
    <lastmod>2026-09-30</lastmod>
  </url>
</urlset>

The file must be UTF-8, and characters such as & in a URL must be entity-escaped (&). A single sitemap can hold up to 50,000 URLs or 50 MB uncompressed, and each loc must be shorter than 2,048 characters (sitemaps.org, Google). Put it at the root of the site, which is where Google recommends, so it can cover every path.

Past those limits, split the list into several sitemaps and point to them from a sitemap index. An index is a sitemapindex file with one sitemap and loc entry per file. It can list up to 50,000 sitemaps, and they must be on the same site as the index (Google on large sitemaps).

To tell crawlers where the sitemap is, add a Sitemap: line with the full URL to robots.txt. It can go anywhere in the file, you can have more than one, and if you use an index you only need to list the index. Submit it in Search Console as well, which reports invalid values in the file. That setup is in getting indexed on Google and Bing.

The file must be named robots.txt, be UTF-8 text, and sit at the root of the host. It applies only to the protocol, host and port it is served from, so a subdomain needs its own (Google’s robots.txt guide). The rules Google documents there and in its specification:

• Rules come in groups. Each group starts with one or more User-agent lines, followed by Disallow and Allow lines. User-agent: * matches every crawler that follows the standard, except Google’s AdsBot crawlers, which have to be named.
• A crawler obeys only one group: the one with the most specific user agent that matches it. If you add a group for one bot, that bot stops reading the * group, so repeat any rules it still needs.
• Anything not disallowed is allowed. Paths start with / and are case-sensitive.
• * matches any run of characters and $ marks the end of the URL, so /*.php$ matches any path ending in .php.
• When an allow and a disallow both match, Google applies the most specific rule, judged by path length.
• Google supports user-agent, allow, disallow and sitemap. It doesn’t support crawl-delay.

Google generally caches the file for up to 24 hours, so a fix isn’t picked up at once. The robots.txt report in Search Console can request a recrawl of the file (source).

The key limit: robots.txt controls crawling, not indexing. Google says it is not a mechanism for keeping a page out of Google, and that a disallowed URL can still be indexed if other pages link to it, shown without a description (source). It is also a request. Google notes that reputable crawlers obey it and others might not, so it protects nothing. Private data needs authentication.

User-agent: *
Disallow: /app/
Disallow: /api/
Disallow: /auth/
Disallow: /login
Disallow: /search

Sitemap: https://yoursite.com/sitemap.xml

Change the paths to match your routes. The reasoning for each:

• The app. Dashboard URLs only lead a crawler to a login redirect. Blocking them saves requests on both sides.
• The API. JSON endpoints aren’t pages. One exception: if a public page loads its content from /api/ in the browser, blocking that path hides the content from Google’s renderer.
• Auth. Login, callback and password reset routes. Keep the signup page crawlable if you want it to appear in search.
• Internal search and filters. These produce an unlimited number of URLs. For filter URLs, Google’s advice is to disallow them in robots.txt, because crawlers fetch a very large number of them before working out they’re useless and have less time for new pages (source).

Never block the CSS, JavaScript and images your public pages use. Google warns against blocking resources if their absence makes the page harder for its crawler to understand (source). In practice, don’t disallow your framework’s asset path. On Next.js that is /_next/.

AI crawlers are a separate decision with their own user agents. See blocking AI crawlers.

• A leftover staging file. User-agent: * followed by Disallow: / blocks every compliant crawler from the whole site. Open yoursite.com/robots.txt in a private window after each launch or migration and read it.
• Blocking a page that also has noindex. For noindex to work, the page must not be blocked by robots.txt. If it is, the crawler never sees the rule and the URL can still appear in results (source). Pick one: noindex to keep a page out of search, disallow to stop it being fetched.
• Using robots.txt to pick between duplicates. Google says not to (source). See canonical tag mistakes.
• Listing blocked or redirected URLs in the sitemap. Every sitemap URL should be crawlable, return 200 and be canonical to itself.
• A relative sitemap path. The Sitemap: line takes a full URL, not /sitemap.xml.

The indexability checker tests a single URL against robots.txt, noindex and its canonical in one go.

In the App Router, a sitemap.ts file in the app directory that exports a default function is served at /sitemap.xml (Next.js docs). The docs’ own example sets lastModified: new Date(), which stamps every URL with the time the file was generated. Pass real dates:

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getPublishedPosts } from "@/lib/posts"; // your own data

const base = "https://yoursite.com";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getPublishedPosts();
  return [
    { url: base, lastModified: "2026-11-02" },
    { url: base + "/pricing", lastModified: "2026-10-14" },
    ...posts.map((post) => ({
      url: base + "/blog/" + post.slug,
      lastModified: post.updatedAt,
    })),
  ];
}

changeFrequency and priority are optional in the returned objects. The docs note that the route is cached by default unless it uses a request-time API, so check that new pages show up without a redeploy if your list comes from a database. For more than 50,000 URLs, generateSitemaps splits the output into files served at /sitemap/[id].xml.

robots.ts works the same way and is served at /robots.txt (Next.js docs):

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      disallow: ["/app/", "/api/", "/auth/", "/login", "/search"],
    },
    sitemap: "https://yoursite.com/sitemap.xml",
  };
}

sitemap also accepts an array if you have several. Load both URLs in a browser after deploying.