What the indexability checker looks at
A page only shows up in Google if Google can crawl it, is allowed to index it, and agrees that this URL is the main version. The checker loads the page the way a crawler does and reads every signal that decides that:
- HTTP status and redirects: the full redirect chain, whether each hop is permanent, and the final status code.
- robots.txt: the rules in the Googlebot group (or the
*group when there isn't one), the exact line that matches this URL, and anySitemap:lines. - noindex and nofollow: in the robots meta tag, a googlebot meta tag, and the
X-Robots-TagHTTP header, which is easy to miss because it isn't in the HTML. - Canonical: from the
<link rel="canonical">tag or aLinkheader, and whether it points to this page or somewhere else. - One version of the site: whether http:// redirects to https:// and whether www and non-www both load.
- Sitemap: whether one exists, whether it's a valid XML sitemap and whether it lists this page.
How to read the results
Red items stop the page from being indexed at all: an error status, a robots.txt rule that blocks Googlebot, or a noindex tag or header. Fix those first. Amber items don't block indexing, but they can make Google index a different URL than you expect, split ranking between copies, or slow down discovery.
The checker sees what a crawler sees on one visit. Google also weighs things it learns over time, like links from other sites and how often a page changes, so a clean result means nothing on your side is in the way, not that the page is guaranteed to rank. To confirm what Google itself has indexed, use the URL Inspection tool in Google Search Console.
Common reasons a page isn't indexed
- A
noindexleft on from a staging site, often set by a CMS option like "discourage search engines" or a header added in hosting settings. - A broad robots.txt rule such as
Disallow: /, sometimes copied from a template. - A canonical link pointing at the homepage or at another domain, added by a theme or plugin.
- The page answering with a redirect or error to crawlers, or only being reachable through JavaScript.
- Duplicate copies on http:// and https://, or on www and non-www, competing with each other.
Frequently asked questions
- What's the difference between robots.txt and noindex?
- robots.txt controls crawling: a Disallow rule stops Google from fetching the page. noindex controls indexing: Google fetches the page, sees the tag or header and leaves it out of results. A page blocked in robots.txt can still appear as a bare link if other sites link to it, because Google never sees its noindex. To keep a page out of results, allow crawling and use noindex.
- Which robots.txt rules does Googlebot follow?
- Googlebot obeys the group that names it (User-agent: Googlebot) and ignores the * group when one exists. Within the group the most specific rule, the longest matching path, wins, and Allow wins a tie with Disallow. The checker applies the same logic and shows the line that decided.
- What is the X-Robots-Tag header?
- It's an HTTP header that can carry the same directives as the robots meta tag, such as noindex or nofollow. Because it isn't in the HTML, it's easy to overlook. Hosting platforms and CDNs sometimes add it to preview or staging domains, and it can be left on after launch.
- My canonical points to another URL. Is that a problem?
- Only if you want this URL to rank. A canonical tells Google which URL is the main version, so pointing it elsewhere asks Google to index that URL instead. That's right for duplicates, like a page with tracking parameters, and wrong for a page that should appear on its own.
- Do I need a sitemap?
- Small, well-linked sites get indexed without one, but a sitemap helps Google find new pages faster and lets you see indexing status per URL in Search Console. List it in robots.txt with a Sitemap: line so every search engine can find it.
- Why does the checker get a 403 when my site works in the browser?
- Some firewalls and bot filters block unfamiliar crawlers like ours (IndieTideToolsBot) but let Googlebot through. If you see a 403, confirm in Google Search Console with URL Inspection. If Google is blocked too, allow verified search engine bots in your firewall settings.
- Is the page I check stored anywhere?
- No. The page, robots.txt and sitemap are fetched once to run the check and aren't saved. Each visitor can run a limited number of checks per minute.