“Crawled – currently not indexed”: causes and fixes

Google’s definition is one line: the page was crawled by Google but not indexed. The help page adds that Google doesn’t index all pages across the web, that this is a common message for many sites, and that it doesn’t necessarily indicate an issue that needs attention (Page indexing report help).

The same page sets the goal for the whole report: get the canonical version of every important page indexed, and don’t expect every URL on your site to be. So the first question for any URL in this list is whether you need it in search at all.

Several reasons in the report look alike and need different responses. These summaries follow Google’s definitions on the same help page.

• Discovered – currently not indexed. Google knows the URL but hasn’t crawled it. Google says that typically it wanted to crawl the URL, expected that to overload the site, and rescheduled. The last crawl date is empty. Nothing about the page’s content has been judged yet.
• Duplicate without user-selected canonical. Google treats the page as a duplicate of another one, and the page doesn’t declare a canonical. Google picked the other URL and won’t serve this one. Google describes this as working as intended. If the choice is wrong, declare a canonical. If the pages aren’t really duplicates, make their content differ substantially.
• Excluded by noindex. Google’s help page lists this as “URL marked ‘noindex’”. Google found a noindex rule and obeyed it. If you want the page indexed, remove the rule.
• Soft 404. The page returns a success status, but Google thinks it’s really a “not found” page. An empty page or an error message served with a 200 status can trigger it (Google on HTTP status codes).

Google gives no per-URL reason for this status. The causes below are inferences from what Google’s other documentation says about crawling and indexing, so treat them as things to check and not as a diagnosis.

Thin or near-duplicate pages

Google says its systems prioritise the fast inclusion of high quality, useful content (source). Pages with a heading and one sentence, tag and filter pages that list the same few items, and template pages where only a name changes give it little reason to keep a separate copy. Google’s starter guide notes that duplicate content doesn’t break its spam policies, but it can be a bad experience for users and waste crawling on URLs you don’t care about (SEO Starter Guide).

Few internal links

Google primarily finds pages through links from pages it has already crawled (source), and it advises that every page you care about should have a link from at least one other page on your site (source). A page that exists only in the sitemap, with nothing on the site pointing to it, looks unimportant because the site itself treats it that way.

A new domain with no links

The starter guide is plain that there’s no guarantee any particular site will be added to Google’s index, and that some changes take effect in hours while others take several months. See getting your first backlinks.

Content that only exists after JavaScript runs

Google processes JavaScript pages in three phases: crawling, rendering and indexing. With an app shell, the first HTML response doesn’t contain the actual content, so Google has to run your JavaScript before it sees anything. Google still recommends server-side rendering or pre-rendering, partly because not all bots can run JavaScript (JavaScript SEO basics). If rendering fails or a client-side route shows an empty state, Google may be looking at a blank page.

Inspect the URL in Search Console. The first result describes the last indexed version and is not a live test (URL Inspection help). Read these fields:

1. Discovery. Which sitemaps list the URL and which page referred Google to it. No referring page is a hint that internal links are missing, though Google notes the field can be empty even when links exist.
2. Last crawl. If the crawl happened before you last changed the page, Google hasn’t seen the current version.
3. Crawl allowed, Page fetch, Indexing allowed. These should all be clear. If one isn’t, you have a technical block. The indexability checker tests the same things from outside Search Console.
4. User-declared canonical and Google-selected canonical. If they differ, Google sees the page as a duplicate of the other URL. Canonical tag mistakes covers the usual reasons.

Then run Test live URL and open the tested page. Look at the HTML and the screenshot, which is only available in a live test. If your main text is missing from both, the problem is rendering. The live test has limits: it doesn’t check sitemaps or referring pages, and it can’t predict whether the page will be chosen as canonical or pass Google’s quality checks. A clean live test means the page can be indexed, not that it will be.

• Sort the list first. Go through the URLs and mark the ones you need in search. Feeds, paginated lists, internal search results and old parameter URLs often land here, and that is fine.
• Make the important pages worth a slot. Add what only that page can say: specifics, examples, numbers, screenshots described in text. If two pages answer the same question, merge them and redirect one to the other. Google treats a redirect as a strong canonical signal (source).
• Link to them. Add links from your homepage, navigation, footer or related pages, as real <a href> elements. Google can’t reliably follow links built from click handlers or elements without an href (source). Point internal links at the canonical URL.
• Send the content in the HTML. Render the main text on the server or at build time, so it is in the first response.
• Tidy the sitemap. List only canonical pages you want indexed. More in sitemap and robots.txt for a small SaaS.

Request indexing

After a live test, Request indexing adds the URL to a crawl queue. The documented limits (URL Inspection help, recrawl guide):

• There is a daily limit on requests. Google doesn’t publish the number.
• You can’t request indexing if the live test finds the page non-indexable.
• Requesting the same URL several times won’t get it crawled faster.
• Crawling can take from a few days to a few weeks.
• A request doesn’t guarantee the page will appear in the index.

Use it once per page, after you’ve changed something. For many pages, Google recommends a sitemap with accurate lastmod dates over individual requests.

Validate fix

Validate fix on the issue’s detail page asks Google to recheck every URL it knows with that status. Search Console first tests a sample. If the issue is still there, validation ends. Otherwise it works through the rest. Google says this typically takes up to about two weeks and sometimes much longer, and asks you not to click the button again until the run has passed or failed (source).

Validation only rechecks. It doesn’t change how Google judges the pages, so starting it without changing anything gets the same result. Check the Source column too: Google says that in general you can fix only issues where the source is listed as “Website”.

Do nothing when:

• The URL is one you don’t need in search. Google’s own advice is to make sure your key pages are indexed and not to expect the rest.
• The page is new, the technical checks pass, and you’ve already requested indexing once. Google recrawls pages on its own and suggests asking again only after an important change that it hasn’t picked up for a week or more.

If you’ve ruled out everything above and the whole site is affected, Google’s help page points to a form for reporting a suspected site-wide indexing bug.