How to get your site indexed by Google and Bing

There are three separate steps, and it helps to know which one is failing:

1. Discovery. The search engine learns that a URL exists, from a link on another page, your sitemap or a direct submission.
2. Crawling. Its crawler fetches the page, if robots.txt allows it.
3. Indexing. The engine processes the page and decides whether to store it. A page can be crawled and still not indexed.

Being indexed is not the same as ranking. An indexed page can be eligible to appear and still not show up for the searches you care about. But a page that isn’t indexed can’t appear at all, so this is the step to get right first.

Search Console is Google’s free dashboard for site owners. It shows which pages are indexed, why others aren’t, and which searches you appear for.

1. Add a Domain property for your domain, like yoursite.com. It covers every subdomain and both http and https, and it’s verified with a DNS record at your domain registrar (Google’s instructions). If you can’t edit DNS, a URL-prefix property works too, but it only covers the exact address you enter.
2. Submit your sitemap in the Sitemaps report (more on sitemaps below).
3. Inspect your homepage with the URL Inspection tool and choose Request indexing. Do the same for a few of your most important pages. There’s a daily quota, and Google says resubmitting the same URL doesn’t speed things up (source).

Bing Webmaster Tools does the same job for Bing. Once your site is in Search Console, the quickest way in is to sign in to Bing Webmaster Tools and choose Import on the My Sites page. Bing asks for access to your Search Console account, then adds and verifies the sites you pick, along with their sitemaps (Bing’s walkthrough).

You can also verify manually with an XML file, a meta tag or a DNS record. Either way, check afterwards that your sitemap appears in Bing’s Sitemaps section.

A sitemap is a list of the URLs you want indexed. It helps search engines discover pages, especially on a new site with few links pointing to it. Google’s sitemap documentation sets out the rules:

• Use full, absolute URLs, like https://yoursite.com/pricing, not /pricing.
• A single sitemap can hold up to 50,000 URLs or 50 MB uncompressed. Split larger sites into several.
• Only list pages you want indexed: the canonical version of each page, returning a 200 status, without noindex.
• Google ignores priority and changefreq. It uses lastmod when the dates are consistently accurate, so only set it to when the page really changed.

Submit the sitemap in Search Console and Bing Webmaster Tools, and add a line to robots.txt so any crawler can find it:

Sitemap: https://yoursite.com/sitemap.xml

Google retired its old sitemap “ping” URL in 2023 (announcement), so scripts that still ping it do nothing. Most frameworks and site builders can generate a sitemap for you. Either way, a submitted sitemap is a hint, not a guarantee that every URL will be crawled.

robots.txt, at the root of your domain, tells crawlers which URLs they may fetch. The most common launch mistake is a file copied from a staging site that blocks everything:

User-agent: *
Disallow: /

For most small sites, the right file allows everything and points to the sitemap. Only disallow areas that have no business in search, like an admin panel or internal API.

robots.txt is not a way to keep a page out of search results. Google notes that a disallowed page can still be indexed if other sites link to it, just without a description (source). To keep a page out, use noindex instead, and leave the page crawlable so the noindex can be seen.

A noindex rule tells search engines to drop a page from their results. It can be a meta tag in the page’s <head> or an X-Robots-Tag HTTP header:

<meta name="robots" content="noindex">
X-Robots-Tag: noindex

It’s the right tool for thin or private pages, and a silent disaster when it ends up on the wrong ones:

• Left over from development. A site-wide noindex added while building, or a CMS setting that discourages search engines, still switched on at launch.
• Sent by the host. Some hosting platforms add an X-Robots-Tag: noindex header to preview deployments. Make sure your production domain doesn’t send it. You won’t see a header by viewing the page source.
• Combined with a robots.txt block. For noindex to work, the page must not be blocked in robots.txt, otherwise the crawler never sees the rule (source).
• Added by JavaScript. Keep robots rules in the HTML your server sends, where every crawler reads them on the first pass.

To check a page for these blockers in one go, use the indexability checker.

When the same content is reachable at several URLs (with and without www, with tracking parameters, or sorted differently), search engines pick one as the canonical and mostly show that. A rel="canonical" link tells them which one you prefer:

<link rel="canonical" href="https://yoursite.com/pricing">

Google treats canonical tags, redirects and sitemaps as signals, not commands, and may pick a different URL if they disagree (Google’s guide). The mistakes to look for:

• Every page’s canonical pointing at the homepage, usually from a template.
• Canonicals pointing to a staging domain, or to http:// when the site is https://.
• Relative canonical URLs. Use absolute ones.
• Different signals for the same page, such as a sitemap listing one URL and the canonical naming another.
• Using noindex or robots.txt to choose between duplicates, which Google advises against.

Each indexable page should usually have a canonical that points to itself, at its one clean address. Check yours with the meta tag checker.

IndexNow is an open protocol for telling search engines that a URL was added, updated or deleted, instead of waiting for them to recrawl. Participating engines share submissions with each other, so one ping reaches all of them (IndexNow FAQ). The current list includes Bing, Yandex, Seznam, Naver and Yep (full list). Google isn’t on it, so for Google, rely on your sitemap and Search Console.

Setting it up takes three steps:

1. Create a key of 8 to 128 characters, using letters, numbers and hyphens.
2. Host it as a UTF-8 text file named after the key, for example https://yoursite.com/your-key.txt, containing only the key.
3. Send changed URLs to an IndexNow endpoint, up to 10,000 in one request.

https://api.indexnow.org/indexnow?url=https://yoursite.com/new-page&key=your-key

You may not need to write any code: Bing lists site builders with IndexNow built in, WordPress plugins that add it, and Cloudflare’s Crawler Hints setting (Bing’s IndexNow guide). Submitting a URL does not guarantee it will be crawled or indexed.

There’s no fixed timeline. Google says it can take a few weeks to notice a new site (source), and that crawling a requested URL can take anywhere from a few days to a few weeks, with no guarantee the page will be included (source).

Things that tend to help:

• Links to your site from other sites. Google recommends this to help it find a new site.
• A clean sitemap and internal links, so every important page is reachable from your homepage.
• Real, useful text on each page. Google says it prioritises including high-quality, useful content, and crawlers can’t read text that only exists in images.

Google also warns against paying for links to speed things up, which breaks its spam policies (source).

• URL Inspection in Search Console. The most reliable answer for a single page. It shows whether the URL is indexed, which canonical Google chose, and when it was last crawled. The live test checks the current version (Google’s help). Bing Webmaster Tools has its own URL Inspection tool.
• The Page indexing report. Lists indexed and non-indexed pages with a reason for each (Google’s help). “Discovered – currently not indexed” means Google knows the URL but hasn’t crawled it yet. “Crawled – currently not indexed” means it crawled the page and chose not to index it for now, which Google says isn’t always a problem.
• A site: search. Searching site:yoursite.com in Google or Bing gives a rough picture, but it doesn’t list every indexed URL, so don’t rely on it for counts.

If a page you care about isn’t indexed, work back through the steps: is it in the sitemap, linked from your site, allowed by robots.txt, free of noindex, and canonical to itself?