How to test whether AI assistants know your product

Decide the questions before you open any assistant, and keep the wording fixed. If you rephrase each time, you can’t compare one month with the next. Eight to twelve questions across four types is enough.

• Brand. “What is Acme Invoices?” and “Who is Acme Invoices for?” These show whether the assistant knows you exist and whether its description is right.
• Category. “What are the best invoicing apps for freelance designers?” This is what a buyer asks, and the hardest list to get onto.
• Problem. “How do I chase late invoices without sounding rude?” The person hasn’t named a product type. See whether a tool is suggested at all, and which.
• Comparison. “Acme Invoices vs BigCo Billing” and “What are some alternatives to BigCo Billing?” These show how you’re positioned against a name people already know.

Write them the way a customer would type them, not the way your landing page is worded. Support tickets, sales calls and search queries in Search Console are good sources of real phrasing.

Test the assistants your users are likely to use. A reasonable set is ChatGPT, Claude, Perplexity, Gemini, Microsoft Copilot, and Google Search itself for AI Overviews and AI Mode. For each one:

1. Start clean. Use a new conversation for every question, so earlier answers don’t leak into later ones. Turn off memory and personalisation, or use the assistant’s temporary mode if it has one. Claude’s incognito chats, for example, don’t use its existing memory (Anthropic’s help centre). An account that has spent months discussing your product is not a neutral tester. A signed-out browser window is another option where the assistant works without an account.
2. Run with search on. This tests whether the assistant can find and cite you today. Record the sources it cites, since they tell you which pages it read.
3. Run with search off, where you can. This tests what the model knows from training alone. Not every assistant gives you the choice. Claude has a web search toggle in some versions of its interface and none in others (source). Perplexity describes itself as searching the internet in real time (source). Where you can’t switch search off, note whether the answer came with citations.
4. Repeat. Ask each question three times in separate sessions.
5. Don’t lead. Leave your product name out of category and problem questions, and don’t follow up with “what about Acme?” until you’ve recorded the first answer.

Note the model name shown in the interface and the date. An answer from one model version says little about the next.

One row per run. These columns are enough:

date | assistant | model | search on/off | question | run #
mentioned (y/n) | position in list | description accurate (y/n/partly)
what was wrong | sources cited (URLs) | competitors named

Paste the full answer into a notes column or a separate document. You will want the exact wording later, when you try to work out where a wrong detail came from.

Summarise as counts, such as “named in 2 of 9 category runs with search on”. With this few runs the numbers are rough. They are good for spotting zero versus sometimes versus usually, and for catching a wrong description. They can’t support a claim like “visibility rose 10%”.

The same question can produce a different list a minute later. Several documented things cause that:

• The models are not deterministic. OpenAI’s developer docs say outputs are non-deterministic by default and may differ from request to request (source).
• The searches differ. Google says AI Overviews and AI Mode may issue multiple related searches across subtopics for one question (source). A different set of searches returns a different set of pages to draw on.
• The web changes. With search on, the answer depends on what is indexed and ranking that day.
• Memory and context. An assistant that remembers earlier chats can tailor its answer, which is why the clean session matters.
• The system changes. The same OpenAI page notes that outputs can differ because of changes made on its systems. You won’t always know when an assistant was updated.

So a single screenshot of your product in an answer proves it can happen, not that it does. The same goes for a single run where you’re missing.

Chat tests show the output. Logs show the input: whether the crawlers that feed these assistants reach your site at all. Search your access log for the documented user-agent names:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Applebot|bingbot|Googlebot" access.log

What each group tells you:

• Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, bingbot). If these never appear, you can’t be cited from that company’s search. Check robots.txt and any firewall rules.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). Each hit means a person’s question caused the assistant to load that page. The URLs they fetch show which pages assistants reach for.
• Training crawlers (GPTBot, ClaudeBot, CCBot). Their visits say your pages may be collected for training or, for CCBot, for an open dataset. They say nothing about search answers.

The names come from each company’s documentation: OpenAI, Anthropic, Perplexity, Apple and Common Crawl. Two things to keep in mind when reading the results:

• Google-Extended and Applebot-Extended will never show up. Google-Extended has no user-agent string of its own (source), and Apple says Applebot-Extended does not crawl.
• A user-agent string can be faked. Common Crawl warns of crawlers falsely identifying as CCBot. OpenAI, Anthropic, Perplexity and Common Crawl publish their IP ranges, so check suspicious traffic against those.

If your host doesn’t give you raw access logs, look for request logs at your CDN, or log the user agent from your own middleware for a few weeks. Client-side analytics is not a substitute: it only counts visits where your script ran, and you shouldn’t assume a crawler runs it.

• Unknown even on brand questions with search on. This is an access or indexing problem. Confirm the search crawlers aren’t blocked (Should you block AI crawlers?) and that your pages are indexed (How to get indexed by Google and Bing).
• Known, but described wrongly. Open the sources the assistant cited and look for the error on them: an old listing, a stale review, or your own page being vague. Fix your page first, then ask the other sites to update theirs.
• Known by name, absent from category answers. Look at which pages are cited for those answers. They are the round-ups, comparisons and threads the assistant read. Being included there is the work, and How to get your product mentioned by ChatGPT and Perplexity covers it.
• Found with search, unknown without. Expected for a new product. A model can’t have learned about something that launched after its training data was collected.

Keep expectations in line with what’s known. Nothing here guarantees a mention, and no assistant offers a way to request one.

Monthly is enough for a small product. Also rerun after a launch, a rename, a pricing change or a homepage rewrite, once crawlers have had time to revisit the changed pages (your logs will show when). Use the same questions, the same sheet and the same clean-session routine, and add a new tab per round so you can compare.

Two first-party reports are worth checking on the same schedule. Google’s Search Console has a generative AI performance report (Google’s AI optimisation guide), and Bing Webmaster Tools has an AI Performance report with citation counts and sampled grounding queries for Copilot and Bing’s AI summaries (Bing’s announcement).

Paid tools exist that run prompts like these at scale and chart the results. They automate the same kind of sampling you’ve just done by hand. Google points out that no third-party tool has access to its internal ranking or AI systems (source), so what they show is observed output, the same thing your sheet records. Do the manual test for a few rounds first, so you know what question you’d be paying to have answered.