How do I measure whether changes to our site improved AI recommendations?
Lock a set of questions, record how often you are named before you change anything, make the change, then ask the identical questions again over several weeks. Because answers vary from run to run, compare repeated results and keep some questions you did not touch as a control.
Why is this harder than it sounds?
AI answers are generated each time. Ask the same question twice and you may get two different lists of businesses. If you ask once before a change and once after, a difference could be your change or just that variation. Good measurement is mostly about separating the two.
What is the method?
Follow these rules.
Fix the questions. Write down 20 or more questions and do not change their wording during the test. New wording is a new test.
Record a baseline first. Ask each question on each engine at least three times before touching the site. Count how often you are named.
Change one thing at a time where you can. If you fix robots rules, rewrite five pages and add markup in the same week, you will not know which helped.
Keep a control. Leave some questions unrelated to your changes. If those also improve, the gain probably is not from your change.
Wait for the engines to catch up. OpenAI and Perplexity say crawler changes take about 24 hours to be picked up (OpenAI, Perplexity). Re-indexing and answer changes take longer, so compare over several weeks.
Compare rates, not single answers. "Named in 4 of 12 runs, now 8 of 12" is evidence. "It named us today" is not.
What should I look at besides being named?
Whether the sources cited changed, whether the description became correct, and which competitors appear alongside you. A cited page that is now your updated one is stronger proof than a mention alone.
For traffic, Google reports AI feature clicks inside the Performance report in Search Console, not separately (Google), so compare against branded search too.
What if the numbers do not move?
Check the chain in order. Was the page crawlable when you checked? Search crawlers need time to return, and OpenAI and Perplexity both note about a day for robots changes. Does the page actually answer the question in its opening lines? Did another source still repeat the older version? Were you measuring the same questions as before? If all four are fine, wait another week and repeat, because a flat result after only one check is not a failure. If it is still flat after several weeks, try a different change rather than more of the same one.
Frequently asked questions
What counts as a real improvement?
A rise that holds across several weekly checks and shows up in questions you targeted more than in the control questions.
Can I trust a single big jump?
Be cautious. Wait for the next checks to confirm it.
Do I need analytics too?
It helps. Compare AI-referred visits and branded searches, remembering that many mentions never produce a click.
Is it a problem to change questions midway?
Yes, it breaks comparison. Start a new baseline instead.