What proof should I ask a vendor for about their AI recommendation track record?
Ask for dated before-and-after results on a fixed question set, with repeated runs, the engines named, the sources cited and a client you can speak to. Treat screenshots of a single answer and any guaranteed placement as weak or false proof, because AI answers vary from run to run.
What counts as good proof?
Good proof lets you check the claim yourself. Ask for these.
- A fixed question list. The exact questions asked, unchanged before and after.
- Dates. When the baseline was taken and when the later check was.
- Repeated runs. How many times each question was asked. AI answers are generated each time, so one run is not evidence.
- Named engines. Which ones, and which were left out.
- The sources cited. The pages behind the answers, so you can open them.
- What changed on the client's side. The fixes made, so you can judge whether the result is plausible.
- A reference you can contact. With their permission, a real client.
What counts as weak proof?
A single screenshot of one answer. It could be a lucky run, or the result of the vendor's own location and chat history. A percentage with no question list behind it. "Top ranking in ChatGPT", since there is no ranking list. Testimonials with no result attached.
What should make you walk away?
Any guarantee of a place in an AI answer. The providers document crawlers and settings, but none promises placement; Google states that indexing and serving in its AI features are not guaranteed (Google). Also be wary of anything involving fake reviews or invented authors, which Google calls deception (Google).
Can a vendor without case studies still be good?
Yes, if they are new and honest about it. Ask them to run your own baseline for free or cheaply and to show you the method. A vendor who cannot show how they measure cannot show that they improved anything.
How do I check the proof myself?
Take one claimed result and reproduce it. Ask the vendor for the question list and the date of the later check. Ask the same questions yourself on the same engines, several times, and see whether you are named at roughly the rate they report. Allow for variation: you will not match exactly. If your results are far from theirs, ask why. Also check the cited sources: open them and confirm they say what the vendor claims. A vendor who welcomes this check is showing confidence. One who resists it, or says it cannot be reproduced, has given you useful information.
Frequently asked questions
Is a screenshot ever acceptable?
As an illustration, yes, as proof, no. Ask for the question list and the number of runs.
Should I ask for a refund guarantee?
A refund tied to deliverables you can verify is reasonable. A guaranteed placement is not.
How many clients should a vendor show?
One real reference with a dated result beats ten logos.
Can I run my own test first?
Yes, and you should. Ask a vendor to work from your baseline.