How to evaluate an AI consulting firm before you hire one
Most AI consultants will tell you what you want to hear. Here are the six questions that separate practitioners from storytellers — and the red flags that should make you walk.
The AI consulting market is full of people who learned about AI six months ago and are now charging enterprise rates to explain it to you.
That’s not a cynical take — it’s a market reality. The demand for AI expertise is massive. The supply of people who’ve actually built AI systems inside real businesses is thin. The gap gets filled with storytellers.
Here’s how to tell the difference before you write a check.
The six questions
1. “What AI systems have you built that are running in production today?”
Not “what projects have you done.” Not “what clients have you worked with.” Specifically: what systems are running, processing real data, being relied on by a real business, right now?
If the answer is vague or drifts toward strategy work, that’s your signal.
Red flag: “We’ve done 40+ AI implementations” with no specifics about what those implementations were, what systems they’re running, or what outcomes they produced.
2. “What’s a case where you built something and it didn’t work? What happened?”
Practitioners have failure stories. Storytellers don’t — because they’ve never been close enough to the work to fail at it.
The failure story also tells you how they think. Do they diagnose clearly? Do they take ownership? Do they understand why it failed? These are all signals about whether they’ll be useful when things get hard (and things will get hard).
Red flag: “I can’t think of one” or a story where external factors explain 100% of the failure.
3. “How do you think about AI that will still be working in 18 months?”
AI moves fast. Systems built on specific models, specific APIs, or specific prompts can break as the underlying technology changes. Good practitioners think about durability — they build systems that degrade gracefully, that can be updated cheaply, that don’t require constant maintenance.
This question reveals whether they’re thinking about your long-term success or just the engagement deliverable.
Red flag: “We’ll cross that bridge when we come to it” or an answer heavy on technology specifics without any durability thinking.
4. “What happens after you leave?”
This is the accountability question. If they ship you something and walk away, what’s your team actually left with?
The answer should include: documentation, training, runbooks, and a clear model for how your team operates and extends the system. Not “we’re available for follow-up questions.”
Red flag: Engagement ends at delivery. No documentation. No handoff process. “We can extend for maintenance.”
5. “What would you NOT recommend AI for, in our situation?”
This is the honesty test. AI has real limits. A practitioner will tell you clearly where it adds leverage and where it doesn’t. A storyteller will tell you AI can solve everything.
The best answer here is specific to your business — they ask follow-up questions before answering, and they’re willing to say “that’s not actually a good AI use case.”
Red flag: AI is the answer to everything they’re asked about.
6. “Can I talk to two or three clients who’ve been through your process?”
Not references they provide (everyone provides flattering references). Clients who’ve been through the full cycle — including the hard parts. Ask specifically: what was harder than expected? What would they do differently?
Red flag: Reluctance to provide references, or references who can only speak to the engagement quality and not the outcomes 6+ months later.
What good looks like
A good AI consulting partner:
- Has built things and run them in production, not just designed them
- Can tell you clearly what didn’t work and why
- Thinks about your system’s life after they leave
- Has a point of view on what AI shouldn’t do
- Documents everything and trains your team
- Can produce clients who’ll speak honestly about working with them
That’s a high bar. There aren’t many firms that clear it. But the ones who do will make you more competitive in ways that compound.
The ones who don’t will leave you with a deck and a system that breaks in three months.
We try to hold ourselves to exactly this standard. If you want to put us through our own screening process, book a call — we’d rather earn your trust than pitch you.