The short answer
The vendor criteria that predict outcomes are: production references with measurable KPIs (not demos), an explicit evaluation methodology they will run on your data, a clear stance on human-in-the-loop design, documented governance and security posture, and a stated build philosophy on prompts, tools, memory, and ops. The criteria that win RFP scorecards but do not predict outcomes are: case study count, partnership badges, headcount, and 'years in AI' (the field is not old enough for that to mean what it implies).
Five questions every vendor evaluation should ask
These questions surface the difference between a consultancy that ships agents that survive 18 months in production and one that ships impressive demos and disappears.
- ›Show me a production deployment with measured KPIs — not a demo, a deployment.
- ›What is your evaluation methodology, and will you run it on our data before we sign?
- ›Where does the human stay in the loop on this workflow, and why there specifically?
- ›Walk me through your security and governance posture — data handling, model selection, audit logging.
- ›What is your build philosophy on prompts, tool design, memory, and ops? What would you not do?
Red flags that consistently predict failure
A vendor that cannot show a production deployment with KPIs. A vendor whose evaluation methodology is 'we test extensively'. A vendor whose human-in-the-loop answer is 'whatever you want'. A vendor with no documented security posture. A vendor with no opinions on what they would not build. Each of these by itself is a yellow flag; two or more is a no.
Yellow flags worth probing
Heavy partnership-badge collection without depth in any one stack. Headcount as the lead pitch instead of senior practitioners on your account. Case studies with no measured outcomes. Disinterest in your existing data and tools. Reluctance to share evaluation results in writing.
Green flags worth weighting
A vendor that says 'we would not build this — buy it' on a horizontal workflow. A vendor that brings a written evaluation plan to the second meeting. A vendor whose references answer specific KPI questions without hedging. A vendor with documented architecture decisions you can read. A vendor that has shipped agents that ran for 12+ months without quality drift.
Key takeaways
- Production references with KPIs, not demos, predict outcomes.
- Evaluation methodology applied to your data is the single most predictive criterion.
- Strong opinions about what a vendor will not build are a buy signal.
- Partnership badges, headcount, and 'years in AI' do not predict 18-month outcomes.
Frequently asked questions
How many vendors should we evaluate?+
Three is usually right. Five becomes a procurement exercise rather than a partnership selection. One is risky — without comparison, you cannot pressure-test your own assumptions.
Should we require a paid pilot before a larger engagement?+
Yes. A small, scoped, paid pilot is the cheapest insurance you will ever buy. It surfaces working style, evaluation discipline, and governance posture in ways no RFP response will.
Is a boutique consultancy or a Big 4 the right choice?+
Depends on your governance constraints. Big 4 wins on procurement-heavy environments and global rollouts. Boutiques win on speed, senior-practitioner-per-dollar, and depth in agentic-specific patterns. Both can ship — the test is the five questions above, not the firm size.