The Demo Is Designed to Impress You
Every enterprise AI vendor has a polished demo. The foundation models underlying those demos are now accessible enough that a team of three engineers can build something that looks like a sophisticated AI system within weeks. The demo will handle the questions you think to ask. It will not reveal what happens in the eighth month of a production deployment when the data is messier than the demo data, the edge cases are real, and the team that built the proof-of-concept has moved on to the next engagement.
The enterprise AI vendor market has expanded dramatically in the past 24 months. Every consulting firm, system integrator, and boutique technology company has an AI offering. The market signals that historically distinguished quality vendors - size, brand recognition, client logos - are poor predictors of AI delivery capability specifically. A major consulting firm may have thousands of AI practitioners globally but route your engagement to junior teams with limited production deployment experience. A boutique firm with strong conference presence may have impressive technical capability and no enterprise delivery infrastructure.
The buyer's job is to assess actual delivery capability, not market presence. The questions that reveal that capability are specific, require concrete answers, and vendors who have genuinely delivered in production are rarely uncomfortable answering them.
The Five Questions That Reveal Everything
There are five questions that reliably differentiate vendors who have delivered AI in production from vendors who have delivered impressive proofs-of-concept that never survived contact with production environments.
First: 'How many AI models do you currently have in production, at what scale, and how do you monitor them?' Vendors who can answer this specifically - naming use cases, describing monitoring approaches, discussing what they do when models drift - have production experience. Vendors who answer with process descriptions and capability claims do not. Second: 'Walk me through the data pipeline for one of your production deployments - from source system to model serving.' This separates vendors who understand the full stack from those who understand models but not the infrastructure that makes models production-viable. Third: 'What was the last engagement where something went wrong, and how did you handle it?' Every honest answer reveals delivery maturity. A vendor who claims no significant problems has not done enough complex production work.
Fourth: 'Can we speak with three client references whose AI is in production, not in development?' Reference conversations are the most reliable signal available. Ask the references what went wrong, not just what went right. Fifth: 'What does your knowledge transfer process look like, and what can your client operate independently after you leave?' The answer reveals whether the vendor's business model depends on long-term dependency or on client success.
Red Flags That Are Easy to Miss in a Competitive Process
The competitive bid process for AI services systematically advantages vendors who are good at competitive bids - not necessarily vendors who are good at delivery. Large firms invest heavily in proposal quality, presentation polish, and reference packaging. The correlation between proposal quality and delivery quality is low.
Specific red flags: proposals that lead with team size and pedigree rather than delivery track record; engagement structures with long 'design and discovery' phases that delay any working output; 'AI strategy' engagements that produce recommendations and slide decks but no working systems; pricing that front-loads cost before value is demonstrated; vendors who describe what they would build without committing to specific performance criteria.
The most reliable red flag: a vendor who resists defining success criteria for the initial engagement in specific, measurable terms. Organizations that know they can deliver against defined criteria welcome the accountability. Those who are not confident prefer the ambiguity. The willingness to commit to specific outcomes - deflection rate, accuracy metric, latency target, cost reduction - before the project starts is one of the best single-variable tests of vendor confidence in their own delivery capability.
The Due Diligence That Protects the Investment
The investment in AI vendor due diligence is proportional to the cost of getting it wrong. A wrong vendor choice on a 6-month, $800K engagement represents the direct cost plus 6 months of organizational opportunity cost plus the credibility damage that makes the next AI program harder to fund. Organizations that have absorbed that cost consistently report wishing they had spent more time on reference conversations and less time on proposal evaluation.
The most effective evaluation process combines: a focused technical challenge - a bounded proof-of-concept on a small slice of real data - that reveals operational capability under realistic conditions; reference conversations with production clients, not development clients; explicit success criteria for the initial engagement agreed before any technical work begins; and a structured comparison of delivery methodology, not just technical approach.
Knowledge transfer is a contractual requirement that many buyers overlook. Insist on: documented model architecture and training pipeline; runbooks for monitoring and incident response; training sessions for internal teams; and source code ownership. Vendors who resist committing to knowledge transfer deliverables are signaling a business model that depends on your dependency.
Why Isotropic Is Worth Evaluating
We recognize this article is self-serving coming from an AI vendor, and it is fair to apply this framework here. So here is what our answers look like against the questions we recommend asking every vendor.
Production deployments: Isotropic has delivered AI in production for Vietnam International Bank, the Central Bank of Oman, ETG World (commodity trading across 48 countries), telecommunications infrastructure providers, and enterprise clients across North America, Africa and Southeast Asia. We provide reference conversations with clients who speak to specific capabilities in production - not our characterization of what the client experienced, but direct conversations.
Delivery methodology: Isotropic's POD model is a documented, repeatable delivery process with defined milestones, explicit success criteria agreed before technical work begins, and a scale/stop decision point at the end of each proof-of-value. Knowledge transfer - architecture documentation, runbooks, training sessions, and source code ownership - is a contractual deliverable in every engagement, not an optional extra.
Contact business@isotrp.com to request reference conversations, a sample engagement structure, or a proposal scoped to your specific AI priority.
FAQ
Frequently asked questions
What are the five most important questions to ask an enterprise AI vendor?
The five questions that reliably differentiate vendors with real production experience from those with impressive demos: (1) 'How many AI models do you currently have in production, at what scale, and how do you monitor them?' - vendors with production experience name use cases and monitoring approaches specifically; (2) 'Walk me through the data pipeline for one of your production deployments, from source system to model serving'; (3) 'What was the last engagement where something went wrong, and how did you handle it?' - every honest answer reveals delivery maturity; (4) 'Can we speak with three client references whose AI is in production, not in development?'; (5) 'What does your knowledge transfer process look like, and what can your client operate independently after you leave?'
What red flags indicate an enterprise AI vendor cannot deliver in production?
Key red flags: proposals that lead with team size and pedigree rather than production delivery track record; engagement structures with long 'design and discovery' phases that delay any working output; 'AI strategy' engagements that produce recommendations and slide decks but no working systems; pricing that front-loads costs before value is demonstrated; and most reliably - a vendor who resists defining success criteria for the initial engagement in specific, measurable terms. Organizations that know they can deliver against defined criteria welcome the accountability; those who are not confident prefer ambiguity.
How should an enterprise evaluate AI vendor delivery capability before committing?
The most effective evaluation process combines: a focused technical challenge - a bounded proof-of-concept on a small slice of real data - that reveals actual operational capability under realistic conditions (not just demo conditions); reference conversations with production clients, not development clients, asking specifically what went wrong and how the vendor handled it; explicit success criteria for the initial engagement agreed before any technical work begins; and a structured comparison of delivery methodology, not just technical approach.
Why does AI vendor proposal quality correlate poorly with delivery quality?
The competitive bid process for AI services systematically advantages vendors who are good at competitive bids - not necessarily vendors who are good at delivery. Large firms invest heavily in proposal quality, presentation polish, and reference packaging. A major consulting firm may have thousands of AI practitioners globally but route your engagement to junior teams with limited production deployment experience. A boutique firm with strong conference presence may have impressive technical capability and no enterprise delivery infrastructure. The correlation between proposal quality and delivery quality is low - requiring direct reference conversations and technical challenges to assess actual capability.
What knowledge transfer requirements should enterprises include in AI vendor contracts?
Knowledge transfer requirements that should be contractual in every enterprise AI engagement: documented model architecture and training pipeline (written for the team that will maintain the system, not for the team that built it); runbooks for model monitoring, retraining and incident response; training sessions for internal teams on model operation and modification; and source code ownership - enterprises should own the code, not license it. Vendors who resist committing to specific knowledge transfer deliverables are signaling a business model that depends on client dependency, not client success.