Operational AI, Part 4: What the Demo Won’t Tell You

Most product demos are designed to show what is possible. Your job is to determine what is practical for your organization.

You don’t need to be an AI expert to evaluate a vendor well. You need questions that move the conversation beyond ideal conditions and reveal whether the product is mature, evidence-based, grounded in community behavioral health, and ready for the realities of implementation.

Welcome to Part 4 of our four-part series on operational AI:

1. The Biggest AI Opportunity Is in Your Back Office: How Operational AI Solutions Can Drive Performance
2. The DSM of Operational AI: A Practical Way to Organize the Operational AI Landscape
3. Beyond Asking AI About AI: How Behavioral Health Leaders Find AI Solutions
4. What the Demo Won’t Tell You: How to Evaluate Operational AI Vendors

Use this guide to focus the conversation on six decisions: Does the solution work? Does it fit behavioral health? Will it fit your workflows? Can you implement and sustain it? Can you govern it over time? Will it create enough value?

Scope of this Guide
This guide focuses on operational AI. It does not address systems that independently diagnose a patient, determine or recommend treatment, provide autonomous therapy, replace a licensed clinician, or make other clinical decisions. Those uses require a different level of evidence, governance, risk review, and regulatory scrutiny.
Privacy, safety, and security are essential parts of vendor evaluation, but they are not covered in depth here. Confirm that every selection follows your organization’s AI policy and vendor-review process, including applicable requirements such as HIPAA and 42 CFR Part 2. A separate toolkit should address those questions comprehensively rather than treating them as a quick checklist.

How to Use This Guide

For each decision, begin with the three must-ask questions. Use the deeper questions when the answer is incomplete, the risk is higher, or the solution will affect a critical workflow. Pay as much attention to how the vendor answers as to the answer itself.

• Ask for evidence from real environments, not only a sandbox or a best-case demo.
• Request baselines, sample sizes, time frames, failure examples, and customer references that resemble your organization.
• Treat vague promises, universal success claims, and “coming next quarter” road maps as signals to investigate further.

1. Does it really work?

A polished demo can make an emerging feature look finished and an average result look exceptional. Test both product maturity and accuracy before you discuss scale.

Product Maturity

Determine what is available today, what depends on custom services, and what exists only on the road map.

Must-ask Questions

• Which features are generally available today, and which are still on the road map?
• Can you show this working in a real customer environment, not only a sandbox?
• What has changed most in the product during the past six months, and why?

Go Deeper if Needed

• What percentage of the workflow depends on AI rather than traditional software, rules engines, or automation?
• Which parts use generative AI, predictive AI, rules, or another method?
• How many organizations and end users have used each major feature, and for how long?

Strong answers include:	Be cautious if you hear:
Clear distinctions among released, pilot, beta, custom, and planned capabilities	A road map where nearly every needed feature is “coming next quarter”
Dates, adoption figures, customer examples, and implementation proof points	A refusal to show the product outside a controlled demo
Candid limitations and curiosity about how your organization operates	Answers that change when you ask whether a feature is standard, optional, or custom

Accuracy and failure modes

“Accurate” is not a useful claim without a defined task, benchmark, population, and error rate. Ask how the vendor knows the system works and how it detects when the system does not.

Must-ask Questions

• Which metrics matter most for this product, and what results have customers achieved against those metrics?
• What errors occur most often? Walk us through a recent example and what changed afterward.
• When does the AI intentionally refuse, escalate, or require human review?

Go Deeper if Needed

• How do you look proactively for new errors rather than waiting for customers to report them?
• What level of human review does your team maintain? What level should our team maintain?
• What assumptions has the system made incorrectly, and how were they detected and corrected?

Strong answers include:	Be cautious if you hear:
Metrics tied to a defined workflow and customer outcome, with a benchmark and sample size	“It is very accurate” without a metric, comparison point, or evidence
A clear taxonomy of failure modes and a repeatable process for finding them	Only ideal examples, with no recent failure the team can explain
Defined human oversight and explicit thresholds for refusal or escalation	The assumption that customer staff will catch errors without time, training, or workflow support

2. Does it work for community behavioral health?

A product that works in a broad health care setting may still fail in community behavioral health. Test whether the vendor understands the clinical, operational, financial, payer, and crisis-service realities that shape your work.

Must-ask Questions

• Who on your team has community behavioral health expertise, and how does that expertise shape the product?
• How many community behavioral health organizations use the platform, and how long have they used it?
• Which behavioral health workflows have you intentionally chosen not to automate, and why?

Go Deeper if Needed

• Which payer models have your customers worked under?
• How does the product identify and handle crisis workflows, including escalation pathways?
• How does the product support the Medicaid documentation and reporting requirements that apply to our programs?

Strong answers include:	Be cautious if you hear:
Examples spanning clinical, operational, financial, and administrative perspectives	A single clinical adviser presented as proof of organization-wide expertise
Experience with Medicaid, crisis services, community-based care, and organizations similar to yours	Health-system examples that do not translate to community behavioral health
Recognition that some workflows should remain human-led	No clear poor-fit workflow, customer profile, or use case

3. Will it fit the way your organization works?

A clean interface does not prove workflow fit. Map what disappears, what remains, what moves into the background, and what new work the solution creates.

Workflow Fit

Must-ask Questions

• Which steps in our current workflow disappear, and can you map those changes against our actual process?
• What manual work remains, and what new work does the solution create?
• If staff stop using the AI for one week, what continues automatically, what piles up, and what fails to happen?

Go Deeper if Needed

• Which processes move into the background, and what reporting makes them visible?
• Which gaps typically require configuration or customization?
• How do you measure whether the workflow is improving outcomes, not only whether users complete it?

Strong answers include:	Be cautious if you hear:
A before-and-after workflow map with gaps, handoffs, exceptions, and owners	A generic workflow that assumes your organization will adapt to the product
Improvements staff and people receiving services will notice	Hidden background work with little monitoring or auditability
Metrics for outcomes, burden, adoption, and unintended effects	A “completed workflow” metric that ignores quality, burden, or countermeasures

Integration Strategy

“Yes, we integrate” can describe anything from a robust API to a nightly file transfer or an external link. Make the vendor define what moves, how quickly it moves, and who maintains it.

Must-ask Questions

• What type of integration is this: API, file exchange, embedded workflow, external link, or another method?
• Which data move automatically, which require manual entry, and which never move between systems?
• Who owns implementation and ongoing maintenance on both sides?

Go Deeper if Needed

• Where will users see the product in their current software?
• What data return to the source system, and how quickly?
• Which tasks or documentation will still be duplicated?
• Which privacy and consent rules apply to each data flow, including HIPAA and 42 CFR Part 2 where relevant?

Strong answers include:	Be cautious if you hear:
A specific architecture, refresh schedule, field-level data map, and test plan	“We have an API” without naming the endpoints, data, workflow, or customer effort
Clear ownership for failures, upgrades, vendor changes, and maintenance	An external link described as a seamless integration
Minimal duplicate entry and a realistic plan for exceptions	Unclear ownership when either vendor changes its system

4. Can you implement and sustain it?

Implementation is not easy simply because the vendor has done its part. Estimate the total work required from your teams and what it will take to reach stable adoption after go-live.

Implementation Reality

Must-ask Questions

• What work falls on our team, what work falls on your team, and who owns each decision?
• For a community provider similar to us, how long did implementation, training, and stabilization take?
• What causes implementations to fail today, and what have you changed because of those failures?

Go Deeper if Needed

• Which customer teams most often become bottlenecks?
• What customer characteristics predict success or difficulty?
• What milestones, dependencies, and hard deadlines should be in the implementation plan?

Strong answers include:	Be cautious if you hear:
A responsibility matrix, realistic timeline, named milestones, and explicit dependencies	A timeline based only on the vendor’s configuration work
Lessons from difficult implementations and evidence that the process improved	“Implementation is easy” without estimating customer hours or naming required roles
Change management, training, adoption support, and operational ownership beyond the technical launch	Go-live treated as the end of implementation rather than the start of adoption

Customer Success After Go-live

Must-ask Questions

• What percentage of customers renew, and what percentage of community providers do not renew?
• What parts of the product do customers underuse, and why?
• Can we speak with a similar organization that struggled, not only one of your happiest customers?

Go Deeper if Needed

• How often do customers expand use after the first implementation?
• How do your strongest customers use the product differently?
• What support, optimization reviews, and product education are included after launch?

Strong answers include:	Be cautious if you hear:
Adoption and renewal data separated by relevant customer type	Renewal claims that combine very different customer segments
Candid discussion of churn and specific improvements made in response	References limited to unusually large, well-resourced, or highly satisfied customers
A continuous partnership model with scheduled optimization and outcome reviews	Support that becomes primarily ticket-based immediately after launch

5. Can you trust and govern it over time?

The solution you buy will change. Make sure your organization can understand important outputs, control material changes, respond to performance problems, and override the system when necessary.

Model Governance

Must-ask Questions

• How frequently do the models or decision logic change, and how are changes tested before release?
• Are changes automatic or opt-in, and how are customers informed when functionality or performance changes?
• How do you detect model drift, and what reporting would help us see that performance has changed?

Go Deeper if Needed

• How do you validate new releases over time, not only immediately before launch?
• Where can your staff override the system? Where can our staff override it?
• What happens when monitoring identifies a material drop in performance?

Strong answers include:	Be cautious if you hear:
A structured release process, versioning, documented validation, and rollback procedures	Silent model substitutions or automatic releases with no practical opt-out
Ongoing performance monitoring across relevant customer groups and workflows	Monitoring limited to system uptime rather than output quality
Advance communication and meaningful customer control over material changes	No defined threshold for investigation, rollback, or customer notification

Trust and Explainability

Automation should reduce review burden, not replace one opaque review process with another. Users need enough information to understand important outputs and act when the system is wrong.

Must-ask Questions

• Can users understand why a recommendation or output was produced? Show us what they see.
• Can users override an AI output directly, and what happens to the rest of the workflow when they do?
• How can we report an error, and what service-level agreement applies to review and response?

Go Deeper if Needed

• How do you communicate confidence, uncertainty, or risk?
• Does customer feedback improve our configuration, the global model, or neither?
• Can administrators review patterns across users and workflows without conducting individual chart-by-chart review?

Strong answers include:	Be cautious if you hear:
Explanations that are specific enough to support action, not only generic disclosures	A “because the model said so” answer or an explanation too technical for the intended user
A straightforward correction and escalation process with defined response times	An override that requires a support ticket or does not stop downstream actions
Aggregate monitoring that helps leaders see patterns without recreating manual review at scale	Feedback collection with no clear owner, deadline, or learning process

6. Will it create enough value?

Online return-on-investment (ROI) calculators can create a precise-looking answer from assumptions that do not match your organization. Establish your own baseline and ask the vendor to show how value was measured for customers like you.

Must-ask Questions

• Which key performance indicators improve first, and what baseline, benchmark, and time frame support that claim?
• What assumptions are built into your ROI estimate?
• Which customers did not realize the expected value, and why?

Go Deeper if Needed

• Which customers achieved value fastest, and what did they have in common?
• What baseline data should we collect before implementation?
• What nonfinancial value have customers reported, such as reduced burden, faster access, or more time for higher-value work?

Strong answers include:	Be cautious if you hear:
Customer outcomes supported by defined baselines, evidence, and time frames	Savings based entirely on theoretical staff hours that were never measured or redeployed
A value model that includes customer labor, integration, training, change management, and ongoing maintenance	A single ROI figure applied to every organization
Willingness to discuss where value did not materialize and what remains unknown	Benefits counted immediately while implementation and ongoing costs are excluded

Questions that move vendors off script.

The best buyers do not necessarily ask more questions. They ask questions vendors cannot answer from a standard sales deck. Use a few of these near the end of the conversation to test candor, judgment, and fit.

• What have you had to unlearn after working with community behavioral health providers?
• If you were in my seat, what would concern you most about buying this product?
• What do you not yet understand about our organization that could materially affect our success?
• Which customer assumptions most often turn out to be wrong?
• Which competitors do you lose to, and why?
• What type of organization is not a good fit for this product?
• What would cause you to tell us not to buy the product today?

Listen for: thoughtful answers, productive discomfort, questions back to your team, and a willingness to identify poor-fit situations. Be cautious when every limitation becomes a strength and every organization is described as an ideal customer.

What good looks like.

A credible vendor does not need a perfect answer to every question. It should be able to show what it knows, name what it does not know, and explain the evidence behind the difference.

• Evidence: The product works beyond the demo, with relevant measures, sample sizes, time frames, and customer examples.
• Fit: The vendor understands community behavioral health and can map the solution to your workflows, systems, and constraints.
• Candor: The team can discuss limitations, failures, poor-fit customers, and unresolved questions without retreating into sales language.
• Implementation reality: The plan accounts for the work required from your staff, not only the vendor’s technical tasks.
• Governance: Your organization can monitor, understand, challenge, and override the solution as it changes.
• Value: Expected outcomes are tied to your baseline, measured over a realistic time frame, and weighed against the full cost of adoption.

The strongest vendor may not be the one that promises the most. It may be the one that can tell you, clearly and specifically, where the product works, where it does not, and what it will take to succeed in your organization.