How to evaluate AI vendors when you're not a tech company

If you run a development company, a GC, or a lender, you are about to be sold software. A lot of software. Demos that all look good, pricing that's "happy to discuss," and ROI claims that conveniently can't be measured.

You don't need to become a CTO. You need a framework that filters most vendors out in 30 minutes so you can spend time on the ones that matter.

Here's the one we use.

Step 1: Anchor on the workflow, not the demo

Most demos walk you through what the software can do. That tells you almost nothing about whether it will work in your workflow.

Before you take a demo, write down — in plain English — the steps of the workflow you want to change, who does each step today, and what the failure modes are. Then make the vendor demo against that workflow, not their canned script.

If they can't, or they get hand-wavey when you ask them to, that's your answer.

Step 2: Score four dimensions, not features

Feature checklists are vanity. Score every vendor on four dimensions:

Fit. Does the workflow they're built for actually match yours? A platform built for institutional REIT operations may not fit a value-add owner. A construction monitoring tool built for ground-up may not fit TI work. Most "no" decisions should be made here.

Data. What do they need from you to work? Where does it sit, who owns it, how does it integrate? "API-first" is marketing language. The right question is: who is going to physically wire this up, and how long will that take?

Adoption. Who at your firm has to learn this, and what's their incentive? If it requires a superintendent to log into a new system at 6:30am, you have an adoption problem the demo won't reveal.

Defensibility. What stops a competitor from doing the same thing six months from now? You're not buying technology, you're buying a relationship and a roadmap. If the vendor can't articulate why they're hard to copy, the price will collapse and so will their support.

Score each 1–5. Anything under 3 in any dimension is a no, no matter how the others scored.

Step 3: Demand a real reference call, not a logo slide

Logo slides on the website are nearly worthless. Vendors will surface their three happiest customers; everyone else is paying.

Ask for a reference call with a customer of similar size, similar workflow, and at least 12 months in. On that call, ask:

• What were the three biggest surprises?
• What share of your team actually uses it daily?
• What's the support like when things break?
• What would you do differently?
• Would you renew tomorrow at full price?

If the vendor won't give you that call, that's also your answer.

Step 4: Insist on a paid pilot with a measurement plan

Free pilots are a trap. The vendor over-services them, the team gets the executive demo experience, and at month three you can't tell what's the product and what's the white-glove service.

Pay a fair price for a defined pilot. Set:

• A baseline metric you measure today (cycle time, error rate, FTE hours)
• A target improvement at 90 days
• A clear go/no-go decision criterion
• A rollback plan if it doesn't work

Then run it like a project. Weekly checkpoints, named owner on your side, documented decisions.

Step 5: Read the contract like an operator

Two clauses matter more than the price:

Data ownership and portability. Your data is yours. The contract should say so explicitly. If you cancel, can you export everything in a standard format? Within how many days?

Service levels and exit. What happens when something breaks? What's the response time? Can you exit without penalty if SLAs aren't met? What does a 90-day off-ramp look like?

If a vendor is healthy and confident in their product, both should be standard. If they're not, you've learned something useful.

Step 6: Decide who owns the result

Last and most important: when the dust settles, who at your firm is responsible for whether this works?

If the answer is "the vendor's customer success manager," you will not get adoption. If the answer is a named senior person at your firm with the authority to change workflows, you have a chance.

Most failed AI rollouts we've reviewed in distressed-project resolution work fail at this step, not the technology step.

---

The bottom line

You don't need a CTO to evaluate AI vendors well. You need a structured process, a clear-eyed read of your own operation, and the discipline to say no to good-looking demos that don't fit.

Most of the value of an AI advisory engagement is exactly this: running this process across vendors so your team can stay focused on running the business. We're vendor-agnostic, paid by you, and our recommendations are documented and defensible.

If you'd rather run it yourself, this framework is a fine starting point.

---

Jerod Bauer, CCM, is the founder of Waypoint Advisors. He spent 20+ years in construction operating and lending advisory roles, including evaluating dozens of construction technology vendors in his Director of Construction role at Hall Structured Finance.