How to Evaluate an AI Project Before You Build It

AI projects rarely fail for technical reasons. The model usually works about as well in month six as it did in the demo. What fails is everything around it: unclear value, missing data, no owner, no plan for being wrong. All of that is visible before a dollar is spent, if you ask the right questions.

1. What decision or work product does it change?

"We want to use AI" is not a project. "We want incoming COAs read and their values entered into the system without a person retyping them" is a project. The test: can you name the specific hours saved, the error eliminated, or the decision improved? If the value statement does not mention a workflow, the project is a demo in search of a purpose.

Corollary: estimate the volume. A task that takes four hours a month does not justify a build, no matter how elegantly AI could do it.

2. Does the data exist, and can the system reach it?

Every AI system is downstream of its inputs. Before building, answer plainly:

• Where does the input data live today, and is there an API or export?
• Is it clean enough that a human could do the task from it? (If a person cannot, a model cannot.)
• Are there permission or privacy constraints on moving it? In regulated industries this question kills or reshapes more projects than any technical issue.

A surprising amount of "AI readiness" is just data plumbing, and it is often worth doing even if the AI part never ships.

3. What happens when it is wrong?

It will be wrong sometimes. The project's viability depends on what that costs:

• Cheap errors, easily caught: drafting, summarizing, internal search. Ship with light review.
• Costly errors, catchable in review: compliance-sensitive content, financial entries, customer replies. Ship with a mandatory human gate, and design the review screen as carefully as the AI.
• Costly errors, invisible until damage: silent data corruption, automated actions with no audit trail. Redesign the project until errors become visible, or do not build it.

If the error analysis was never done, the review step gets discovered after the incident instead of before.

4. Who owns it in month six?

An AI system is a living process. Prompts drift out of date, an API changes, volume doubles, a new document format shows up. Someone inside your business needs to own the system: watch its metrics, review its exceptions, and decide when it needs adjustment. If no one can be named, the project will decay quietly no matter how well it launches.

Scoring it honestly

We suggest a simple bar: a named workflow with real volume, reachable data, an error plan appropriate to the stakes, and a named owner. Four for four, and the project is probably worth a prototype sprint. Anything less, and the cheapest move is to fix the missing piece first. The most expensive AI project is the one that answers these questions after launch.