A pilot that "works in the demo" is the easy part. Pilots die in the boring middle: when the data drifts, the champion gets pulled, and nobody owns the eval.
There's something about AI pilots that doesn't get talked about enough.
Most people expect them to fail because the model gives bad answers, the prompts aren't good enough, or the infrastructure can't handle the workload.
Sometimes that's true. But often, the real problems have very little to do with the model itself.
Three months in, the person who pushed for the project is busy with something else. The data has changed since the system was built. Nobody is checking whether the agent's answers are getting worse, and the team doesn't even realize there's a problem until someone complains.
The demo looked great. The system worked. But somewhere along the way, nobody was looking after it.
I've been thinking about what separates AI pilots that actually make it into production from those that quietly get abandoned. It usually comes down to four habits.
Before you spend weeks experimenting with models and prompts, answer one simple question:
How will you know if your AI system is actually doing a good job?
If you don't have a clear answer, you're building without a way to measure success.
Start with a collection of real questions your system will encounter, the answers you'd consider correct, and a repeatable way to evaluate its responses. Turn that into an evaluation harness before the project gets too far along.
It doesn't need to be complicated. It just needs to give you a reliable baseline.
Here's why this matters.
A better model comes along next month. You switch to it, run a few examples, and the answers look better. But what about the other 500 cases? Did accuracy improve? Did the model start making mistakes it didn't make before? Did latency or cost increase?
Without evaluations, you're guessing.
With them, you can test changes before they reach users and catch regressions before they become production problems.
One of the biggest mistakes teams make is treating evaluation as something they'll add once the system is working. By then, they're already dealing with production issues, and it's much harder to establish what good performance should have looked like in the first place.
Don't wait until the system breaks to figure out how to measure it.
It's tempting to look at a business and imagine all the ways AI could improve it.
Automate operations. Rebuild customer support. Make the entire back office intelligent.
The problem is that these goals are too broad to build, test, or measure effectively.
Instead, find one specific workflow that wastes time or creates unnecessary friction for a team.
For example:
These are small enough to understand, build, and measure without turning the project into a six-month transformation initiative.
More importantly, you can establish a baseline. How long does the task take today? How often do people make mistakes? How much of the work can AI handle without human intervention?
Once the system proves its value, you have a concrete reason to expand it.
You don't need to convince the entire organization to adopt an AI strategy. You can show one team that a particular problem is now cheaper or faster to solve.
Then you move to the next problem.
Start with a workflow you can explain in one sentence, not a vision that needs a presentation deck.
This is probably the least exciting part of building AI systems, which is exactly why teams tend to underestimate it.
Everyone wants to build the agent. That's the visible part. You can demonstrate it, show it using tools, and put it in front of a customer.
But the agent is only as useful as the information it can access.
Business data is usually scattered across databases, internal tools, spreadsheets, CRMs, support platforms, and legacy systems. Some records are duplicated. Others are outdated or inconsistent. Different systems may describe the same thing in different ways.
You can build a sophisticated agent on top of that mess, but you're going to spend a lot of time fixing problems that have nothing to do with the model.
A better approach is to build the data foundation alongside your first use case.
Start with the systems your workflow actually needs. Connect them, clean up the relevant data, establish consistent schemas, and make the information accessible through infrastructure your team can maintain.
That could mean Postgres, object storage such as S3, or a table format such as Apache Iceberg. The right choice depends on your existing stack and requirements.
You don't need to build an entire data platform on day one. You just need to avoid treating every new AI project as a completely separate integration exercise.
Because once the first workflow works, you'll probably want to build another one.
If you've already established reliable connectors and reusable data pipelines, the next project becomes much easier. If you haven't, you're back to connecting systems, cleaning data, and solving the same problems all over again.
The goal isn't to build the biggest data platform. It's to make the next AI project easier to ship than the last one.
AI is moving quickly enough that there's always something new to try.
A new agent framework. A different orchestration pattern. A new way to manage tool calls. A model that promises better reasoning at a lower cost.
It's worth paying attention to these developments. But adopting every new tool is not the same as building a better system.
Once an AI application starts doing real work, the fundamentals of software engineering still apply.
You need code reviews, automated tests, staging environments, version control, monitoring, and a way to roll back changes when something goes wrong.
You also need clear ownership. Someone should be responsible for monitoring the system, investigating failures, and deciding when changes are safe to deploy.
And because AI systems can behave unpredictably, you need to monitor more than whether the application is running. You should also track answer quality, failure rates, latency, cost, and how often people have to correct the system.
The exact setup will depend on what the agent does. An internal assistant that summarizes documents doesn't need the same safeguards as an agent that changes customer records or executes financial transactions.
But the principle remains the same: the more responsibility you give an AI system, the more seriously you need to treat its reliability.
The model might be new. The engineering discipline shouldn't be.
If you're building an AI pilot today, here's where I'd start:
1. Evaluate first. Define what a good result looks like and build a repeatable way to measure it.
2. Pick a narrow use case. Solve one real problem for one team before trying to transform an entire business function.
3. Build reusable data infrastructure. Connect the systems your use case needs, and make those integrations useful for future projects too.
4. Engineer for production. Test, monitor, review, and maintain the system as you would any other piece of software that people depend on.
None of this is particularly flashy. That's the point.
I think we spend too much time discussing how to make AI agents more capable and not enough time discussing what it takes to keep them useful.
A model can improve. An agent can gain new tools. An orchestration framework can become more efficient.
But none of those improvements will save a project if the data is unreliable, nobody measures performance, and no one is responsible for maintaining the system.
The teams that get the most out of AI won't necessarily be the ones that experiment with the most models or build the most complicated agents.
They'll be the ones that learn from each deployment, reuse what they've already built, and make every subsequent project easier to deliver.
Getting the first pilot to work is an achievement.
Building a system that makes the second, third, and tenth pilots cheaper, more reliable, and easier to maintain is where the real value starts.
Written by
Founder | CEO
