How to plan an AI project that makes it to production
To get an AI project into production, start with one specific task that has a clear before and after, prove a model can do it with real examples before you build anything, and plan data access, testing, cost and human review up front rather than after the demo. In our experience, AI projects stall when they start from the technology instead of a task, or when nobody decides in advance what “good enough” means. Below is the order we work in, from choosing the task to measuring whether it worked.
Start with a task, not with “AI”
“We should use AI” is not a project. “Draft first replies to billing emails so the support team only edits and sends” is.
A good AI task has a clear before and after. Write down who does it today, what goes in, what comes out, how long it takes and how you would tell a good result from a bad one. If you cannot fill that in, the project is not ready yet.
Give the task an owner on the business side too: someone who does or manages the work today and will decide whether the AI version is good enough. Engineers can build the feature, but they are rarely the right people to judge a support reply or a contract summary. Without that owner, projects tend to drift from one impressive demo to the next without ever being signed off.
Tasks that tend to work well:
- Sorting or tagging incoming messages, tickets or documents.
- Drafting text a person reviews, such as replies, summaries or first versions of reports.
- Pulling fields out of documents, such as invoice details, into a system.
- Answering questions from a fixed set of your own documents.
Tasks that tend to struggle: ones with no agreed right answer, ones where every mistake is costly and nobody checks the output, and ones that happen too rarely to justify the work. It is also worth checking whether an off-the-shelf tool already does the job. Our article on whether to build or buy software covers that decision.
Test it in a chat tool first
Before anyone writes code, take a set of real examples of the task, including the awkward ones, and try them in a chat tool such as Claude. Write the instructions as you would brief a capable new colleague: what the task is, what good looks like, what to avoid.
Then read the results honestly. Where it fails, is that because the instructions were unclear, because it was missing information, or because the task is too hard? Adjust and try again.
This step costs very little and answers the biggest question in the project. If a model cannot do the task well in a chat window, building software around it will not change that. Stopping here is a good outcome, because it saved you a build.
One caution: only paste examples you are allowed to share with that tool. Check your company’s policy, and remove personal or confidential details first.
What data does the AI need, and who can see it?
Most useful AI features need your own information: customer records, product details, policies, past tickets. List what the feature needs, where each piece lives and whether it is kept up to date.
Then decide on permissions. A good rule is that the AI should only see what the person using it is allowed to see. If a support agent cannot open payroll records, neither should the assistant answering their questions.
Also decide what is never sent to an AI provider at all, and read the provider’s terms for business use, including how long data is kept. Business terms often differ from those of consumer chat apps.
If the information the feature needs is scattered across several systems and disagrees with itself, fix that first. Our guide to what a data platform is explains how companies bring it together.
Build an evaluation set before you build the feature
An evaluation set is a fixed list of real inputs, each paired with the answer you would accept. It is the exam the system has to pass.
Include easy cases, hard cases, unusual ones and cases where the right response is to refuse or hand over to a person. Where the task has a right answer, such as a category or an extracted field, ordinary code can score the results. Where it involves writing, agree a short scoring guide and have people score a sample.
Agree the pass mark with the person who owns the task before building starts. Then run the set after every change: a new prompt, a new model, a new data source. Without it, nobody can say whether a change made things better or worse.
What will it cost to run?
AI features cost money each time they run, and the bill grows with use. The main levers are the size of model, how much text you send each time, and whether work can be cached or batched. Before approving a build, estimate the cost per use from your real examples, multiply by expected volume, and agree a spending cap. Our article on what it costs to run AI features explains what drives the bill and how we keep it predictable.
How do you stop it making things up?
AI models can state false things with confidence. The defence is to give the model the facts it should work from, check its output with something other than another AI, and hold back anything that fails.
In FFSN, the fantasy football product we build and run ourselves, every article starts from a block of facts pulled from the league’s data. After it is written, a checker that is ordinary code compares every name, number and quote against those facts, and anything that does not match is removed or held for review. Our article on how we stop AI from making things up describes each layer and its limits.
Roll out with a human in the loop
Do not switch straight from “demo” to “fully automatic”. Move through stages:
- Shadow mode. The AI runs on real work, but its output is not used. Compare it with what people actually did.
- Draft mode. The AI drafts and a person approves, edits or rejects every output.
- Partial automation. Low-risk cases go through on their own, and the rest still go to a person.
At every stage, keep a way to switch the feature off and fall back to the old process. Record what reviewers change. Their edits show where the system is weak, and the best ones become new examples in the evaluation set.
How do you measure whether it worked?
Go back to the before you wrote down at the start and measure against it. Useful measures include time spent per task, how often drafts are accepted as they are, edited or rejected, errors found in review, cost per task and what the people using it say.
Keep measuring after launch. Your data changes, and providers release new model versions. Re-run the evaluation set when either happens, so a drop in quality shows up in your tests before it shows up with your customers.
Where to start
Write one paragraph describing the task: who does it today, what goes in, what comes out and how you would judge a good result. Then gather real examples and try them in a chat tool. If the results hold up, our AI integration service covers taking it from that test to a feature your team relies on, with the evaluation, cost controls and review built in. You are welcome to get in touch with your paragraph.
Questions people ask
Why do AI projects fail to reach production?
They often start from the technology rather than a specific task, skip testing with real examples, or never agree what good enough looks like. Defining the task, building an evaluation set and planning human review before building avoids most of these problems.
How do you test an AI idea before building it?
Paste a set of real examples of the task into a chat tool such as Claude, with instructions written as you would brief a new colleague, and judge the results honestly. If a model cannot do the task well there, building software around it will not fix that.
What is an evaluation set in an AI project?
An evaluation set is a fixed list of real inputs, each paired with the answer you would accept. You run the AI system against it before launch and after every change, so you can see whether quality went up or down.