When an AI feature in an existing product pays off, and why most pilots stall
Most AI features never make it past the pilot. MIT NANDA’s 2025 study found that only 5% of organisations got a task-specific AI tool into production with a lasting effect on productivity or profit. The model is rarely the reason. The usual reasons are a feature attached to no real task, a cost nobody priced per user, and a success metric nobody wrote down before the build.
This article is for founders and product owners who already have a working product and are asked, often by their own board, “where is our AI feature?”. We show how we decide whether one is worth building, how to scope the first one small, and which four numbers tell you whether it pays.
Where AI features stall
The drop happens between the demo and daily use. In the MIT NANDA sample of 52 organisations, 300 public deployments and 153 surveyed leaders, 60% of companies looked at embedded or task-specific AI tools, 20% ran a pilot and 5% got one into production with a measurable effect. The figures come from interviews and surveys, so they are directional rather than audited. The authors call the cause a learning gap: tools that do not fit the workflow, forget context and do not improve with feedback.
Larger surveys tell the same story from the profit side. In McKinsey’s State of AI 2026 (1,719 respondents), 37% say AI has contributed to EBIT at all, essentially unchanged from 2025, and only 6% qualify as high performers. BCG’s 2025 survey of 1,250 executives found 60% of companies report minimal revenue and cost gains from AI. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027 because of rising costs, unclear business value or weak risk controls.
None of this says AI features do not work. It says that a feature without a clear job, a price and a metric usually ends as a demo.
Where an AI feature does pay off
The features that survive have something in common: they take over a step users already do, often and by hand, and the result is easy to check. MIT found the clearest savings in back-office work, such as document processing and replacing outsourced tasks, even though about half of AI budgets go to sales and marketing. In a product, the equivalent is the tedious step inside a workflow your users already pay for.
A feature is likely to pay off when it:
replaces a frequent manual step
produces output that is easy to check or undo
sits inside an existing workflow
saves time you can count
costs little when it gets an answer wrong
A feature is likely to stall when it is a chat box added to the menu, when users cannot tell whether its answer is right, when it needs a new habit to be useful, when its value is described as “engagement”, or when one error turns into a support ticket.
A good first candidate is usually boring: summarising a long record, drafting a reply the user edits, sorting incoming items, filling a form from a document. Each has a clear before and after, so you can measure it.
Price it per action before you build it
Inference keeps getting cheaper. Stanford’s AI Index 2025 found that the cost of running a model at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024. Yet cost still matters: in McKinsey’s 2026 survey about 20% of respondents said AI operating costs, including tokens, limited their use of AI. The price per token is low. The number of actions per user per month is what makes it expensive.
We price every candidate feature as cost per action, then multiply by realistic usage. Take a feature that reads about 3,000 tokens of context and writes about 500, used 20 times a working day over 22 working days. At Anthropic’s list prices in November 2026, with no caching or batch discounts, the model cost per active user per month comes to about $0.24 on a small model (Haiku 5.5), $4.84 on a mid-size model (Sonnet 5.5) and $19.36 on the largest (Fable 5.2). The model you choose changes the monthly cost per user roughly eightyfold.
Put that number next to what a user pays you. On a $30 seat, $0.24 is noise and $19 eats most of the margin. The usual answer is to test the task on the smallest model that does it well enough, and to send only the hard cases to a larger one.
Build, or call an API
For a first feature in an existing product, the choice is rarely between training your own model and buying one. It is between calling a hosted model through an API and adopting a vendor’s ready-made tool. MIT reports that externally built, customised tools reached deployment about 67% of the time against about 33% for internal builds, with the caveat that this comes from self-reported interviews. Our practice for products is a middle path: call a hosted model through a thin layer you own, so you can switch providers when prices or quality change, and keep your prompts, test cases and data on your side.
Measure four numbers from day one
Decide before launch what result keeps the feature alive. We track four numbers, and each one has a value that tells you to stop:
Adoption: the share of eligible users who use the feature weekly. Warning sign: it stays flat after the first month.
Task completion: the share of uses where the user keeps the result. Warning sign: most outputs are rewritten or discarded.
Time saved: minutes per task against the old way. Warning sign: no difference you can measure.
Cost per action: model and infrastructure cost per use. Warning sign: it rises faster than usage or revenue.
Adoption without completion means curiosity. Completion without time saved means users are polite. Only when all four hold do you have a feature worth scaling.
How we scope a first AI feature
Week 1, pick one task: frequent, manual and easy to check. Write down today’s time and error rate.
Week 2, price it: cost per action on two or three models, against revenue per user.
Weeks 3 to 5, build small: one entry point, a fallback to the manual path, and logging of every output.
Weeks 6 to 8, ship to a slice: a share of users, with the four metrics on a dashboard.
Week 9, decide: scale it, change the model, or remove it.
The timings are a planning guide, not a promise. The point is that the decision date is set before the first line of code.
What to take from it
Start from a task, not from a model. Name the manual step the feature removes and how often users do it.
Price it per action and per user. Compare the monthly model cost with what that user pays you.
Use the smallest model that does the job. Route only hard cases to larger models.
Write the kill criteria first. Adoption, completion, time saved and cost per action, with a date to decide.
Keep a manual fallback. Users forgive an AI feature that is sometimes wrong if the old path is one click away.
If you are weighing an AI feature for a product you already run, we are happy to help you price and scope it. Talk to us: https://www.commonsaltx.com/contact
