A startup changes the model behind its AI feature. The new model is faster, cheaper and performs better on public benchmarks. The engineering team runs its tests, deploys the update and waits for the improvement.
Instead, support tickets begin to arrive. The assistant is less accurate on short questions. It misunderstands customers who mix languages. Nothing is completely broken, but the product is noticeably worse.
This is one of the difficult realities of building with generative AI: a system can pass every conventional software test while its behaviour deteriorates. The cheapest protection is not another monitoring platform or a more powerful model. It is a small, carefully maintained collection of real examples known as a golden dataset.
AI regressions are different
Traditional software is usually tested against predictable outputs. Give a function a particular input and it should return an exact result.
Generative AI does not work that way. Two answers can use entirely different words and still be equally correct. Conversely, an answer can sound fluent and professional while omitting a critical fact or inventing something the system does not know.
This makes informal testing dangerously attractive. Someone tries five prompts in a staging environment, reads the answers and concludes that the new version “looks better”.
That judgement becomes unreliable as the product grows. A support assistant serving several Southeast Asian markets may encounter English, Thai, Vietnamese and code-switching within the same conversation. Users submit fragments, screenshots, misspellings and requests that the product team never anticipated.
The polished questions used in demonstrations rarely resemble this traffic. A golden dataset makes quality visible. It contains representative user inputs together with a description of what a successful response must and must not do. The team runs the same examples whenever it changes a model, prompt, retrieval pipeline or tool.
It does not replace production monitoring, user feedback or A/B testing. It catches problems before those slower signals arrive.
Begin with real behaviour
The first version does not need thousands of examples. Thirty to fifty inputs from actual usage are enough to begin testing one important product behaviour.
For a customer-support assistant, that behaviour might be answering refund questions, for a financial product, it might be explaining a transaction without offering unauthorised advice, and so on.
Production examples are far more valuable than questions invented during a workshop. They contain the ambiguity, incomplete information and unusual phrasing that expose weaknesses in the system.
Also Read: SEA’s AI boom has a water problem it cannot offset away
Naturally, using production traffic requires appropriate consent, access controls, retention rules and removal of personal information. If the product has not launched, examples from internal testing or a closed beta can be used temporarily. Synthetic examples are useful for getting started, but they should gradually be replaced by real interactions.
The objective is not to construct a perfect benchmark. It is to capture a small but recognisable sample of how people actually use the product.
Test failures, not just the happy path
Many teams create evaluation sets that resemble product demonstrations: clear questions, correct terminology and complete information. Unsurprisingly, the system performs well.
A useful dataset should contain the situations most likely to cause damage. These might include ambiguous requests, unsupported languages, missing account information, contradictory documents, attempts to override instructions or questions that should be escalated to a person. Support tickets and user complaints are often the best source of such examples.
Every production failure should leave something useful behind. Once the immediate problem has been resolved, the interaction should become a new evaluation case. That ensures the same class of failure is less likely to return quietly after the next update.
Over time, the dataset becomes a record of what the team has learned about its users and its product.
Evaluate outcomes, not identical wording
For open-ended AI outputs, requiring an exact answer is usually the wrong approach.
Consider a user asking for a refund. A successful response might need to mention the refund policy, correctly identify the order, avoid promising an outcome and call the account lookup tool before answering. Many different responses could satisfy those requirements.
The evaluation should therefore describe the outcome:
- What information must appear?
- What must never appear?
- Which tool or source must the system use?
- When must the request be escalated?
- Are there limits on length, latency or cost?
Also Read: Thailand’s mobility future will be decided by data, not just vehicles
Some checks are inexpensive and objective. A test can verify whether the response contains a required fact, follows a defined structure, calls the correct tool or stays within a token limit.
Subjective qualities such as tone or clarity may require human review or another model acting as a judge. But expensive AI-based scoring should not be the default. Use the cheapest test capable of detecting the problem.
Both OpenAI’s evaluation guidance and Anthropic’s work on agent evaluations emphasise structured, task-specific evaluation rather than relying on general benchmarks alone.
Keep the dataset small enough to trust
A dataset becomes useless when it is too large for anyone to inspect. Each regression report should show the affected example, the previous response, the new response and the reason it failed. A dashboard announcing that quality fell from 84 to 81 per cent is not enough. Engineers need to see what became worse.
The dataset should also be versioned alongside the product. When expected behaviour changes, the reason should be recorded. A small portion of the examples can be held back from everyday development so the team does not unconsciously tune the system only to the cases it sees.
Finally, someone must own the dataset. Shared responsibility often means no responsibility. The owner should add newly discovered failure modes, remove obsolete examples and ensure evaluations continue to run as the product changes.
A practical starting point
A startup can establish a useful regression process within a week. Choose the single AI behaviour that matters most to customers. Collect 30 to 50 representative inputs. For each one, write down two to four conditions that define a successful outcome. Automate the inexpensive checks and run them whenever the relevant system changes.
AI teams will never eliminate uncertainty completely. Models change, products evolve and users find new ways to surprise us. But a team should always be able to answer one basic question before releasing an update: did this make the product better or worse? A golden dataset is the cheapest reliable way to find out.
—
Editor’s note: e27 aims to foster thought leadership by publishing views from the community. You can also share your perspective by submitting an article, video, podcast, or infographic.
The views expressed in this article are those of the author and do not necessarily reflect the official policy or position of e27.
Join us on WhatsApp, Instagram, Facebook, X, and LinkedIn to stay connected.
The post The cheapest way to stop your AI product from regressing appeared first on e27.