Championships Are Won in the Garage
Display None Text
What Formula 1 can teach us about building high-performance software architecture that remains fast, reliable, and efficient when the pressure is on.
TORONTO, CA
The real test of an AI system begins when the perfect prompt, curated data, and friendly audience disappear.
There is a familiar moment in almost every AI demonstration.
A carefully selected question appears on the screen. The system searches a collection of documents, pauses briefly, and returns a polished answer with confident language, useful detail, and perhaps even a few citations.
The room becomes noticeably more interested.
Someone asks how quickly it can be deployed. Another begins imagining the work it could automate. For a moment, the system feels ready.
But the demonstration has only shown what happens under favourable conditions.
The prompt was clear. The data was prepared. The environment was controlled. There were no conflicting documents, missing permissions, malicious instructions, failed tools, or hundreds of users arriving at the same time.
The demo showed the system on its best day.
Production will test it on every other one.
That is why LLM evaluation cannot be reduced to reading a handful of responses and deciding whether they sound intelligent. It must determine whether the complete system remains accurate, grounded, secure, responsive, and economically practical when real work becomes messy.
Traditional software usually fails visibly. A page crashes, an API returns an error, or a transaction does not complete.
AI systems can fail while sounding completely reasonable.
They may retrieve the wrong policy, omit an important exception, combine incompatible sources, invent a missing detail, or provide information the user is not authorized to see. The response may still be fluent, confident, and professionally written.
That is the central risk.
An AI system should not be judged by how intelligent its answer sounds. It should be judged by whether it completes the intended work correctly, with evidence and acceptable risk.
Before testing anything, teams must therefore define what “good” means.
What task should the system complete? Which errors are inconvenient, and which are dangerous? When should it answer, ask for clarification, refuse, or involve a human? How quickly must it respond? What is an acceptable cost per completed task?
An internal knowledge assistant, a healthcare copilot, and an agent updating customer records may use similar models, but the consequences of failure are very different.
A meaningful LLM testing framework begins with the business outcome, not the model score.
The next step is a representative test set, often called a golden dataset.
Despite the name, it should not be clean or idealized. It should reflect the way people actually work.
Real users omit details, use unfamiliar terminology, ask several questions at once, refer to outdated information, and assume the system understands context they never provided.
A useful golden dataset should include:
Each scenario should define more than an ideal sentence. It should identify the evidence required, the permitted sources, the expected outcome, and the conditions under which the system should stop.
This is especially important for AI hallucination testing. Sometimes the correct answer is not an answer at all. It may be an admission that the evidence is missing.
When a retrieval-augmented generation system gives a poor answer, teams often begin changing the prompt.
But the model may not be the problem.
The system may have retrieved the wrong document, missed the decisive section, selected an outdated version, or divided the source into chunks that destroyed its meaning.
This is why RAG evaluation must separate retrieval from generation.
Two measures are particularly useful:
Poor precision surrounds the model with noise. Poor recall deprives it of essential evidence. Both can produce polished but unreliable responses.
Good RAG accuracy testing should verify that the correct documents were found, permissions were respected, current versions were used, and the strongest evidence appeared before irrelevant material.
Otherwise, teams may spend weeks improving prompts to compensate for a retrieval problem.
An answer can be factually correct and still be unsuitable for enterprise use.
Suppose the system gives the right policy answer, but the cited document does not contain the claim. The model may have relied on its general training or unrelated context.
That answer is correct, but it is not grounded.
There are several levels of quality:
These should be evaluated separately.
Exact checks work well for dates, identifiers, calculations, classifications, and tool parameters. Semantic scoring can compare answers where several wordings may be valid. Expert review remains essential for complex or high-risk decisions.
An LLM-as-a-judge can help evaluate responses at scale, but it should never become the only authority. A judging model has its own biases and inconsistencies.
The strongest approach combines deterministic checks, reference-based scoring, model-based evaluation, expert review, and production feedback.
Citations also need direct testing. A citation icon does not prove that the source supports the claim beside it.
The evaluator must confirm that the cited passage contains the evidence, applies to the correct context, and can be verified by the user.
A dependable AI system does not attempt to answer everything.
It recognizes missing evidence, ambiguous intent, permission limits, and situations that require human approval.
Evaluation should test whether the system invents facts, overstates confidence, refuses harmless requests, exposes restricted information, or continues acting when it should stop.
The balance matters.
A system that answers too freely becomes dangerous. A system that refuses too often becomes useless.
The goal is not maximum confidence or maximum caution. It is appropriate behaviour for the situation.
AI security is not limited to authentication and database permissions.
A user may attempt to override the system’s instructions. A retrieved document may contain a hidden prompt injection. An agent may be manipulated into calling the correct tool with unauthorized parameters.
LLM security testing should therefore include:
Role-based access control must be tested through real conversations and workflows. It is not enough for the database to have permissions if the AI layer can still retrieve, summarize, or infer restricted information.
Evaluating an AI agent requires more than scoring its final response.
An agent may understand the request, choose the right tool, and still fail because it passes the wrong customer ID or updates the wrong record.
AI agent evaluation should measure the complete workflow:
This is where AI agent observability becomes essential. Teams need visibility into retrieved context, decisions, tool calls, retries, errors, and final state—not only the message shown to the user.
Time to first token affects how responsive the system feels. Total latency shows how long the work actually takes. Concurrent-user testing reveals where performance begins to degrade.
Cost should be tied to successful outcomes.
The useful question is not simply, “How much did the model call cost?”
It is, “How much does it cost to complete one valid business task?”
In our work supporting Deko Automotive’s AI transformation, evaluation was treated as part of the system—not as a final checkpoint.
The process included golden datasets, deterministic checks, reference-based metrics, LLM-as-a-judge assessments, retrieval precision and recall, groundedness testing, guardrail validation, role-based access testing, security checks, time-to-first-token analysis, throughput measurement, and offline and online evaluation.
The objective was not to create one impressive score.
It was to create several independent signals capable of exposing different failures.
Retrieval metrics could reveal missing evidence. Groundedness tests could expose unsupported claims. Security checks could uncover permission weaknesses. Performance testing could show that an otherwise accurate workflow became unusable under load.
One score tells a reassuring story.
A layered evaluation system tells the truth.
AI systems do not remain static.
Documents change. Prompts are revised. Models are upgraded. Tools evolve. Users discover new behaviours. New security threats emerge.
A system that performed well three months ago may behave differently today.
That is why deployment should be followed by regression testing, production monitoring, failure analysis, user feedback, security reviews, and periodic expert evaluation.
Offline testing provides control before release. Online evaluation reveals what happens in real workflows.
Both are necessary.
The next time an AI system produces a flawless response in a meeting, the right question is not only, “How quickly can we launch it?”
Ask what happens when the documents disagree. When the user lacks permission. When the tool fails. When the evidence is incomplete. When hundreds of people use it at once.
Those questions do not weaken the promise of AI.
They are what allow it to survive contact with the real world.
Do not wait for real users to discover where your AI system fails. Evaluate its accuracy, groundedness, security, workflow behaviour, and performance before it becomes business-critical.
What Formula 1 can teach us about building high-performance software architecture that remains fast, reliable, and efficient when the pressure is on.
The strongest systems are rarely made from extraordinary parts. They become extraordinary through the way those parts are connected.
The real test of an AI system begins when the perfect prompt, curated data, and friendly audience disappear.
Before AI can understand your business, your business has to understand where its knowledge lives.
Automation is a multiplier. Before making a process faster, make sure it deserves to be multiplied.
We learned to talk to artificial intelligence. Now we are teaching it how to move work forward.
The smartest software strategy is not choosing between build and buy—it is knowing what your business should own, what it should borrow, and what it should connect.
