The short answer: Evaluation is a gate you pass before an AI system touches live work, measured on a labeled set against a threshold you agreed in advance. Observability is the record of what the system actually did afterwards. They are usually bought as separate tools and treated as separate projects, which is the mistake. Capture the right handful of fields while the work happens and the same record serves the engineer debugging on Tuesday and the auditor asking in eighteen months.
Why does this matter to the accountable operator?
Because the expensive version of this is reconstruction.
An auditor, a customer’s security team, or your own legal counsel asks a question about something an AI system did nine months ago. Which model version handled it. What it was given. What it produced. Whether a person reviewed it, and who. If the answer has to be assembled from application logs that were never designed to answer it, you are paying engineers to do archaeology, and the answer you eventually produce is an argument rather than a record.
The cheap version is not a bigger logging budget. It is deciding, before deployment, which fields turn an ordinary trace into an answer, and capturing those.
There is also a regulatory floor underneath this for anyone in scope of the EU AI Act. Article 12(1) requires that high-risk AI systems “technically allow for the automatic recording of events (logs) over the lifetime of the system,” and Article 12(2) ties that logging to identifying risk situations, supporting post-market monitoring, and monitoring the system in operation. Article 19 then requires providers to keep those logs for a period appropriate to the purpose, and “at least six months” unless other law says otherwise. That is a design input, not a compliance afterthought. A system that cannot produce those records is not a system with a documentation gap, it is a system that was built wrong.
What is the difference between evaluation and observability?
They answer different questions at different times, and confusing them is why organizations end up with neither.
Evaluation asks: is this good enough to switch on? You build a set of real past cases with known correct answers, agree what accuracy and exception rate you will accept, run the system against that set, and record the result. It happens before go-live, it produces a number, and the number is the gate. If the system does not clear the threshold, it does not go live. That is the whole mechanism, and it is what lets the accuracy conversation happen in numbers rather than adjectives.
Observability asks: what is it doing now, and what did it do then? It is continuous, it runs in production, and it produces a record per operation rather than a score.
Most teams that buy an “AI observability” product get dashboards of latency, token spend and error rates. Useful for running the thing. Almost useless for proving anything, because the fields that answer a governance question were never captured.
What does an evaluation gate actually look like?
Four things, and none of them are a tool purchase.
A labeled set built from your own past cases. Not a public benchmark. A public benchmark tells you how a model performs on somebody else’s problem. Fifty to a few hundred of your real cases, with the answer a competent person gave, is worth more than any leaderboard.
A threshold agreed before you see the result. This matters more than the number itself. A threshold set after the run is not a gate, it is a rationalization. Write down the accuracy and the acceptable exception rate while you still have no idea what the system will score.
A documented run against that set. Date, model version, configuration, result, and who signed it.
A rule about what happens on failure. If the system does not clear the threshold, it does not go live. If that sentence is negotiable, you do not have a gate.
Note what this makes possible. Confabulation, which is the term NIST uses in AI 600-1 for a model stating something false with confidence, stops being a thing you worry about in the abstract and becomes a rate on a set. You can argue about a rate.
What has to be in the record while it runs?
Here the open standards work has done you a favor. The OpenTelemetry project maintains vendor-neutral semantic conventions for generative AI, which means there is a common vocabulary emerging for naming what an AI call did, rather than every tool inventing its own. The work is still moving, so treat the specifics as a direction rather than a frozen standard, but the shape is settled enough to build on. Spans carry attributes including gen_ai.request.model for the model used, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for consumption, and gen_ai.response.finish_reasons for why generation stopped.
Those are the engineering fields. They are necessary and they are not sufficient, because none of them answers a governance question. For that you need a small number of additions that almost nobody captures by default:
- Which human was accountable, and did they approve. The identity of the reviewer and the decision, recorded at the moment it was made rather than inferred later from a ticket.
- Which rule or risk tier applied. The classification that determined whether this action needed a person at all.
- What the system was permitted to reach. The scope in force for that operation, so a later question about access has an answer.
- A stable identifier tying the operation to the business record. The matter, the claim, the account. Without it, you have a trace nobody can connect to anything that matters.
- The model version and configuration in force. “We upgraded in April” is not an answer when the question is about March.
What should you deliberately not log?
This is the part that gets skipped, and in a regulated setting it is the part that creates a new problem while solving an old one.
The OpenTelemetry conventions include optional attributes for the content itself, covering system instructions, input messages and output messages. Capturing those is genuinely useful for debugging. It is also how a healthcare organization ends up with protected health information sitting in a monitoring platform that has no business associate agreement, or how a firm’s privileged material ends up in a vendor’s cloud outside the matter it belongs to.
The rule we apply: log the metadata by default and the content by exception. Enough to prove what happened, reach the underlying record if someone is entitled to, and reconstruct a decision. Not a verbatim copy of the sensitive material in a second system with a different access model and a different retention schedule.
When content capture is genuinely needed, it is a deliberate decision with its own scope, its own retention, and its own answer to the question of who can read it. Turning it on because the vendor documentation showed it in an example is not that.
How long do you have to keep it?
Long enough to answer the question that has not been asked yet, which is longer than most retention defaults.
For organizations in scope of the EU AI Act, Article 19 sets a floor: providers of high-risk systems keep the automatically generated logs for a period appropriate to the intended purpose and at least six months, unless other Union or national law requires otherwise, and data protection law is explicitly named as one of those. Read that as a floor and not a target, because your sector’s own retention rules usually run longer, and the awkward questions tend to arrive after the six months are up.
The practical consequence is architectural. If the evidence lives in a monitoring tool with a thirty day rolling window because that is what the pricing tier includes, you have a six month obligation and a thirty day memory. Decide where the durable copy lives, and decide it before the first audit rather than during one.
How does this become audit evidence without a second project?
By noticing that the two audiences want the same thing.
An engineer debugging a bad output wants to know what the model was given, what it returned, which version was running, and what the system was allowed to touch. A reviewer establishing what happened wants exactly that, plus who approved it and under what rule. The overlap is almost total. The gap is the handful of governance fields above.
That is the entire argument for capturing evidence during execution rather than assembling it afterwards. Assembled evidence is a project with a deadline, and it is weaker: it is a reconstruction, it depends on the people who were there, and it gets more expensive every month you wait. Captured evidence is a byproduct of running the system properly.
It also lines up with what the frameworks already ask for. The measure function in the NIST AI Risk Management Framework is about analyzing and tracking identified risks with the metrics to back it, and NIST is explicit that “we cannot measure this yet” is itself a finding worth recording rather than a gap to paper over. An ISO/IEC 42001 management system expects monitoring, internal audit and management review to run on something. None of those frameworks tells you which fields to capture. That decision is yours, it is made once, and it is made before deployment or not at all.
For how the evidence trail fits the wider governance structure, see our AI governance framework and the detail on what an audit trail has to capture. What certification does and does not prove about any of this is covered in ISO/IEC 42001 certification.
Frequently asked questions
What is the difference between AI evaluation and AI observability?
Evaluation answers whether a system is good enough to deploy. It runs before go-live against a labeled set of your own past cases, measures against a threshold agreed in advance, and can refuse the deployment. Observability answers what the system actually did in production. It runs continuously and produces a durable record per operation. Evaluation is a gate that can say no; observability is a record that answers questions later. A system needs both, and they are usually treated as separate projects when they are two halves of the same evidence problem.
What does the EU AI Act require you to log?
Article 12(1) requires that high-risk AI systems technically allow for the automatic recording of events over the lifetime of the system, and Article 12(2) ties that logging to identifying situations where the system may present a risk, supporting post-market monitoring, and monitoring operation. Article 19 requires providers to keep those logs for a period appropriate to the intended purpose and at least six months, unless other Union or national law provides otherwise. Treat six months as a floor, because sector retention rules and data protection law often set a different period.
Should we log the prompts and responses themselves?
By default, no. Log the metadata and treat content capture as an exception with its own justification, scope, retention and access rules. Verbatim prompts and outputs frequently contain the sensitive material the rest of your controls exist to protect, and copying them into a monitoring platform creates a second store with a different access model, a different retention schedule, and often a different vendor agreement. Metadata plus a stable identifier back to the business record answers most governance questions without that exposure.
Can we add observability after the system is already live?
Partly, and never retroactively. You can instrument from today forward, and you should. What you cannot do is produce a record of what happened before you started capturing it. That asymmetry is why the field list is a design decision rather than an operations one: adding the governance fields at build time costs almost nothing, and adding them later does nothing for the period anyone is actually asking about.
What is a realistic evaluation set to start with?
Fifty to a few hundred of your own past cases with the answer a competent person gave, drawn from real work rather than a public benchmark. A benchmark tells you how a model performs on somebody else’s problem. The point of the set is to produce a defensible number about yours, so the accuracy conversation happens with evidence rather than adjectives, and so there is a documented threshold the system either clears or does not.
Deciding which fields you need, and whether your current systems can produce them, is scoping work rather than a build. That is what our AI Readiness and Cost-Reduction Assessment settles before anyone writes instrumentation code.
Sources
- Article 12: Record-Keeping, EU Artificial Intelligence Act
- Article 19: Automatically Generated Logs, EU Artificial Intelligence Act
- Regulatory framework for AI, European Commission
- Inside the LLM Call: GenAI Observability with OpenTelemetry, OpenTelemetry, May 2026
- GenAI semantic conventions repository, OpenTelemetry
- NIST AI Risk Management Framework, NIST
- NIST AI RMF Core Functions: Govern, Map, Measure, Manage, NIST AI Resource Center
- Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1), NIST, July 2024
- ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system, ISO