LLM observability
LLM observability is the practice of recording what happens on every call your app makes to a language model, so you can see why a feature behaved the way it did. It covers the input, the output, the model that answered, the time it took and what it cost, kept together so any single call can be replayed later.
What LLM observability means
What is LLM observability?
LLM observability means you can answer questions about your AI feature from records instead of guesses. When a customer forwards a strange reply, you can open that exact call, read the prompt your app sent, see which model handled it and check how long it ran and what it cost. To turn those per-call costs into a cost per customer, see the Supagen guide to AI token costs.
The idea comes from ordinary software, where observability means a system emits enough data that you can work out its internal state from the outside. Language models need their own version because their failures look different. A crashed server throws an error, while a model that gives a confident wrong answer returns a normal, successful response.
AI observability is the broader term. It includes language models plus image, speech and video models, and sometimes classic machine learning systems. People often use the two terms interchangeably, and the practices overlap almost entirely.
Monitoring is a narrower idea. Monitoring tells you that something changed, such as a rise in errors or slower responses. Observability lets you ask why, by going from that chart to the individual calls behind it.
Per call, the records should let you answer five kinds of question:
- Behavior. What the model was asked and what it said.
- Cost. How many tokens (small chunks of text) went in and out, and the price.
- Routing. Which model and provider answered, and whether a backup took over.
- Safety. Whether the answer leaked private data or followed a jailbreak attempt.
- Accountability. Which prompt version, user and feature produced the call.
How it shows up in practice
Most teams discover they need this the first week an AI feature has real users. The symptoms repeat across products, whether the feature is a support bot, a contract reviewer or an image generator in a marketing tool.
- A reply you cannot reproduce. A customer screenshots an odd answer, and without the stored input and output there is nothing to debug.
- A bill with no breakdown. The provider invoice rises, and nobody can say which feature, prompt or customer drove the extra tokens.
- A slow feature. Users complain that a chat reply takes too long, and you cannot tell whether the delay is the model, the prompt length or your own code.
- A silent quality drop. A prompt edit ships, answers get worse, and no error ever fires because every call technically succeeded.
- A safety incident. Someone coaxes the model into revealing personal data, and you need to find every other call where it happened.
A support bot starts telling customers they can get refunds the company does not offer. The team filters the logs to refund conversations, notices the bad answers began on one day, and finds that a prompt change that day moved the refund policy below a new instruction about sounding friendly. The model now ranked friendliness over policy.
The team fixed it by restoring the earlier prompt version. Because the prompt lived outside the code, the team changed it in a dashboard with no redeploy, and the logs confirmed the next calls followed policy again.
The industry use cases differ in what they watch most closely. A legal or finance tool cares about accuracy and an audit trail, a healthcare or HR tool about private data in outputs, and an ecommerce or media tool about cost per generated description or image.
Different people on the team read the same records for different reasons. Engineers open one trace and read it. Product managers watch quality and cost per feature. Support staff pull up the calls behind a complaint. Anyone responsible for compliance needs an audit trail of what the model was told and what it said.
Why it happens
Language model calls are hard to see into for three reasons, and each one breaks a habit carried over from ordinary software.
The first is that success and failure look the same. A traditional API either returns the right data or an error code. A model returns text every time, and whether that text is correct, on brand or safe is a judgment the response code cannot make.
The second is that the output varies. The same prompt can produce different answers on different runs, so you cannot reproduce a bug by sending the request again. The only reliable copy of what happened is the one you saved at the time.
Third, much of the behavior comes from outside your code. A feature's result depends on the prompt text, the model, its settings and the provider's current version of that model. Change any one and the output shifts, with no change to your application code at all.
Cost follows the same pattern. Providers charge per token, so a longer prompt or a chatty model raises the bill one call at a time. Without per call records, the first sign is the invoice.
Teams that build through Claude Code or Cursor see this most, because the agent writes the model call and moves on. Nothing in the generated code records what the model received or returned unless someone asks for it.
What to do about it
Start by logging every model call with the same fields: the input, the output, the model and provider, the token counts, the latency (how long the call took) and the cost. Add the prompt version and the feature name so you can filter later. For a deeper comparison of products in this space, see our guide to AI observability tools.
Then fix a debugging order before you need it, so your team follows set steps during an incident:
- Find the affected segment: which feature, users or inputs
- Check what changed recently: a prompt version, a model or a setting
- Open a failing call's trace and read the exact input and output
- Compare against feedback or test results from before the change
- Roll back the change or route to another model
Write one short runbook (a checklist for a known problem) per major failure mode, such as wrong answers, slow replies, cost spikes and data leaks. Signals worth tracking alongside token counts include time to first token, hallucination rate, jailbreak success rate and how often private data appears in outputs.
The bigger decision is where you keep the records:
| Option | Best for | What it costs |
|---|---|---|
| Logging calls in your own code | A prototype with one feature and a developer to maintain it | Your engineering time and storage |
| A dedicated LLM observability tool | Teams that already route calls elsewhere and want deep evaluation | Varies by vendor, plus the work of wiring it into every call |
| Supagen | Small teams and solo founders who want calls, routing and logs in one place | USD 0 or USD 29 a month, plus model usage |
A point tool fits best when you already have a gateway and need advanced evaluation on top. A single backend fits best when you would rather not build and connect logging, routing and prompt storage yourself.
Supagen is the backend for AI features and agents: small teams and solo founders keep prompts, model choice and routing, and call logging out of their code and call every model, from text to image, video and speech, through one API.
Supagen keeps a log of each call your app makes. Each entry shows what was sent, what came back, the token counts, how long it took and what it cost, and you can open any call as a trace and step through it. A Supagen template can hold up to ten routes, each pairing a model with a provider. Supagen tries the first one and keeps the rest as backups, and you set them all in the dashboard. Supagen has two plans, USD 0 a month and USD 29 a month, and offers no free trial.
To see your own calls logged, start building free.
Frequently asked questions
What is the difference between LLM monitoring and LLM observability?
Monitoring watches summary numbers such as error rates and response times and alerts you when they change. Observability keeps the detail behind those numbers, so you can open individual calls and find out why they changed.
Is AI observability the same as LLM observability?
The terms overlap. AI observability also covers image, speech, video and classic machine learning models, while the narrower term focuses on language models.
What should I log for each LLM call?
Log the input, the output, the model and provider, token counts, latency, cost, the prompt version and the feature that made the call.
Do I need LLM observability for a prototype?
A prototype with no users can get by on simple logs. Once real people rely on the feature, you need stored calls to explain bad answers and costs.