AI-powered apps are hard to debug without observability. From one run to another, AI produces different results, goes down different paths, and you’re none the wiser as to why it’s happening. Agent Tracing bridges the gap between prompt, tool calls and final output, all within the context of your app. If things break (it always does at some point, we know from experience), you get everything in one complete trace. Model calls, tool executions, and handoffs, alongside everything else your app handles.

You can debug silent tool failures, find the slow step in a multi-step workflow, fix multi-agent handoffs where vital context goes missing and the AI starts hallucinating. In this post, we’re going to walk through some of our Agent Tracing recipes from the Sentry Cookbook. We’ll show different use cases that you can set up yourself with your agent.

Turn production AI agent conversations into eval datasets

An eval is a test for an AI agent. You give it an input, look at what it said, and score the answer. Some teams write these by hand (or even generate synthetic evals), so let’s automate this process with real conversations from your production data already recorded in Sentry. This recipe pulls these conversations and feeds them to an eval tool of your choice that scores them for you.

The recipe exports conversations with one Sentry CLI command, which then become test records. The first user message is the input, the last agent reply is the output, and cost, tokens, and tool errors come along as metadata. You load the records into Braintrust, or into Langfuse, promptfoo, or Arize Phoenix if you prefer, then score them with code checks where you can and an AI agent judge where you can’t.

Copy the prompt below to have your coding agent guide you through setting it up for your stack, or follow the full eval dataset recipe.

conversations.json

Read one conversation turn by turn: model, tokens, and tool calls per turn

sentry agent-conversation view YOUR_ORG/CONVERSATION_ID --json

Learn more: [Sentry CLI agent-conversation command reference](https://cli.sentry.dev/commands/agent-conversation/)

### Step 5: Map conversations to eval records

Map each row to one record. `input` is `firstInput`, `output` is `lastOutput`, and the other fields become metadata your scorers can use: agent, tokens, cost, tool calls, tool errors, and a link back to the conversation in Sentry.

Drop rows where `firstInput` or `lastOutput` is `null`. Those conversations come from routes where input or output recording is off. Also drop your own noise, like seeded demo data and smoke tests, before it gets into the dataset. Then sort by `totalTokens` so the heaviest conversations come first. That is where agents thrash, and those conversations make the most useful eval cases.

**scripts/export-conversations.ts**
```typescript

type Conversation = {
conversationId: string;
title: string | null;
firstInput: string | null;
lastOutput: string | null;
flow: string[]; // agent names
llmCalls: number;
toolCalls: number;
toolErrors: number;
toolNames: string[];
totalTokens: number;
totalCost: number;
webUrl: string;
};

const { data } = JSON.parse(readFileSync('conversations.json', 'utf8')) as {
data: Conversation[];
};

// Replace with your own convention for test and seeded conversations
const isTestTraffic = (id: string) => id.startsWith('test-');

const records = data
.filter((c) => c.firstInput && c.lastOutput && !isTestTraffic(c.conversationId))
.sort((a, b) => b.totalTokens - a.totalTokens)
.map((c) => ({
id: c.conversationId,
input: c.firstInput,
output: c.lastOutput,
metadata: {
title: c.title,
agents: c.flow,
llm_calls: c.llmCalls,
tool_calls: c.toolCalls,
tool_errors: c.toolErrors,
tools: c.toolNames,
total_tokens: c.totalTokens,
cost_usd: c.totalCost,
sentry_url: c.webUrl,
},
}));

Learn more: Conversation JSON fields

Step 6: Load the conversations into Braintrust

Push each record into a Braintrust dataset. Setting id to the conversation ID makes inserts upserts, so re-running the export syncs new conversations instead of duplicating old ones. Keep the production answer in metadata. Your first experiment should score what the agent actually said in production, not a re-generation.

You can also log each conversation as a trace in Braintrust's logs so its clustering and topic features work on your production traffic.

scripts/export-conversations.ts

const dataset = initDataset({
project: 'my-project',
dataset: 'prod-conversations',
});

for (const record of records) {
dataset.insert({
id: record.id, // upsert key: re-runs overwrite, never duplicate
input: record.input,
metadata: { ...record.metadata, production_output: record.output },
});
}

await dataset.flush();

Learn more: Braintrust TypeScript SDK reference

Step 7: Score real answers with scorers and a judge

Now run an experiment. The task is a passthrough that returns the production output, so you're scoring reality. Write deterministic scorers for anything you can verify with code. If your agent cites entities from your database, groundedness is a lookup, not a judgment call. Telemetry metadata gives you free scorers too: token efficiency and tool success rate come straight from the export.

Save the LLM-as-judge for the one dimension code can't reach (was the answer actually helpful?), and calibrate it periodically against human labels. When you're ready to compare models or prompts, swap the passthrough task for a live call to your agent and run the same dataset through it.

evals/grounding.eval.ts

// Tune to the token count of a normal conversation for your agent
const TOKEN_BUDGET = 15_000;

// Fields written by the export script
type ConversationMetadata = {
production_output: string;
total_tokens: number;
tool_calls: number;
tool_errors: number;
};

Eval<string, string, void, ConversationMetadata>('my-project', {
data: initDataset({ project: 'my-project', dataset: 'prod-conversations' }),
// Passthrough: score what the agent actually said in production
task: async (input, hooks) => hooks.metadata.production_output,
scores: [
// Deterministic: every entity the answer cites must exist in your DB
async ({ output, metadata }) => ({
name: 'grounded',
score: await citationsExistInDb(output, metadata),
}),
// From telemetry: 1 within budget, then decays as tokens grow
({ metadata }) => ({
name: 'efficiency',
score: Math.min(1, TOKEN_BUDGET / Math.max(1, metadata.total_tokens)),
}),
// From telemetry: share of tool calls that succeeded
({ metadata }) => ({
name: 'tool_success',
score: metadata.tool_calls ? 1 - metadata.tool_errors / metadata.tool_calls : 1,
}),
],
});

Learn more: Braintrust TypeScript SDK quickstart

Step 8: Swap in the eval stack that fits you

Nothing above is Braintrust-specific except step 6. The pipeline has three separate stages: extract (sentry agent-conversation list), map (plain TypeScript), and load (your tool's SDK). To change platforms, you rewrite one function. Good options:

The fastest way to adapt it: give the spec to your coding agent. This prompt has everything this recipe covered, including the traps.

• Braintrust (cloud): logs, datasets, and experiments with upserts via dataset.insert({ id }). This is what this recipe uses.
• Langfuse (open source, self-hostable): langfuse.dataset.createItem() per record, then dataset.runExperiment().
• promptfoo (local-first CLI): write records to a JSON tests file and run npx promptfoo eval. Scorers are inline JavaScript assertions.
• Arize Phoenix (open source, runs locally): upload a DataFrame with client.datasets.create_dataset() and run experiments against it.

Prompt for your coding agent

Export my production AI conversations from Sentry into as an
eval dataset.

Extract: run `sentry agent-conversation list --period 30d --limit 1000
--json` (Sentry CLI 0.44 or later). Each item in `data` is one conversation
with conversationId, title, firstInput, lastOutput, flow, llmCalls,
toolCalls, toolErrors, toolNames, totalTokens, totalCost, and webUrl.
Sentry computes the totals per conversation. Do not fetch or sum raw spans.

Map one record per conversation: id = conversationId, input = firstInput,
output = lastOutput, metadata = the other fields. Skip rows where firstInput
or lastOutput is null (recording was off). Filter out test and seeded
conversations by ID convention. Sort by totalTokens, descending. For
per-turn detail or the model of each turn, run
`sentry agent-conversation view / --json`.

Load: upsert into keyed on the conversation id so re-runs sync
instead of duplicating. Then write scorers: deterministic checks for anything
verifiable against my data (grounding is a lookup), and an LLM judge only
for genuinely subjective quality.

Tips

Pro tips:

• Sort the export by total tokens descending. The heaviest conversations are where agents thrash, and they make the most revealing eval cases.
• Tag records from telemetry alone (thrash, tool-error, repeated-tool-call) so you can slice experiments by failure mode without hand-labeling anything.
• Promote a production answer to the dataset's expected field only after it passes your deterministic checks. Verified-clean conversations become golden rows for regression testing.
• Scorers are code, and code goes blind silently. Keep a self-test that plants known hallucinations and asserts they get caught, so a scorer regression fails loudly instead of inflating your scores.
• Real and synthetic data aren't either/or: production conversations cover what users actually do, synthetic cases cover what they haven't done yet. Use both.

Watch out for:

• Summing tokens or cost from raw spans yourself. gen_ai.invoke_agent spans aggregate their children, so a naive sum doubles every figure. The CLI totals are already correct per conversation, so use them as they are.
• Exporting with the defaults. sentry agent-conversation list returns 25 conversations from the last 7 days unless you set --limit and --period.
• Treating lastOutput as the final answer in every case. It is the last assistant message Sentry recorded, so a long or interrupted run can end on a step in the middle. Use sentry agent-conversation view to check the turns of a conversation before you score it.
• Running an older Sentry CLI. The agent-conversation command needs version 0.44 or later.
• Exporting your own noise. Seeded demo data and smoke-test conversations will poison the dataset unless you filter them out by ID convention.
• Using an LLM judge for things code can check. Asking a judge "is this grounded?" invites a hallucinated verdict when the answer is a database lookup away.
• Recording inputs and outputs on routes that handle sensitive data. They're on by default, and you're exporting these payloads to a third-party eval platform, so opt out on those routes or scrub PII before it leaves Sentry.

Frequently asked questions

Q: Do I have to use Braintrust?

A: No. Only the load stage is Braintrust-specific. The extract and map stages work with any target. Langfuse and Arize Phoenix are open source and self-hostable, and promptfoo runs entirely locally from a JSON file. Step 8 has the mapping for each.

Q: Should I use the MCP server or the CLI?

A: Both work, and they read the same Sentry data. The MCP server is good for interactive exploration from a coding agent, but it truncates responses to save context. For bulk export with full inputs and outputs, use sentry agent-conversation list --json.

Q: What about user privacy and PII?

A: In Sentry JavaScript SDK v11, prompts and outputs are recorded by default. Turn them off globally with dataCollection: { genAI: { inputs: false, outputs: false } } in Sentry.init(), or per integration with recordInputs: false and recordOutputs: false. Sentry also supports server-side data scrubbing before storage. Remember the export sends this data to your eval platform too, so apply the same scrubbing standards there, or self-host the eval tool.

Q: Does this replace synthetic eval datasets?

A: No, it complements them. Synthetic datasets are guesses about user behavior; production conversations are the ground truth of it. Real traffic tells you what to fix today, synthetic cases protect flows real users haven't exercised yet.

Q: How many conversations do I need before this is useful?

A: Fewer than you'd think. A few dozen real conversations sorted by token volume will surface your agent's actual failure modes (thrashing, tool errors, hallucinated citations) faster than hundreds of synthetic cases.

Q: Can I compare different models with this?

A: Yes, two ways. If your app already routes users to different models, get the model of each turn with sentry agent-conversation view --json, add it to the record metadata, and group your experiment by it. To test a model your users haven't touched, swap the passthrough task for a live call to your agent with the candidate model and run the same dataset.`} />

Automating AI agent error triage with Claude’s scheduled tasks

Triage means going through what broke and deciding what needs a fix. With AI agents that’s harder than usual since problems often don’t show up as errors. A tool call fails and the agent retries until it works, or hallucinates a plausible but incorrect answer. Reading every conversation yourself takes too long, so why not have an agent read them for you every morning?

This recipe sets up a scheduled task in Claude that runs once a day and connects to Sentry through the MCP server. It pulls stats (collected with Agent Tracing) via the MCP for the last 24 hours of conversations, then picks a mix of failed and successful runs to read closely.

When your scheduled task finds something new, like a broken tool, an agent stuck in a loop, or a run that costs a lot more than it should, it checks your tracker (Linear in this recipe) for an existing ticket and files one if there isn’t one yet. You don’t necessarily need a scheduled task for this either. A cron job or GitHub Action that runs a prompt on a schedule works the same way.

Copy the prompt below to have your coding agent guide you through setting it up, so you can have relevant tickets filed before your day starts. You can also check out the full AI agent triage recipe.

Agents](https://sentry.io/orgredirect/organizations/:orgslug/explore/agents/) in Sentry. You should see your agent's conversations listed with duration, message counts, errors, cost, and tool calls.

Each conversation shows user inputs, model outputs, tool calls, errors, latency, and cost. This is the data your Claude Routine will query.

Step 3: Configure the Sentry MCP connector in Claude

Your Claude Routine needs to authenticate with Sentry to query your agent's traces. Set up the Sentry MCP connector in Claude so the routine can call Sentry APIs directly.

If you want the routine to file tickets automatically, enable the Linear MCP connector (or whichever project tracker you use). The routine will search for existing tickets before creating new ones to avoid duplicates.

Learn more: Sentry MCP server documentation

Step 4: Create the Claude Routine with a triage prompt

Create a new Claude Routine and set it to run daily (mornings work well since the routine analyzes overnight activity). The prompt should tell Claude to use the Sentry MCP to pull aggregate stats, sample conversations with and without errors, and file tickets for new findings.

Keep the prompt intentionally plain. The Sentry MCP handles the data retrieval. Let the LLM handle the judgment: deciding whether a pattern is new, whether an error is self-correcting, and whether a conversation's conclusion actually follows from its reasoning.

Claude Routine prompt (template)

Use the Sentry MCP to look at the last 24hr of
conversations and analyze the results. In particular look for errors
that might be happening during the , say if our tools
are broken. But also sample some success conversations as well to see
if they look correct with secondary inspection.

If you come across new findings/errors, record them in a Linear ticket
for the project.

Step 5: Review and tune the routine's output

After the first few runs, review what the routine flags. A good triage routine catches the following categories of problems.

That last category, reasoning failures, is why sampling successful conversations matters. A conversation can complete without a single tool error and still reach the wrong conclusion. That only shows up when something checks the reasoning against the verdict.

• Tool call failures: tools returning errors, especially at higher-than-normal rates
• Hallucinated inputs: the agent guessing at parameters (like repo names or project slugs) instead of looking them up
• Latency anomalies: conversations with unusually long hangs before an error, which may signal backend issues distinct from the agent's own mistakes
• Agent loops: conversations with far more tool calls than typical, suggesting the agent is thrashing
• Cost anomalies: runs that are significantly more expensive than the baseline
• Reasoning failures: conversations that completed without tool errors but reached the wrong conclusion

Sample triage output

Analyzed the last 24h of Seer conversations (~445-551 total,
~11k tool calls). No verdict-quality problems found. Spot-checked
several conversations and the agent's reasoning was well-grounded,
correctly downgrading when it lacked repo access rather than
fabricating.

Did find an efficiency issue worth tracking: ~21% of conversations
(83/400 sampled) hit at least one tool error, mostly self-corrected
retries. Root causes:

- Search Code sometimes gets fed malformed repo_name values
  (e.g. appending a monorepo subpath onto the real slug, like
  "foo/bar/applications/integrations" instead of "foo/bar"),
  causing "repository not found" until it retries.
- sometimes gets a guessed-wrong project_slug (e.g.
  "mobile") before self-correcting on retry with the right
  slug + time window.
- Two conversations showed 30-56s hangs before an opaque
  "internal error", a real backend latency signal, distinct
  from the guessing pattern above, worth an engineering look.

Filed Linear ticket AIML-123 with full details, example
conversation links, and error-rate tables.

Tips

Pro tips:

• Keep the routine's prompt short and focused on judgment. The Sentry MCP handles data retrieval, so the prompt only needs to describe what to look for and where to file findings.
• Have the routine search your project tracker for existing tickets before filing new ones. This prevents duplicates for known issues.
• Sample both errored and successful conversations. Reasoning failures only surface when you check whether the agent's conclusion actually follows from its work.
• Sort sampled conversations by tool call count or token usage. The heaviest runs are where agents thrash and reveal the most about failure modes.

Watch out for:

• Only looking at conversations with tool errors. A conversation can complete cleanly and still reach the wrong conclusion.
• Hardcoding org-specific details in the routine prompt instead of using variables. This makes the routine harder to share or adapt.
• Writing overly detailed prompts that try to specify exact MCP queries. Let the LLM figure out how to query Sentry. Focus the prompt on what to look for, not how to look for it.
• Not searching for existing tickets before filing. The routine will create duplicates for issues that are already tracked.

Frequently asked questions

Q: Does this only work with Claude Routines?

A: No. The approach works with any scheduled LLM workflow. Claude Routines just handle the scheduling and MCP connectivity out of the box. You could build the same thing with a cron job that calls an LLM API, a GitHub Action, or any automation tool that can run a prompt on a schedule.

Q: What if my agent doesn't use tool calls?

A: Tool error triage is one part of this routine, but the conversation sampling and reasoning checks work regardless. If your agent is a pure chat model, focus the routine on sampling conversations and checking whether the responses are grounded and accurate.

Q: How many conversations can the routine process?

A: The routine doesn't read every conversation in full. It pulls aggregate stats first, then samples a subset for detailed inspection. In practice, routines process 400-800 conversations per run, with the Sentry MCP handling the heavy querying.

Q: How much does this cost in LLM tokens?

A: The routine reads aggregate stats and sampled conversations, not every conversation in full. A typical daily run processes a few dozen sampled conversations. The token cost depends on your LLM pricing, but it's comparable to a single longer chat session.`} />

Track Vercel AI SDK costs and LLM calls

Every LLM call your app makes will cost you money, and the bill usually shows up as one number at the end of the month. That doesn’t tell you much about which model, route, or conversation is behind it. Agent Tracing can track that cost per call and per conversation, and this recipe shows how with a Next.js (we support other libraries and languages too) app that uses the Vercel AI SDK.

This recipe turns on tracing with the Sentry Next.js SDK, which also turns on Sentry’s Vercel AI integration by default. Give each AI SDK call a name and set a conversation ID, and all the turns of one chat then stay together.

In Sentry, the Explore → Agents view shows estimated cost by model, agent runs, and tool calls over time, plus a list of every conversation with its individual cost and errors. Open a conversation to read it in its entirety, then follow it into the full trace to see whether it’s a slow tool call coming from your database, the network, or the AI provider that’s responsible for the issue.

Copy the prompt below to have your coding agent guide you through tracking costs, or follow the full AI agent cost recipe.

Monitor AI agent spend with Sentry dashboards and alerts

An unintended cost and usage spike can go unnoticed for days until the bill ends up in your inbox. Keeping an eye on your agent costs is helpful, but you still have to remember to look. A shared dashboard and an alert that fires when spend jumps mean you hear about it almost instantly. Seer Autofix can act on these alerts, find the root cause, and draft a PR before you’ve even had time to check out the issue.

This recipe runs a script that uses the Sentry CLI to build an LLM Spend per User dashboard. It contains eight widgets covering cost, tokens, top spenders, and models. The script also sets up two hourly alerts on spend. One alert fires when spend crosses a limit set by you, and the other fires when spend moves outside its normal range. Both alerts can be tuned to your liking in Monitors.

It works with any stack that sends standard gen_ai span data to Sentry, so there’s no need for a specific framework or language. Sentry works out an estimate for the cost on its own from the token counts, so your app doesn’t have to send any dollar amounts.

Copy the prompt below to have your coding agent guide you through setting up dashboards, or follow the full AI spend dashboard recipe.

```

Learn more: Dashboard script source

Step 4: Review the spend dashboard

Open the URL that the script prints, or select LLM Spend per User in Dashboards. If the user or conversation widgets are empty, add user.id, user.username, or gen_ai.conversation.id to your model-call spans.

Learn more: Sentry Dashboards

Step 5: Tune the spend monitors

Open Monitors and adjust LLM spend rate high and LLM spend anomaly for your traffic and budget. To route notifications during creation, set WORKFLOW_ID and, optionally, OWNER=user: or OWNER=team: before you run the script. Watch the alert workflow in the demo video.

Learn more: Sentry Monitors and Alerts

Tips

Pro tips:

• Start with the included thresholds in a test project, then replace them with limits that match your model mix and traffic.
• Set user.id to a stable internal identifier and user.username to a useful display value. Do not use email addresses unless your data policy allows them.
• Use TITLE to give a second dashboard a distinct name when you need separate views for environments or teams.
• Add an alert workflow so a detector routes to the team that can reduce spend or stop a faulty agent.

Watch out for:

• Removing the gen_ai.operation.type:ai_client filter from cost or token widgets. Parent agent spans can repeat the model-call values and double the totals.
• Assuming Sentry receives a dollar value from the SDK. Sentry derives gen_ai.cost.total_tokens from token counts and its model price list.
• Running the script again without deleting or renaming the existing dashboard and detectors. The duplicate-name guard stops the run on purpose.
• Treating the starting thresholds as production defaults. A suitable hourly limit depends on your traffic, models, and budget.

Frequently asked questions

Q: Does the script work only with Flue or Eve?

A: No. It works with any Sentry project that receives the standard GenAI span attributes used by the queries.

Q: Why does the cost field end with total_tokens?

A: That is the current Sentry field name for derived model cost. Its value is in US dollars, not tokens.

Q: Why are the per-user widgets empty?

A: Your model-call spans need user.id. The top-spenders and conversation tables also use user.username and gen_ai.conversation.id.

Q: Can the alerts notify Slack or assign an owner?

A: Yes. Connect an existing alert workflow with WORKFLOW_ID, and set OWNER to a Sentry user or team when you create the detectors.`} />

Send Vercel AI SDK telemetry to Sentry via OpenTelemetry

If you already have OpenTelemetry set up with traces going to a different tool, moving all of that to a new setup just to get AI data into Sentry sounds like a lot of work. This recipe isn’t replacing anything you’ve already set up. It keeps your existing pipeline and uses your coding agent to add Sentry as an additional destination.

Once Sentry starts receiving your OTel spans, LLM calls show up in the Trace Explorer and in Explore → Agents. These agent spans appear next to the rest of your app, database, and other spans from the same requests.

This specific recipe uses Next.js with the @vercel/otel SDK already set up, but the same setup works with other environments, like a standalone Node.js server, a worker, or a serverless function.

Copy the prompt below to have your coding agent set up Sentry to receive OTel agent spans, or check out the full OpenTelemetry recipe.

Get started with Agent Tracing

All five of these recipes start from the same place, an agent sending traces to Sentry. Our Set up Agent Tracing docs get you through the very basics, while our recipes go a little deeper into specific use cases. The Sentry Cookbook has even more Agent Tracing recipes you can follow, including ones for monitoring your AI coding tools like Claude Code and OpenCode.

If something breaks along the way, at least this time you’ll have the AI observability needed to tell you (or your agent) where things went wrong. If you’re somehow brand new to Sentry and still read through this whole blog post, you can sign up for free today to get started with Agent Tracing. The Sentry MCP and CLI usage is included in every plan, so giving your agents debugging context is quicker than you think.