Debug AI Agent: Tracing, Observability, and Common Failure Patterns in LLM Agents
TL;DR — Debug AI agents: tracing, observability, failure patterns. LangSmith: "LLM observability: visibility into agent decisions, track cost and latency, debug failures and hallucinations with complete execution trace end-to-end." DigitalOcean: "Every LLM call, tool invocation, reasoning step is observable. Understand why agent took action. Traces turn debugging into analysis." TrueFoundry: "Single prompt triggers 5-6 reasoning steps. Step-level view essential — see each step individually." GoClaw: "Observability: OpenTelemetry, AgentPrism for traces, AgentRx for failure diagnosis, LangSmith for lifecycles, costs, tokens." LangChain: "OTel-compatible instrumentation. LangSmith supports OTel to unify observability stack. Semantic conventions for GenAI." Learn more with what is AI agent, tool loop, LangChain agent, and web search.
LangSmith defines the core need: "LLM observability platforms provide visibility into RAG pipelines, AI agent decisions, track model performance metrics like cost and latency, and help debug complex failures and hallucinations by showing the complete execution trace from end-to-end."
DigitalOcean explains the value: "Observability is foundational for agent reliability. With LangSmith's tracing, every LLM call, tool invocation, and intermediate reasoning step is observable, so you can understand why an agent took a certain action instead of having to infer it from the logs. Traces turn debugging into analysis."
Debug Architecture
model, messages, response"] Step2["Step 2: Tool Call
name, args, result"] Step3["Step 3: LLM Call
process result, decide next"] Step4["Step N: Final Answer
or max steps"] end subgraph Tracing["Tracing & Observability"] LangSmith["LangSmith
automatic tracing
env vars only
framework-agnostic"] OTel["OpenTelemetry
standardized instrumentation
semantic conventions
unified observability"] Logs["Structured Logs
JSON format
step, tool, args, result
tokens, latency"] AgentPrism["AgentPrism
React trace visualization
inspect each step"] end subgraph Failures["Common Failures"] InfiniteLoop["Infinite Loop
repeats same tool call
no progress"] Hallucination["Hallucination
fabricates information
or tool results"] ToolError["Tool Error
exception not handled
agent crashes"] ContextOverflow["Context Overflow
too many messages
exceeds window"] WrongTool["Wrong Tool
calls wrong tool
for the task"] ArgError["Argument Error
wrong arguments
to tool"] end subgraph Fixes["Debug Fixes"] MaxSteps["Max Steps + No-Progress
break infinite loops"] Verify["Verify Tool Outputs
citation validation
check results"] TryExcept["try/except per tool
return error as result"] Summarize["Summarization
sliding window
context management"] BetterDesc["Better Tool Descriptions
clear system prompt
parameter docs"] Schema["JSON Schema Validation
validate arguments
before execution"] end Step1 --> LangSmith Step2 --> OTel Step3 --> Logs Step4 --> AgentPrism InfiniteLoop --> MaxSteps Hallucination --> Verify ToolError --> TryExcept ContextOverflow --> Summarize WrongTool --> BetterDesc ArgError --> Schema style Tracing fill:#4169E1,color:#fff style Failures fill:#FF6B6B,color:#fff style Fixes fill:#39FF14,color:#000
Failure Patterns and Fixes
| Failure | Symptom | Root Cause | Fix |
|---|---|---|---|
| Infinite loop | Repeats same tool call | No termination condition | max_steps, no-progress detection |
| Hallucination | Fabricates information | No output verification | Verify tool outputs, citations |
| Tool error | Agent crashes on exception | No error handling | try/except, return error as result |
| Context overflow | Exceeds context window | Too many messages | Summarization, sliding window |
| Wrong tool | Calls wrong tool | Poor tool descriptions | Better descriptions, system prompt |
| Argument error | Wrong args to tool | No schema validation | JSON schema validation |
Implementation
from dataclasses import dataclass
from typing import Optional
from enum import Enum
class FailureType(Enum):
INFINITE_LOOP = "infinite_loop"
HALLUCINATION = "hallucination"
TOOL_ERROR = "tool_error"
CONTEXT_OVERFLOW = "context_overflow"
WRONG_TOOL = "wrong_tool"
ARGUMENT_ERROR = "argument_error"
@dataclass
class DebugAgentGuide:
"""Debug AI agent implementation guide."""
def get_traced_agent(self) -> str:
"""Agent with full tracing and logging."""
return (
"import json\n"
"import time\n"
"import logging\n"
"from openai import OpenAI\n"
"\n"
"# LangSmith: just set env vars\n"
"# import os\n"
"# os.environ[\n"
"# 'LANGSMITH_TRACING']='true'\n"
"# os.environ[\n"
"# 'LANGSMITH_API_KEY']='ls__..'\n"
"# os.environ[\n"
"# 'LANGSMITH_PROJECT']='debug'\n"
"\n"
"client = OpenAI()\n"
"logger = logging.getLogger(\n"
" 'agent')\n"
"\n"
"def run_traced_agent(\n"
" user_msg: str,\n"
" tools: list,\n"
" tool_map: dict,\n"
" max_steps: int = 10\n"
") -> dict:\n"
" '''Agent with full\n"
" tracing and logging.'''\n"
" run_id = f'run_{int(\n"
" time.time())}'\n"
" messages = [\n"
" {'role': 'system',\n"
" 'content': SYSTEM},\n"
" {'role': 'user',\n"
" 'content': user_msg}\n"
" ]\n"
" \n"
" trace = {\n"
" 'run_id': run_id,\n"
" 'steps': [],\n"
" 'total_tokens': 0,\n"
" 'total_latency': 0,\n"
" 'errors': []\n"
" }\n"
" \n"
" prev_output = None\n"
" no_progress = 0\n"
" \n"
" for step in range(\n"
" max_steps):\n"
" step_start = time.time()\n"
" \n"
" logger.info(\n"
" f'[{run_id}] Step'\n"
" f' {step+1}')\n"
" \n"
" # LLM call\n"
" try:\n"
" resp = client.chat\\\n"
" .completions.create(\n"
" model='gpt-4o',\n"
" messages=messages,\n"
" tools=tools)\n"
" except Exception as e:\n"
" logger.error(\n"
" f'LLM error: {e}')\n"
" trace['errors'].append({\n"
" 'step': step+1,\n"
" 'type': 'llm_error',\n"
" 'error': str(e)\n"
" })\n"
" return trace\n"
" \n"
" msg = resp\\\n"
" .choices[0].message\n"
" messages.append(msg)\n"
" tokens = resp\\\n"
" .usage.total_tokens\n"
" latency = time.time() - \\\n"
" step_start\n"
" \n"
" step_trace = {\n"
" 'step': step+1,\n"
" 'tokens': tokens,\n"
" 'latency': latency,\n"
" 'tool_calls': []\n"
" }\n"
" trace['total_tokens'] \\\n"
" += tokens\n"
" trace['total_latency'] \\\n"
" += latency\n"
" \n"
" # Check termination\n"
" if not msg.tool_calls:\n"
" step_trace['answer'] = \\\n"
" msg.content\n"
" trace['steps'].append(\n"
" step_trace)\n"
" trace['result'] = \\\n"
" msg.content\n"
" logger.info(\n"
" f'[{run_id}] Done in'\n"
" f' {step+1} steps')\n"
" return trace\n"
" \n"
" # Execute tools\n"
" current_output = []\n"
" for call in \\\n"
" msg.tool_calls:\n"
" name = call.function\\\n"
" .name\n"
" args = json.loads(\n"
" call.function\n"
" .arguments)\n"
" \n"
" logger.info(\n"
" f' Tool: {name}')\n"
" f' Args: {args}')\n"
" \n"
" func = tool_map.get(\n"
" name)\n"
" if not func:\n"
" error = (\n"
" f'Unknown tool:'\n"
" f' {name}')\n"
" logger.error(\n"
" f' {error}')\n"
" trace['errors'].append({\n"
" 'step': step+1,\n"
" 'type':\n"
" 'wrong_tool',\n"
" 'tool': name,\n"
" 'error': error\n"
" })\n"
" result = error\n"
" else:\n"
" try:\n"
" result = func(\n"
" **args)\n"
" logger.info(\n"
" f' Result: {str('\n"
" f' result)[:80]}')\n"
" except Exception as e:\n"
" result = (\n"
" f'Error: {e}')\n"
" logger.error(\n"
" f' Tool error: {e}')\n"
" trace['errors'].append({\n"
" 'step': step+1,\n"
" 'type':\n"
" 'tool_error',\n"
" 'tool': name,\n"
" 'error': str(e)\n"
" })\n"
" \n"
" messages.append({\n"
" 'role': 'tool',\n"
" 'tool_call_id':\n"
" call.id,\n"
" 'content': str(\n"
" result)\n"
" })\n"
" current_output.append(\n"
" str(result))\n"
" step_trace['tool_calls']\n"
" .append({\n"
" 'name': name,\n"
" 'args': args,\n"
" 'result': str(\n"
" result)[:200],\n"
" 'error': result\n"
" .startswith(\n"
" 'Error')\n"
" })\n"
" \n"
" trace['steps'].append(\n"
" step_trace)\n"
" \n"
" # No-progress detection\n"
" output_state = ''.join(\n"
" current_output)\n"
" if output_state == \\\n"
" prev_output:\n"
" no_progress += 1\n"
" if no_progress >= 3:\n"
" logger.warning(\n"
" 'No progress -'\n"
" ' stopping')\n"
" trace['errors'].append({\n"
" 'step': step+1,\n"
" 'type':\n"
" 'infinite_loop',\n"
" 'detail':\n"
" 'no progress'\n"
" })\n"
" trace['result'] = \\\n"
" 'Stopped: no progress'\n"
" return trace\n"
" else:\n"
" no_progress = 0\n"
" prev_output = \\\n"
" output_state\n"
" \n"
" trace['result'] = \\\n"
" 'Max steps reached'\n"
" trace['errors'].append({\n"
" 'step': max_steps,\n"
" 'type': 'max_steps',\n"
" 'detail':\n"
" 'limit reached'\n"
" })\n"
" return trace"
)
def get_langsmith_setup(self) -> str:
"""LangSmith setup (env vars only)."""
return (
"# LangSmith: framework-agnostic\n"
"# Just set env vars —\n"
"# no code changes needed\n"
"import os\n"
"\n"
"os.environ[\n"
" 'LANGSMITH_TRACING'] = 'true'\n"
"os.environ[\n"
" 'LANGSMITH_API_KEY'] = 'ls__...'\n"
"os.environ[\n"
" 'LANGSMITH_PROJECT'] = 'my-agent'\n"
"\n"
"# Now ALL OpenAI calls\n"
"# are automatically traced\n"
"# by LangSmith — visible\n"
"# in LangSmith UI with:\n"
"# - Each LLM call\n"
"# - Each tool invocation\n"
"# - Tokens per step\n"
"# - Latency per step\n"
"# - Complete trace tree\n"
"# - Cost per run\n"
"\n"
"# Works with:\n"
"# - LangChain agents\n"
"# - Custom Python code\n"
"# - OpenAI Agents SDK\n"
"# - Any LLM application"
)
def get_otel_setup(self) -> str:
"""OpenTelemetry instrumentation."""
return (
"from opentelemetry import trace\n"
"from opentelemetry.sdk.trace \\\n"
" import TracerProvider\n"
"from opentelemetry.sdk.trace\\\n"
" .export import (\n"
" BatchSpanProcessor,\n"
" ConsoleSpanExporter\n"
" )\n"
"\n"
"# Set up OTel\n"
"provider = TracerProvider()\n"
"provider.add_span_processor(\n"
" BatchSpanProcessor(\n"
" ConsoleSpanExporter()\n"
" )\n"
")\n"
"trace.set_tracer_provider(\n"
" provider)\n"
"tracer = trace.get_tracer(\n"
" 'ai-agent')\n"
"\n"
"# Instrument agent steps\n"
"def run_with_otel(\n"
" user_msg: str\n"
"):\n"
" with tracer.start_as_current_span(\n"
" 'agent_run'\n"
" ) as run_span:\n"
" run_span.set_attribute(\n"
" 'user.message',\n"
" user_msg)\n"
" \n"
" for step in range(\n"
" max_steps):\n"
" with tracer\\\n"
" .start_as_current_span(\n"
" f'step_{step+1}'\n"
" ) as step_span:\n"
" step_span\\\n"
" .set_attribute(\n"
" 'step.number',\n"
" step+1)\n"
" # ... agent logic\n"
" # Each span captures:\n"
" # - LLM call details\n"
" # - Tool calls\n"
" # - Tokens, latency\n"
" # - Errors"
)
def get_failure_patterns(self) -> dict:
"""Common failure patterns."""
return {
"infinite_loop": {
"symptom": "Agent repeats same tool call without progress",
"cause": "No termination condition or no-progress detection",
"fix": "max_steps limit, no-progress detection (compare output state), circuit breaker for failing tools",
"trace": "Look for repeated tool_calls with same args across steps",
},
"hallucination": {
"symptom": "Agent fabricates information or tool results",
"cause": "No output verification, model confabulates",
"fix": "Verify tool outputs, citation validation, cross-check with sources",
"trace": "Look for content not supported by tool results in messages",
},
"tool_error": {
"symptom": "Agent crashes when tool raises exception",
"cause": "No try/except around tool execution",
"fix": "try/except per tool, return error as tool result for LLM adaptation",
"trace": "Look for exceptions in error log, unhandled errors in trace",
},
"context_overflow": {
"symptom": "API error: context length exceeded",
"cause": "Too many messages in context window",
"fix": "Summarization, sliding window, reduce message history",
"trace": "Look for high token counts approaching model limit",
},
"wrong_tool": {
"symptom": "Agent calls wrong tool for the task",
"cause": "Poor tool descriptions, unclear system prompt",
"fix": "Better tool descriptions, clearer system prompt, parameter docs",
"trace": "Look for tool calls that don't match user intent",
},
"argument_error": {
"symptom": "Tool receives wrong argument types or values",
"cause": "No JSON schema validation, poor parameter descriptions",
"fix": "JSON schema validation before execution, better parameter descriptions",
"trace": "Look for TypeError/ValueError in tool execution",
},
}
def get_debug_workflow(self) -> dict:
"""Debug workflow."""
return {
"1_reproduce": (
"Reproduce the failure. "
"Use same input, same "
"model, same tools. "
"Capture the trace."
),
"2_trace": (
"Inspect trace in "
"LangSmith or logs. "
"Find the step where "
"things went wrong."
),
"3_isolate": (
"Isolate the failing "
"step. Was it LLM call, "
"tool call, or argument "
"parsing?"
),
"4_diagnose": (
"Diagnose root cause. "
"Check failure patterns "
"table. Match symptom "
"to cause."
),
"5_fix": (
"Apply fix. Test with "
"same input. Verify "
"fix resolves issue "
"without regressions."
),
"6_log": (
"Log the failure and "
"fix for future "
"reference. Update "
"checklist."
),
}
Debug AI Agent Checklist
- [ ] Debug AI agents with tracing, logging, and observability — not guesswork
- [ ] LangSmith: LLM observability — visibility into agent decisions, cost, latency, failures, hallucinations
- [ ] LangSmith: complete execution trace end-to-end — every LLM call, tool invocation, reasoning step
- [ ] LangSmith: framework-agnostic — works with LangChain, custom code, OpenAI SDK
- [ ] LangSmith: setup via env vars only — LANGSMITH_TRACING, LANGSMITH_API_KEY, LANGSMITH_PROJECT
- [ ] LangSmith: traces turn debugging into analysis — understand why agent took action
- [ ] OpenTelemetry: standardized instrumentation for GenAI — semantic conventions
- [ ] OpenTelemetry: major vendors converging on OTel-compatible instrumentation
- [ ] OpenTelemetry: LangSmith supports OTel to unify observability stack across services
- [ ] Structured logging: JSON format with step, tool, args, result, tokens, latency
- [ ] AgentPrism: React library for trace visualization interfaces
- [ ] AgentRx: in-depth failure diagnosis framework
- [ ] Step-level view: single prompt triggers 5-6 reasoning steps — see each individually
- [ ] TrueFoundry: "step-level view essential — see each step, not just final output"
- [ ] Common failure: infinite loop — agent repeats same tool call without progress
- [ ] Fix infinite loop: max_steps limit, no-progress detection (compare output state), circuit breaker
- [ ] Common failure: hallucination — agent fabricates information or tool results
- [ ] Fix hallucination: verify tool outputs, citation validation, cross-check with sources
- [ ] Common failure: tool error — agent crashes when tool raises exception
- [ ] Fix tool error: try/except per tool, return error as tool result for LLM adaptation
- [ ] Common failure: context overflow — too many messages exceed context window
- [ ] Fix context overflow: summarization, sliding window, reduce message history
- [ ] Common failure: wrong tool — agent calls wrong tool for the task
- [ ] Fix wrong tool: better tool descriptions, clearer system prompt, parameter docs
- [ ] Common failure: argument error — tool receives wrong argument types or values
- [ ] Fix argument error: JSON schema validation before execution, better parameter descriptions
- [ ] Traced agent: capture run_id, steps, total_tokens, total_latency, errors per run
- [ ] Traced agent: per step — step number, tokens, latency, tool_calls with name/args/result/error
- [ ] Traced agent: no-progress detection — compare output_state to prev_output, break after N
- [ ] Traced agent: error tracking — append to trace['errors'] with step, type, detail
- [ ] Debug workflow: (1) reproduce, (2) trace, (3) isolate, (4) diagnose, (5) fix, (6) log
- [ ] OTel instrumentation: spans for agent_run and each step, attributes for metadata
- [ ] LangSmith UI: trace tree with each LLM call, tool call, tokens, latency, cost
- [ ] Production: set up LangSmith tracing in all environments (dev, staging, prod)
- [ ] Production: alert on error rate, high token usage, high latency
- [ ] Read what is AI agent for architecture
- [ ] Read tool loop for loop debugging
- [ ] Read LangChain agent for LangSmith integration
- [ ] Read web search for citation debugging
- [ ] Test: trace captures all steps with metadata
- [ ] Test: no-progress detection stops infinite loops
- [ ] Test: tool errors captured in trace and returned to agent
- [ ] Test: LangSmith traces visible in UI with full trace tree
- [ ] Test: OTel spans capture agent run and step details
- [ ] Test: error log includes step, type, tool, error message
- [ ] Test: debug workflow reproduces and resolves failures
- [ ] Document tracing setup, failure patterns, debug workflow, alerting thresholds
FAQ
How do you debug an AI agent?
Use tracing, logging, and observability tools to inspect every LLM call, tool invocation, and reasoning step. LangSmith: "LLM observability platforms provide visibility into RAG pipelines, AI agent decisions, track model performance metrics like cost and latency, and help debug complex failures and hallucinations by showing the complete execution trace from end-to-end." DigitalOcean: "Observability is foundational for agent reliability. With LangSmith tracing, every LLM call, tool invocation, and intermediate reasoning step is observable, so you can understand why an agent took a certain action instead of having to infer it from the logs. Traces turn debugging into analysis." GoClaw: "Setting up observability includes: tagging telemetry with OpenTelemetry, standardizing log formats, using tools like AgentPrism for trace interfaces, AgentRx for failure diagnosis, and LangSmith for tracking lifecycles, costs, and tokens." Debug: (1) Tracing: capture every LLM call, tool invocation, reasoning step. (2) Logging: structured logs with step, tool, args, result, tokens. (3) Observability: end-to-end execution trace. (4) LangSmith: framework-agnostic, works with custom code. (5) OpenTelemetry: standardize instrumentation across services.
What is LangSmith for agent debugging?
LangSmith provides tracing, evaluation, and debugging for LLM agents — framework-agnostic. LangSmith: "LLM observability: visibility into RAG pipelines, AI agent decisions, track cost and latency, debug failures and hallucinations by showing complete execution trace end-to-end." LangChain Agent Observability: "LangSmith captures complete conversation traces. Semantic conventions for generative AI provide common vocabulary. Major vendors converging on OTel-compatible instrumentation. LangSmith supports OTel to unify observability stack across services." DigitalOcean: "Every LLM call, tool invocation, and intermediate reasoning step is observable. Understand why agent took action instead of inferring from logs. Traces turn debugging into analysis." LangSmith: (1) Tracing: every LLM call, tool call, reasoning step captured. (2) Framework-agnostic: works with LangChain, custom code, OpenAI SDK. (3) Setup: env vars only (LANGSMITH_TRACING, LANGSMITH_API_KEY). (4) Metrics: cost, latency, tokens per step. (5) Evaluation: test agent outputs against expected. (6) OTel: supports OpenTelemetry for unified observability.
What are common AI agent failure patterns?
Infinite loops, hallucinations, tool errors, context overflow, and wrong tool selection are common failures. TrueFoundry: "In an agentic workflow, a single user prompt can trigger five or six internal reasoning steps. A step-level view is essential — you need to see each step individually, not just the final output." GoClaw: "AgentRx: in-depth failure diagnosis framework. Common failures require structured diagnosis." LangSmith: "Debug complex failures and hallucinations by showing complete execution trace end-to-end." Failures: (1) Infinite loops: agent repeats same tool call without progress. Fix: max_steps, no-progress detection. (2) Hallucinations: agent fabricates information or tool results. Fix: verify tool outputs, citation validation. (3) Tool errors: tool raises exception, agent does not handle. Fix: try/except, return error as result. (4) Context overflow: too many messages exceed context window. Fix: summarization, sliding window. (5) Wrong tool selection: agent calls wrong tool for task. Fix: better tool descriptions, system prompt. (6) Argument errors: agent passes wrong arguments to tool. Fix: JSON schema validation, better parameter descriptions.
How do you set up observability for AI agents?
Use LangSmith tracing via env vars, OpenTelemetry for standardized instrumentation, and structured logging. LangChain Agent Observability: "Semantic conventions for generative AI systems provide common vocabulary for describing LLM operations. Major vendors converging on OTel-compatible instrumentation. LangSmith supports OTel to unify observability stack across services." GoClaw: "Setting up observability: tagging telemetry with OpenTelemetry, standardizing log formats, AgentPrism for trace interfaces, AgentRx for failure diagnosis, LangSmith for tracking lifecycles, costs, and tokens." TrueFoundry: "Step-level view essential — see each reasoning step individually, not just final output." Setup: (1) LangSmith: set LANGSMITH_TRACING=true, LANGSMITH_API_KEY, LANGSMITH_PROJECT env vars. (2) OpenTelemetry: instrument with OTel SDK, use semantic conventions for GenAI. (3) Structured logging: JSON logs with step, tool, args, result, tokens, latency. (4) AgentPrism: React library for trace visualization. (5) AgentRx: framework for in-depth failure diagnosis. (6) Step-level: capture each reasoning step, not just final output.
How do you trace agent tool calls and reasoning steps?
Capture each step with metadata: step number, tool name, arguments, result, tokens, latency, and reasoning. DigitalOcean: "Every LLM call, tool invocation, and intermediate reasoning step is observable. Understand why agent took action instead of inferring from logs. Traces turn debugging into analysis." TrueFoundry: "A single user prompt can trigger five or six internal reasoning steps. A step-level view is essential — see each step individually." LangSmith: "Complete execution trace from end-to-end. Track cost and latency." Tracing: (1) Step number: increment per iteration. (2) LLM call: model, messages, response, tokens, latency. (3) Tool call: tool name, arguments, result, error. (4) Reasoning: LLM thought process (if visible). (5) State: messages list length, context window usage. (6) Metadata: timestamp, run_id, thread_id. (7) LangSmith: automatic tracing via env vars. (8) Manual: logging module with structured JSON. (9) Visualize: AgentPrism or LangSmith UI for trace inspection.
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →