AI API Monitoring: OpenTelemetry, Prometheus, and Grafana for LLM Observability in 2026
TL;DR — AI API monitoring: OpenTelemetry, Prometheus, Grafana. Teknasyon: "FastAPI serving, OpenTelemetry tracing and metrics, Prometheus + Grafana. Track tokens in/out, API latencies. Tracer, counters, histograms, Prometheus exporter." Dev.to: "Context manager wraps LLM call: latency, tokens, cost, finish_reason. finish_reason=length as error. Cost by feature, P99 by operation, error by type." DevOpsBoys: "GenAI semantic conventions: operation.duration, token.usage, client.cost. Token counts, model, prompt version, cache hit, quality scores." OGX: "Five metric domains: API requests, inference (TTFT, tokens/sec), vector IO, tool runtime, safety. PromQL for P95/P99, token usage, error rates." Learn more with FastAPI tutorial, async backend, Docker deployment, and rate limiting.
DevOpsBoys frames the problem: "Running LLMs in production without observability is flying blind. You need to answer: Which model is slowest? Which prompts are most expensive? What's the error rate? Where are the latency spikes?"
Dev.to adds the LLM-specific challenge: "Standard application monitoring was not built for this. Your existing latency dashboards will show LLM calls as outliers. Your error rate alerts will fire on model refusals that aren't actually errors. Your cost monitoring won't exist at all unless you build it."
AI API Monitoring Architecture
POST /v1/chat
POST /v1/embed
GET /health"] OTel["OpenTelemetry SDK
TracerProvider
MeterProvider
auto-instrumentation
manual spans"] end subgraph Metrics["LLM Metrics"] Latency["Latency Metrics
TTFT (time to first token)
total response time
P50, P95, P99
histogram"] Tokens["Token Metrics
input tokens
output tokens
cached tokens
tokens per second
counter"] Cost["Cost Metrics
per-call cost USD
cost by feature
cost by model
hourly aggregation"] Errors["Error Metrics
error type: rate_limit
error type: timeout
error type: content_filter
finish_reason: length
error rate %"] end subgraph Tracing["Distributed Tracing"] Spans["LLM Spans
gen_ai.request.model
gen_ai.token.usage
gen_ai.cost
gen_ai.finish_reason
gen_ai.error_type"] Flow["Request Flow
HTTP request span
LLM call span
tool call span
vector store span
parent/child links"] end subgraph Backend["Observability Backend"] Prom["Prometheus
scrape /metrics
time-series storage
PromQL queries
alerting rules"] Grafana["Grafana
dashboards
panels: tokens, latency
P95/P99 histograms
cost, error rate
real-time visualization"] Jaeger["Jaeger
distributed traces
request flow
span details
debug latency"] end subgraph Alerts["Alerting"] Rules["Alert Rules
P99 latency >10s
error rate >5%
cost >$50/hour
truncation >5%
rate_limit >10/min"] Manager["Alertmanager
routing rules
notification channels
Slack, email, PagerDuty
escalation policies"] end Endpoints --> OTel OTel --> Latency OTel --> Tokens OTel --> Cost OTel --> Errors OTel --> Spans Spans --> Flow Latency --> Prom Tokens --> Prom Cost --> Prom Errors --> Prom Prom --> Grafana Flow --> Jaeger Prom --> Rules Rules --> Manager style App fill:#4169E1,color:#fff style Metrics fill:#39FF14,color:#000 style Tracing fill:#2D1B69,color:#fff style Backend fill:#FF6B6B,color:#fff
LLM Metrics Reference
| Metric | Type | Unit | Description |
|---|---|---|---|
| inference.duration | Histogram | s | End-to-end LLM latency |
| inference.ttft | Histogram | s | Time to first token |
| inference.tokens_per_sec | Histogram | - | Output token throughput |
| token.usage | Counter | token | Input/output/cached tokens |
| cost | Counter | USD | Estimated cost per call |
| requests.total | Counter | 1 | HTTP requests by status |
| request.duration | Histogram | s | HTTP latency by endpoint |
| concurrent.requests | Gauge | 1 | In-flight requests |
| errors.total | Counter | 1 | Errors by type |
| tool.invocations | Counter | 1 | Tool calls by name/status |
Implementation
from dataclasses import dataclass
from typing import Optional
from enum import Enum
class MonitorComponent(Enum):
OTEL = "otel"
METRICS = "metrics"
GRAFANA = "grafana"
ALERTS = "alerts"
@dataclass
class AIApiMonitoringGuide:
"""AI API monitoring implementation guide."""
def get_otel_setup(self) -> str:
"""OpenTelemetry setup for FastAPI."""
return (
"# === OPENTELEMETRY SETUP ===\n"
"# pip install opentelemetry-sdk\n"
"# opentelemetry-exporter-prometheus\n"
"# prometheus_client\n"
"\n"
"from opentelemetry import (\n"
" trace, metrics)\n"
"from opentelemetry.sdk.trace import (\n"
" TracerProvider)\n"
"from opentelemetry.sdk.metrics import (\n"
" MeterProvider)\n"
"from opentelemetry.sdk.trace.\n"
" export import (\n"
" BatchSpanProcessor)\n"
"from opentelemetry.exporter.\n"
" prometheus import (\n"
" PrometheusMetricReader)\n"
"from prometheus_client import (\n"
" make_asgi_app)\n"
"from fastapi import FastAPI\n"
"import time, httpx\n"
"\n"
"# Tracer setup\n"
"tracer_provider = (\n"
" TracerProvider())\n"
"trace.set_tracer_provider(\n"
" tracer_provider)\n"
"tracer = trace.get_tracer(\n"
" 'llm-service')\n"
"\n"
"# Metrics setup\n"
"reader = PrometheusMetricReader(\n"
" 'ai_api')\n"
"meter_provider = MeterProvider(\n"
" metric_readers=[reader])\n"
"metrics.set_meter_provider(\n"
" meter_provider)\n"
"meter = metrics.get_meter(\n"
" 'llm-service')\n"
"\n"
"# Define metrics\n"
"llm_duration = (\n"
" meter.create_histogram(\n"
" 'gen_ai.client'\n"
" '.operation.duration',\n"
" unit='s',\n"
" description=\n"
" 'LLM call duration'))\n"
"\n"
"llm_tokens = (\n"
" meter.create_counter(\n"
" 'gen_ai.client'\n"
" '.token.usage',\n"
" unit='{token}',\n"
" description=\n"
" 'Token usage'))\n"
"\n"
"llm_cost = (\n"
" meter.create_counter(\n"
" 'gen_ai.client.cost',\n"
" unit='USD',\n"
" description=\n"
" 'Estimated cost'))\n"
"\n"
"llm_errors = (\n"
" meter.create_counter(\n"
" 'gen_ai.client.errors',\n"
" description=\n"
" 'LLM errors by type'))\n"
"\n"
"app = FastAPI()\n"
"# Mount Prometheus endpoint\n"
"app.mount(\n"
" '/metrics',\n"
" make_asgi_app())"
)
def get_llm_instrumentation(self) -> str:
"""Instrument LLM calls with tracing and metrics."""
return (
"# === LLM INSTRUMENTATION ===\n"
"\n"
"TOKEN_COSTS = {\n"
" 'llama3.2': {\n"
" 'input': 0.15,\n"
" 'output': 0.15},\n"
" 'qwen2.5': {\n"
" 'input': 0.10,\n"
" 'output': 0.10},\n"
"}\n"
"\n"
"async def call_llm_traced(\n"
" prompt: str,\n"
" model: str = 'llama3.2',\n"
" feature: str = 'chat'):\n"
" '''Call LLM with full\n"
" observability.''' \n"
" with tracer.start_as_current_span(\n"
" 'llm.inference'\n"
" ) as span:\n"
" # Set span attributes\n"
" span.set_attribute(\n"
" 'gen_ai.request.model',\n"
" model)\n"
" span.set_attribute(\n"
" 'gen_ai.feature',\n"
" feature)\n"
" span.set_attribute(\n"
" 'gen_ai.operation',\n"
" 'chat')\n"
" \n"
" start = time.time()\n"
" ttft = None\n"
" \n"
" try:\n"
" async with httpx.AsyncClient(\n"
" timeout=300.0\n"
" ) as client:\n"
" # Stream for TTFT\n"
" async with client.stream(\n"
" 'POST',\n"
" 'http://'\n"
" 'localhost:11434'\n"
" '/api/chat',\n"
" json={\n"
" 'model': model,\n"
" 'messages': [{\n"
" 'role':'user',\n"
" 'content':prompt}],\n"
" 'stream': True}\n"
" ) as resp:\n"
" full = ''\n"
" input_tokens = 0\n"
" output_tokens = 0\n"
" finish_reason = 'stop'\n"
" \n"
" async for line in (\n"
" resp\n"
" .aiter_lines()):\n"
" if not line:\n"
" continue\n"
" chunk = (\n"
" __import__('json')\n"
" .loads(line))\n"
" \n"
" if ttft is None:\n"
" ttft = (\n"
" time.time()\n"
" - start)\n"
" \n"
" token = (\n"
" chunk.get(\n"
" 'message',{})\n"
" .get(\n"
" 'content',''))\n"
" if token:\n"
" full += token\n"
" output_tokens += 1\n"
" \n"
" if chunk.get('done'):\n"
" input_tokens = (\n"
" chunk.get(\n"
" 'prompt_count',0))\n"
" output_tokens = (\n"
" chunk.get(\n"
" 'eval_count',\n"
" output_tokens))\n"
" break\n"
" \n"
" duration = (\n"
" time.time() - start)\n"
" \n"
" # Record metrics\n"
" llm_duration.record(\n"
" duration,\n"
" {'model': model,\n"
" 'feature': feature})\n"
" \n"
" if ttft:\n"
" span.set_attribute(\n"
" 'gen_ai.ttft_s',\n"
" ttft)\n"
" \n"
" llm_tokens.add(\n"
" input_tokens,\n"
" {'model': model,\n"
" 'type':'input'})\n"
" llm_tokens.add(\n"
" output_tokens,\n"
" {'model': model,\n"
" 'type':'output'})\n"
" \n"
" # Cost estimate\n"
" costs = (\n"
" TOKEN_COSTS.get(\n"
" model, {}))\n"
" cost = (\n"
" input_tokens * costs\n"
" .get('input',0)\n"
" + output_tokens * costs\n"
" .get('output',0)\n"
" ) / 1_000_000\n"
" llm_cost.add(\n"
" cost,\n"
" {'model': model,\n"
" 'feature': feature})\n"
" \n"
" span.set_attribute(\n"
" 'gen_ai.token.input',\n"
" input_tokens)\n"
" span.set_attribute(\n"
" 'gen_ai.token.output',\n"
" output_tokens)\n"
" span.set_attribute(\n"
" 'gen_ai.cost_usd',\n"
" cost)\n"
" span.set_attribute(\n"
" 'gen_ai.finish_reason',\n"
" finish_reason)\n"
" \n"
" # finish_reason=length\n"
" # = error\n"
" if (finish_reason\n"
" == 'length'):\n"
" span.set_attribute(\n"
" 'error', True)\n"
" span.set_attribute(\n"
" 'gen_ai.error_type',\n"
" 'truncated')\n"
" llm_errors.add(1,\n"
" {'type':'truncated',\n"
" 'model':model})\n"
" \n"
" return {\n"
" 'response': full,\n"
" 'tokens': {\n"
" 'input':\n"
" input_tokens,\n"
" 'output':\n"
" output_tokens},\n"
" 'cost': cost,\n"
" 'duration': duration,\n"
" 'ttft': ttft}\n"
" \n"
" except httpx.ReadTimeout:\n"
" span.set_attribute(\n"
" 'error', True)\n"
" span.set_attribute(\n"
" 'gen_ai.error_type',\n"
" 'timeout')\n"
" llm_errors.add(1,\n"
" {'type':'timeout',\n"
" 'model':model})\n"
" raise\n"
" except httpx.ConnectError:\n"
" span.set_attribute(\n"
" 'error', True)\n"
" span.set_attribute(\n"
" 'gen_ai.error_type',\n"
" 'connection')\n"
" llm_errors.add(1,\n"
" {'type':'connection',\n"
" 'model':model})\n"
" raise"
)
def get_grafana_queries(self) -> str:
"""PromQL queries for Grafana dashboards."""
return (
"# === GRAFANA PROMQL QUERIES ===\n"
"\n"
"# P99 latency by model\n"
"# histogram_quantile(0.99,\n"
"# rate(gen_ai_client_operation\n"
"# _duration_bucket[5m]))\n"
"# by (gen_ai_request_model)\n"
"\n"
"# P95 HTTP server latency\n"
"# histogram_quantile(0.95,\n"
"# rate(ogx_http_server_duration\n"
"# _milliseconds_bucket[5m]))\n"
"\n"
"# P99 inference duration\n"
"# histogram_quantile(0.99,\n"
"# rate(ogx_inference_duration\n"
"# _seconds_bucket[5m]))\n"
"\n"
"# P95 TTFT by model\n"
"# histogram_quantile(0.95,\n"
"# rate(ogx_inference_time_to\n"
"# _first_token_seconds\n"
"# _bucket[5m]))\n"
"\n"
"# Input token usage by model\n"
"# sum by(gen_ai_request_model)(\n"
"# gen_ai_client_token_usage_sum\n"
"# {gen_ai_token_type=\"input\"})\n"
"\n"
"# Output token usage by model\n"
"# sum by(gen_ai_request_model)(\n"
"# gen_ai_client_token_usage_sum\n"
"# {gen_ai_token_type=\"output\"})\n"
"\n"
"# Cost per hour by feature\n"
"# sum(rate(\n"
"# gen_ai_client_cost_total[1h]))\n"
"# by (feature) * 3600\n"
"\n"
"# Error rate\n"
"# rate(gen_ai_client_errors\n"
"# _total[5m])\n"
"# / rate(gen_ai_client\n"
"# _requests_total[5m])\n"
"\n"
"# Cache hit rate\n"
"# sum(rate(\n"
"# gen_ai_client_token_usage\n"
"# _total{gen_ai_token_type\n"
"# =\"cached\"}[5m]))\n"
"# / sum(rate(\n"
"# gen_ai_client_token_usage\n"
"# _total{gen_ai_token_type\n"
"# =\"input\"}[5m]))\n"
"\n"
"# Median tokens/sec\n"
"# histogram_quantile(0.5,\n"
"# rate(ogx_inference_tokens\n"
"# _per_second_bucket[5m]))\n"
"\n"
"# Tool invocation errors\n"
"# rate(ogx_tool_runtime\n"
"# _invocations_total\n"
"# {status=\"error\"}[5m])"
)
def get_alerts(self) -> str:
"""Alertmanager rules for LLM monitoring."""
return (
"# === ALERTMANAGER RULES ===\n"
"# alertmanager_rules.yml\n"
"\n"
"groups:\n"
" - name: llm_alerts\n"
" rules:\n"
" - alert: LLMHighLatency\n"
" expr: >\n"
" histogram_quantile(0.99,\n"
" rate(\n"
" gen_ai_client_operation\n"
" _duration_bucket[5m])\n"
" ) > 10\n"
" for: 5m\n"
" annotations:\n"
" summary:\n"
" 'LLM P99 >10s'\n"
" description:\n"
" 'P99 latency above 10s\n"
" for 5 minutes'\n"
" \n"
" - alert: LLMHighErrorRate\n"
" expr: >\n"
" rate(\n"
" gen_ai_client_errors\n"
" _total[5m])\n"
" / rate(\n"
" gen_ai_client\n"
" _requests_total[5m])\n"
" > 0.05\n"
" for: 5m\n"
" annotations:\n"
" summary:\n"
" 'Error rate >5%'\n"
" \n"
" - alert: LLMHighCost\n"
" expr: >\n"
" increase(\n"
" gen_ai_client_cost\n"
" _total[1h]) > 50\n"
" annotations:\n"
" summary:\n"
" 'LLM spend >$50/hour'\n"
" \n"
" - alert: LLMTruncationSpike\n"
" expr: >\n"
" rate(\n"
" gen_ai_client_errors\n"
" _total\n"
" {type='truncated'}[5m])\n"
" / rate(\n"
" gen_ai_client\n"
" _requests_total[5m])\n"
" > 0.05\n"
" annotations:\n"
" summary:\n"
" 'Truncation >5%'\n"
" \n"
" - alert: LLMRateLimited\n"
" expr: >\n"
" rate(\n"
" gen_ai_client_errors\n"
" _total\n"
" {type='rate_limit'}[1m])\n"
" > 10\n"
" annotations:\n"
" summary:\n"
" 'Rate limit >10/min'\n"
" \n"
" - alert: LLMProviderDown\n"
" expr: >\n"
" rate(\n"
" gen_ai_client\n"
" _requests_total\n"
" {status='success'}[2m])\n"
" == 0\n"
" for: 2m\n"
" annotations:\n"
" summary:\n"
" 'Provider down'"
)
def get_production_tips(self) -> dict:
"""Production tips."""
return {
"otel": "OpenTelemetry SDK: TracerProvider, MeterProvider. Auto-instrumentation for HTTP, manual for LLM-specific metrics.",
"metrics": "LLM metrics: latency (TTFT, total), tokens (input/output/cached), cost, finish_reason, error_type, cache hit rate.",
"finish_reason": "finish_reason=length = truncated response. Treat as error in span. Alert on truncation spike >5%.",
"error_types": "Group errors by type: rate_limit, timeout, content_filter, connection. Different remediation paths.",
"p99": "LLM latency distributions are wide. P50 might be 800ms, P99 12s. Alert on P99, not average.",
"cost": "Track cost per call and by feature. One or two features often drive majority of spend.",
"promql": "histogram_quantile for percentiles, rate for counters. Sum by model, feature, error_type.",
"grafana": "Dashboard panels: request count, latency P95/P99, tokens in/out, cost, error rate. Real-time.",
"alerts": "P99 >10s, error >5%, cost >$50/h, truncation >5%, rate_limit >10/min, provider down 2 min.",
"docker": "Docker Compose: Prometheus + Grafana + Jaeger + OTel Collector. One-command setup.",
}
AI API Monitoring Checklist
- [ ] OpenTelemetry SDK: TracerProvider for distributed tracing, MeterProvider for metrics
- [ ] Auto-instrumentation captures HTTP requests, database queries, GenAI calls
- [ ] Manual instrumentation for domain-specific metrics: inference latency, tool execution, vector store
- [ ] Export via OTLP to any backend: Prometheus, Grafana, Jaeger, Datadog, Tempo
- [ ] Prometheus metric reader: expose /metrics endpoint for scraping
- [ ] Prometheus scrapes FastAPI /metrics endpoint — time-series storage
- [ ] Grafana: import dashboard, add Prometheus data source, build panels
- [ ] Grafana panels: request count, latency P95/P99, tokens in/out, cost, error rate
- [ ] Jaeger: distributed traces, request flow, span details, debug latency
- [ ] Docker Compose: Prometheus + Grafana + Jaeger + OTel Collector — one-command setup
- [ ] LLM latency metrics: TTFT (time to first token), total response time, P50/P95/P99
- [ ] LLM latency distributions are wide — P50 might be 800ms while P99 is 12s
- [ ] Alert on P99, not average — catches tail latency users actually experience
- [ ] Token metrics: input tokens, output tokens, cached tokens — separately (different costs)
- [ ] Token throughput: tokens per second as histogram
- [ ] Cost metrics: per-call cost in USD, aggregated by feature and model
- [ ] Cost by feature: group spans by llm.feature, sum llm.estimated_cost_usd
- [ ] One or two features often account for majority of LLM spend
- [ ] Model used: track which model served each request
- [ ] Prompt version: track for A/B testing prompts
- [ ] Cache hit rate: prompt caching effectiveness
- [ ] Finish reason: stop, length, content_filter, tool_calls — each indicates different condition
- [ ] finish_reason=length: response truncated by token limit — treat as error in span
- [ ] Most monitoring records truncated response as success (HTTP 200) — but response is incomplete
- [ ] Alert on truncation spike: finish_reason=length rate >5% of calls
- [ ] Error types: rate_limit, timeout, content_filter, connection — different remediation paths
- [ ] Group errors by type — grouping them together hides what's actually wrong
- [ ] llm.error_type attribute on spans for filtering
- [ ] llm.is_retry and llm.attempt for retry tracking — filter cost dashboard to exclude retries
- [ ] Concurrent requests gauge for capacity monitoring
- [ ] Tool invocation metrics: count and duration by tool name and status
- [ ] Vector IO metrics: insert, query, delete counts with duration
- [ ] Safety spans: shield evaluation traces with attribute context
- [ ] GenAI semantic conventions: gen_ai.client.operation.duration, gen_ai.client.token.usage, gen_ai.client.cost
- [ ] PromQL: histogram_quantile for percentiles, rate for counters, sum by model/feature/error_type
- [ ] Structured logging with trace_id and span_id for correlation
- [ ] Jaeger ingests traces only, not logs — use OTel Collector to route logs
- [ ] Alert: P99 latency >10s sustained 5 minutes
- [ ] Alert: error rate >5% over 5-minute window
- [ ] Alert: cost >$50/hour or >2x baseline
- [ ] Alert: truncation rate >5% of calls
- [ ] Alert: rate_limit errors >10 per minute
- [ ] Alert: content_filter >3 per hour
- [ ] Alert: provider down — zero successful requests for 2 minutes
- [ ] Alert: concurrent requests above capacity threshold
- [ ] Alertmanager: routing rules, notification channels (Slack, email, PagerDuty)
- [ ] Read FastAPI tutorial for API setup
- [ ] Read async backend for async patterns
- [ ] Read Docker deployment for deployment
- [ ] Read rate limiting for API protection
- [ ] Test: /metrics endpoint returns Prometheus format
- [ ] Test: Grafana dashboard shows real-time metrics
- [ ] Test: traces appear in Jaeger with LLM span attributes
- [ ] Test: cost tracking matches expected token costs
- [ ] Test: finish_reason=length marked as error in span
- [ ] Test: alerts fire on threshold breach
- [ ] Test: error types are grouped separately in dashboard
- [ ] Document metrics schema, PromQL queries, alert thresholds, dashboard layout, OTel config
FAQ
How do you monitor LLM API calls with OpenTelemetry in Python?
Instrument LLM calls with OpenTelemetry traces and metrics for latency, tokens, cost, and finish reason. Teknasyon: "Build LLM agent observable from day one. FastAPI serving, OpenTelemetry tracing and metrics, Prometheus + Grafana monitoring. Track tokens in/out, API latencies. Tracer for distributed tracing, counters for tokens and errors, histograms for latency, Prometheus exporter for scraping." Dev.to: "Context manager wraps any LLM call, captures latency, token consumption, estimated cost, finish reason. finish_reason == length as error in span — alert on it separately. Cost by feature, P99 latency by operation, error rate by type." DevOpsBoys: "GenAI semantic conventions: gen_ai.client.operation.duration, gen_ai.client.token.usage, gen_ai.client.cost. Token costs, model used, prompt version, cache hit rate, quality scores." Instrumentation: (1) OpenTelemetry SDK with MeterProvider and TracerProvider. (2) Histograms for latency (operation duration, TTFT). (3) Counters for tokens (input, output, cached) and cost. (4) Spans with attributes: model, finish_reason, token counts. (5) Export to Prometheus + Jaeger + Grafana.
What LLM-specific metrics should you track in production?
Track latency (TTFT, total), token counts (input, output, cached), cost per request, model used, finish reason, error type, and cache hit rate. DevOpsBoys: "Standard: latency (TTFT, total response time), error rate, throughput. LLM-specific: token counts (input, output, cached), cost per request, model used, prompt version, cache hit rate, quality scores." OGX: "API request metrics: total count, duration histogram, concurrent request gauge. Inference metrics: end-to-end duration, TTFT, tokens-per-second. Vector IO metrics: insert, query, delete counts. Tool runtime metrics: invocation count and duration by tool name and status." Metrics: (1) Latency: TTFT, total response time, P50/P95/P99. (2) Tokens: input, output, cached — separately (different costs). (3) Cost: per-call and aggregated by feature. (4) Model: which model used. (5) Finish reason: stop, length, content_filter, tool_calls. (6) Error type: rate_limit, timeout, content_filter — different remediation. (7) Cache hit rate. (8) Throughput: requests/second, tokens/second. (9) Concurrent requests gauge. (10) Quality scores if available.
How do you set up Prometheus and Grafana for AI API monitoring?
Expose metrics via Prometheus endpoint, scrape with Prometheus, visualize in Grafana dashboards. Teknasyon: "Prometheus scrapes metrics from FastAPI service. Grafana dashboard panels: request count, request latency P95, LLM token out, LLM token in. Real-time visualization of traffic, latency, token usage." OGX: "PromQL queries: input token usage by model, P95 HTTP latency, P99 inference duration, P95 TTFT by model, median tokens/sec, tool invocation errors. Auto-provisioned Grafana dashboard with panels for prompt tokens, completion tokens, P95/P99 HTTP duration, request volume." Setup: (1) FastAPI /metrics endpoint with prometheus_client. (2) Prometheus config: scrape target. (3) Grafana: import dashboard, add Prometheus data source. (4) Panels: request count, latency P95/P99, tokens in/out, cost, error rate. (5) PromQL: histogram_quantile for percentiles, rate for counters. (6) Docker Compose: Prometheus + Grafana + Jaeger.
How do you handle LLM finish_reason and error types in monitoring?
Treat finish_reason=length as error in spans, group errors by type for different remediation paths. Dev.to: "finish_reason handling: when LLM response truncated by token limit, most monitoring records as successful (HTTP 200). But from product perspective, response is incomplete. Treating finish_reason == length as error in span means you can alert on it separately. Error rate by error type: rate limits, timeouts, content filters have completely different remediation paths. Grouping them hides what is wrong." DevOpsBoys: "Error rate: API failures, timeouts. LLM-specific: finish reason (stop, length, content_filter, tool_calls)." Handling: (1) finish_reason=length: set span error, alert on truncation spike >5%. (2) Error types: rate_limit, timeout, content_filter, api_error. (3) Group by type for different remediation. (4) llm.error_type attribute on spans. (5) Alert: rate_limit >10/min, content_filter >3/hour. (6) llm.is_retry and llm.attempt for retry tracking. (7) Don't group all errors together — hides root cause.
What alerts should you configure for AI API monitoring?
Alert on P99 latency, error rate, cost spike, truncation rate, and provider downtime. Dev.to: "Alerts: High latency P99 >10,000ms, Truncation spike finish_reason=length >5% of calls, Rate limiting >10 per minute, Cost spike >2x baseline per hour, Content filter >3 per hour." DevOpsBoys: "Alertmanager rules: LLMHighLatency P95 >10s, LLMHighCost >$50/hour, LLMHighErrorRate >5%." OGX: "High latency: P99 inference >10s sustained 5 min. Error rate spike: >5% over 5 min. Provider down: zero successful requests for 2 min. Capacity warning: concurrent requests above threshold." Alerts: (1) P99 latency >10s sustained 5 min. (2) Error rate >5% over 5 min. (3) Cost >$50/hour or >2x baseline. (4) Truncation rate >5%. (5) Rate limit errors >10/min. (6) Content filter >3/hour. (7) Provider down: zero success for 2 min. (8) Concurrent requests above threshold.
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →