AI Platform Health Monitoring: Metrics, Alerts, and Dashboards

TL;DR — AI platform health monitoring requires four metric categories: infrastructure (GPU, VRAM, CPU), LLM inference (latency p50/p90/p99, tokens/sec, error rate), RAG pipeline (retrieval latency, recall, cache hit rate, hallucination rate), and business (cost/query, user satisfaction). Use Prometheus + Grafana for infrastructure and inference metrics. Use Langfuse or LangSmith for AI-specific observability (tracing, faithfulness scoring, drift detection). Key alerts: p99 latency > 30s, GPU memory > 90%, error rate > 1%, hallucination rate > 2%, cache hit < 20%. Ollama exposes /metrics endpoint in Prometheus format. Deploy monitoring as part of Docker Compose AI platform. Traditional APM catches 500 errors; AI observability catches quality degradation — when your LLM starts hallucinating more, when retrieval quality drops, when prompt drift affects outputs. Both are needed for production AI.

Traditional APM catches latency spikes and 500 errors. AI observability should alert on quality degradation — faithfulness drops, safety regressions, drift across prompts. Your existing monitoring tells you when the server is down. AI observability tells you when your LLM starts hallucinating.

For self-hosted AI platforms, this means two monitoring stacks: Prometheus + Grafana for infrastructure, and Langfuse or LangSmith for AI quality. This post covers both.

Monitoring Architecture

Metric Categories

Infrastructure Metrics

LLM Inference Metrics

RAG Pipeline Metrics

Business Metrics

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Ollama LLM inference metrics
  - job_name: 'ollama'
    static_configs:
      - targets: ['ollama:11434']
    metrics_path: /metrics

  # FastAPI RAG pipeline metrics
  - job_name: 'fastapi'
    static_configs:
      - targets: ['api:8000']
    metrics_path: /metrics

  # PostgreSQL metrics
  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres-exporter:9187']

  # FalkorDB metrics
  - job_name: 'falkordb'
    static_configs:
      - targets: ['falkordb:9121']

  # Node exporter (system metrics)
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

# Alerting rules
rule_files:
  - 'alerts.yml'

Grafana Alert Rules

# alerts.yml
groups:
  - name: ai_critical
    rules:
      - alert: LLMHighLatencyP99
        expr: histogram_quantile(0.99, ollama_request_duration_seconds_bucket) > 30
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "LLM p99 latency > 30s"

      - alert: GPUMemoryHigh
        expr: ollama_gpu_memory_used_bytes / ollama_gpu_memory_total_bytes > 0.9
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "GPU memory > 90%"

      - alert: LLMErrorRate
        expr: rate(ollama_errors_total[5m]) / rate(ollama_requests_total[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "LLM error rate > 1%"

      - alert: OllamaDown
        expr: up{job="ollama"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Ollama is down"

  - name: ai_quality
    rules:
      - alert: HighHallucinationRate
        expr: avg_over_time(langfuse_faithfulness_score[1h]) < 0.95
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Hallucination rate > 5%"

      - alert: LowCacheHitRate
        expr: rate(cache_hits_total[1h]) / rate(cache_requests_total[1h]) < 0.2
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Cache hit rate < 20%"

      - alert: RetrievalLatencyHigh
        expr: histogram_quantile(0.99, retrieval_duration_seconds_bucket) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Retrieval p99 > 500ms"

FastAPI Metrics Endpoint

from prometheus_client import Counter, Histogram, Gauge, generate_latest
from fastapi import FastAPI, Response

app = FastAPI()

# Metrics definitions
REQUEST_LATENCY = Histogram(
    'rag_request_duration_seconds',
    'RAG request latency',
    buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0]
)
RETRIEVAL_LATENCY = Histogram(
    'retrieval_duration_seconds',
    'Vector retrieval latency',
    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.0]
)
CACHE_HITS = Counter('cache_hits_total', 'Cache hits')
CACHE_REQUESTS = Counter('cache_requests_total', 'Total cache requests')
HALLUCINATION_SCORE = Gauge('langfuse_faithfulness_score', 'Avg faithfulness score')
ACTIVE_QUERIES = Gauge('active_queries', 'Currently processing queries')

@app.get("/metrics")
async def metrics():
    return Response(generate_latest(), media_type="text/plain")

@app.post("/query")
async def query(req: QueryRequest):
    CACHE_REQUESTS.inc()
    ACTIVE_QUERIES.inc()

    with REQUEST_LATENCY.time():
        # Check cache
        cached = check_cache(req.query)
        if cached:
            CACHE_HITS.inc()
            return cached

        # Retrieval
        with RETRIEVAL_LATENCY.time():
            results = retrieve(req.query)

        # Generate
        answer = generate(req.query, results)

        # Evaluate faithfulness (async)
        score = await evaluate_faithfulness(answer, results)
        HALLUCINATION_SCORE.set(score)

    ACTIVE_QUERIES.dec()
    return {"answer": answer, "sources": results}

AI Observability vs Traditional Monitoring

AI Platform Health Monitoring Checklist

• [ ] Deploy Prometheus + Grafana as part of Docker Compose stack
• [ ] Configure Ollama /metrics endpoint for LLM inference metrics
• [ ] Configure FastAPI /metrics endpoint for RAG pipeline metrics
• [ ] Set up node_exporter for system metrics (CPU, memory, disk, network)
• [ ] Set up postgres_exporter for database metrics
• [ ] Create Grafana dashboards: infrastructure, LLM inference, RAG pipeline, business
• [ ] Set critical alerts: p99 latency > 30s, GPU > 90%, error rate > 1%, Ollama down
• [ ] Set quality alerts: hallucination rate > 2%, cache hit < 20%, retrieval p99 > 500ms
• [ ] Set capacity alerts: disk > 80%, queue > 50, daily cost > budget
• [ ] Route critical alerts to PagerDuty, quality to Slack, capacity to email
• [ ] Deploy Langfuse for AI-specific observability (tracing, evaluation)
• [ ] Implement faithfulness scoring: break response into claims, verify against context
• [ ] Track hallucination rate over time
• [ ] Monitor cache hit rate for cost optimization
• [ ] Track retrieval recall@10 and context relevance scores
• [ ] Monitor cost per query and daily total cost
• [ ] Track user satisfaction (thumbs up/down) in chat UI
• [ ] Set up Flutter app error reporting
• [ ] Monitor pgvector query latency separately
• [ ] Track FalkorDB graph query latency
• [ ] Monitor LiteLLM spend per team/model
• [ ] Set up audit log monitoring for compliance
• [ ] Create SLO: 99.9% uptime, p99 < 30s, hallucination < 2%
• [ ] Test alert firing: manually trigger each alert condition
• [ ] Set up Grafana backup (dashboard JSON export)
• [ ] Configure retention: 15s for 30 days, 1m for 90 days, 5m for 1 year
• [ ] Consider self-hosted AI TCO monitoring
• [ ] Monitor secure ingestion pipeline health
• [ ] Track zero-trust security metrics
• [ ] Set up synthetic monitoring: automated test queries every 5 minutes
• [ ] Document runbook for each alert: what to check, how to fix

FAQ

What metrics should you monitor for an AI platform?

Monitor four categories: (1) Infrastructure — GPU utilization, VRAM usage, CPU, memory, disk I/O, network latency. (2) LLM inference — tokens/second, time-to-first-token, generation latency (p50, p90, p99), queue length, error rate, model load time. (3) RAG pipeline — retrieval latency, recall@k, reranking latency, cache hit rate, context relevance score, hallucination rate (faithfulness). (4) Business — queries per user, cost per query, user satisfaction (thumbs up/down), daily active users. Use Prometheus for metrics collection and Grafana for dashboards and alerting.

How do you monitor LLM inference with Prometheus and Grafana?

Configure Ollama or vLLM to expose /metrics endpoint in Prometheus format. Scrape metrics with Prometheus (scrape_interval: 15s). Create Grafana dashboards with panels for: tokens/second, latency percentiles (p50, p90, p99), GPU memory usage, request queue length, error rate. Set alerts: p99 latency > 30s, GPU memory > 90%, error rate > 1%, queue > 50. Use Grafana alerting with Slack/PagerDuty notifications. For Ollama, the /metrics endpoint provides request duration, GPU memory, and model load time metrics.

What is AI observability vs traditional monitoring?

Traditional monitoring captures infrastructure metrics (CPU, memory, latency, errors). AI observability adds quality monitoring: hallucination rate (faithfulness score), answer relevance, retrieval precision, prompt drift, model behavior changes. Traditional APM catches 500 errors. AI observability catches when your LLM starts hallucinating more than usual, when retrieval quality degrades, or when prompt changes affect output quality. Use Langfuse, LangSmith, or MLflow for AI-specific observability (tracing, evaluation, drift detection) alongside Prometheus + Grafana for infrastructure monitoring.

How do you detect hallucinations in production?

Detect hallucinations with: (1) Faithfulness scoring — break response into atomic claims, verify each against retrieved context (use ragas or DeepEval). (2) Confidence threshold — if LLM logprobs are below threshold, flag for review. (3) Citation check — if response lacks citations or citations don't support claims, flag. (4) Human feedback — thumbs up/down from users, track negative feedback rate. (5) Grounding score — compare response embeddings to context embeddings (low similarity = potential hallucination). Alert when hallucination rate exceeds 2% of responses.

What alerts should an AI platform have?

Critical alerts: LLM p99 latency > 30s, error rate > 1%, GPU memory > 90%, Ollama/vLLM process down, database connection pool exhausted. Quality alerts: hallucination rate > 2%, cache hit rate < 20%, retrieval recall drops > 10%, user negative feedback > 5%. Capacity alerts: GPU queue > 50 requests, disk usage > 80% (model storage), daily cost exceeds budget. Compliance alerts: audit log write failure, unauthorized access attempt, PII detected in prompt. Route critical alerts to PagerDuty, quality alerts to Slack, capacity alerts to email.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →