Observability and SRE Practices for Self-Hosted Large Language Models

You built a self-hosted Large Language Model (LLM). It runs on your hardware, respects your data privacy, and saves you from cloud API bills. Then the first user complains that responses are slow. Or worse, they time out entirely. You check your server. CPU is low. Memory looks fine. But where is the bottleneck? Is it the GPU? The network? The model itself?

This is the reality of running AI in production today. Traditional monitoring tools were built for web servers and databases, not for stochastic systems that generate text token by token. If you treat an LLM like a standard microservice, you will miss critical failures until users start leaving. This guide breaks down how to apply Site Reliability Engineering (SRE) principles specifically to self-hosted models, using real-world metrics and tooling that actually work.

Why Traditional Monitoring Fails for LLMs

Standard health checks ask: "Is the service up?" For an LLM, "up" is a dangerous oversimplification. A model can be technically alive-accepting connections-but functionally broken. It might be hallucinating wildly, returning empty strings, or generating tokens so slowly that the request times out before completion.

Recent experiments by ClickHouse in 2026 tested whether AI could replace human SREs in diagnosing these issues. They fed production observability data into advanced models like GPT-5, asking them to identify root causes of performance anomalies. The result? Autonomous Root Cause Analysis (RCA) failed. The models couldn't consistently outperform experienced engineers. Why? Because LLMs lack the contextual intuition of *why* a specific latency spike matters in a specific business context. They see numbers; humans see patterns.

This doesn't mean AI isn't useful. It means you need to build observability stacks that give humans fast, searchable data, while using AI as an assistant-not the driver. Your goal isn't full automation yet. It's visibility.

The Core Metrics That Matter

If you are serving models via frameworks like vLLM, a high-throughput and memory-efficient serving engine for large language models, you have access to specific Prometheus metrics that reveal what’s happening inside the black box. Ignoring these is like flying blind.

Here are the four non-negotiable metrics you must track:

• vllm_num_requests_running: How many requests are actively being processed right now. If this hits your concurrency limit, new requests queue up.
• vllm_num_requests_waiting: The length of the queue. A growing queue with stable running requests means you are under-provisioned or experiencing head-of-line blocking.
• vllm_gpu_cache_usage_perc: This tracks KV cache utilization. High usage here often precedes out-of-memory errors or severe latency spikes because the model struggles to manage its context window efficiently.
• vllm_avg_generation_throughput_toks_per_s: Tokens per second. This is your primary performance indicator. If this drops, users feel it immediately.

Don't just graph these. Set alerts based on trends. A sudden drop in throughput combined with rising GPU cache usage usually indicates a specific type of request (like very long contexts) is choking the system.

Building the Observability Stack

You need more than just infrastructure metrics. You need application-level insights. This is where LLMOps diverges from traditional MLOps. In classic ML, you care about model accuracy during training. In LLM production, you care about response quality and latency during inference.

Start with Prometheus and an open-source systems monitoring and alerting toolkit. Configure a Kubernetes ServiceMonitor to scrape your vLLM endpoints automatically. This gives you the raw numbers.

But numbers don't tell you *what* the model said. For that, you need tracing and logging. Tools like OpenTelemetry, a set of APIs, SDKs, and tools for collecting telemetry data allow you to instrument your application code. You want to capture the prompt, the response, the total duration, and any errors. Solutions like OneUptime or Openlit specialize in this, providing dashboards that correlate technical metrics with actual user interactions.

Consider this scenario: Users report "bad answers." Without logs, you can't know if the model was confused by a complex prompt, if a retrieval step failed in a RAG pipeline, or if the temperature setting was too high. With proper instrumentation, you can filter traces by latency outliers and inspect the exact input/output pairs that caused dissatisfaction.

SRE Practices for AI Infrastructure

Site Reliability Engineering isn't just about keeping things up; it's about managing trade-offs between velocity and stability. For self-hosted LLMs, this involves three key areas: resource management, failure recovery, and change management.

Resource Management and Autoscaling

Traditional Horizontal Pod Autoscalers (HPA) scale based on CPU or memory. GPUs don't behave like CPUs. A GPU can be at 100% utilization but still handle more concurrent requests if the batch size is optimized. Conversely, low utilization might mask a bottleneck in the PCIe bus or network I/O.

Newer AI-native Kubernetes tools are beginning to address this. Features like "Smart Sizing" use machine learning to analyze historical traffic patterns and recommend optimal CPU/memory requests. While fully autonomous scaling is still emerging, you can manually tune your autoscaler thresholds based on the vllm_num_requests_waiting metric rather than generic CPU stats. If the queue grows, scale up. Simple, effective, and accurate.

Failure Recovery

LLMs fail differently than web apps. A common failure mode is the "crash loop" caused by Out-of-Memory (OOM) errors when a single massive context window exceeds available VRAM. Standard Kubernetes restart policies will bring the pod back up, but if the same heavy request comes in, it crashes again.

Implement circuit breakers. If a specific endpoint or model version starts failing repeatedly, route traffic to a fallback model or a cached response. Use liveness probes that check for actual inference capability, not just HTTP 200 OK status. If the model hangs, the probe should fail, triggering a restart or eviction.

Change Management

Updating an LLM is risky. Unlike deploying a new feature flag, swapping a model version changes the fundamental behavior of every interaction. You cannot rely on unit tests alone. Implement canary deployments for models. Route 5% of traffic to the new model version. Monitor error rates and latency. More importantly, monitor qualitative feedback if possible. Did user satisfaction drop? Did token costs increase unexpectedly?

The Role of AI in Operations

We established that AI can't fully replace SREs yet. But it can make them faster. The CNCF highlights three areas where LLMs assist operations today:

1. Log Summarization: Feed thousands of lines of crash logs into an LLM. Ask it to summarize the error pattern. This saves hours of manual grep-ing.
2. Anomaly Correlation: When an alert fires, an LLM can scan recent deployment events, config changes, and related service metrics to suggest probable causes.
3. Drafting Post-Mortems: After an incident, use an LLM to draft the initial timeline and impact analysis based on your observability data.

Treat these AI assistants as junior engineers. They provide drafts and suggestions, but you verify everything. Do not let an LLM auto-restart your production cluster without human approval unless you have extremely robust guardrails.

Common Pitfalls to Avoid

Many teams stumble when moving from prototype to production. Here are the most frequent mistakes:

• Ignoring Token Costs: Self-hosting shifts cost from API fees to electricity and hardware depreciation. Track token throughput per dollar spent. Are you getting enough value from your GPU hours?
• Over-Provisioning Context Windows: Allocating maximum context length for all requests wastes memory. Implement dynamic batching and context truncation strategies.
• Lack of Quality Monitoring: You monitor uptime, but do you monitor accuracy? Implement automated evaluation pipelines that periodically test known prompts against expected outputs to catch model degradation.
• Neglecting Network Latency: In distributed setups, moving weights or activations between nodes adds overhead. Profile your network interconnects (NVLink, InfiniBand) alongside compute metrics.

Future Outlook: AI-Native Kubernetes

The industry is moving toward tighter integration between LLMs and infrastructure control loops. By late 2026, we expect broader adoption of features like Pod Recovery AI, which analyzes failure events and suggests specific remediation steps. However, until these tools prove themselves in diverse production environments, keep humans in the loop.

Your strategy today should focus on building a solid foundation of data. Clean, structured logs and comprehensive metrics are the fuel for future automation. If your data is messy, no amount of AI magic will fix your reliability problems.