Overview

Applies SRE rigor to AI systems to detect and respond to outages, quality regressions, unsafe outputs, and budget spikes. It defines AI incident classes (availability, quality, safety, cost), a severity framework, and golden signals to guide rapid containment and postmortems.

How This Skill Works

Uses a structured taxonomy and metrics to triage AI incidents: classify events as availability, quality, safety, or cost, and apply SEV levels SEV1–SEV3. It relies on golden signals like request success rate, latency, hallucination proxy, cost per minute, and guardrail violation rate, with predefined playbooks for containment, diagnosis, and recovery plus postmortem templates.

When to Use It

• LLM outages or provider unavailability causing user-visible failures
• Quality regression where answer accuracy or tool success falls below SLO
• Safety incidents with harmful or policy-violating outputs increase
• Cost spikes or runaway spending due to tokens, calls, or tools
• Budget explosions requiring quick containment and escalation

Quick Start

1. Step 1: Map AI incidents to availability, quality, safety, and cost; identify golden signals to monitor.
2. Step 2: Implement the severity framework SEV1 to SEV3 and predefine response playbooks for model outage, quality regression, and cost spike.
3. Step 3: Establish a postmortem template and run drills to validate detection, containment, and hardening tasks with owners and due dates.

Best Practices

• Define a consistent AI incident taxonomy: availability, quality, safety, cost, and assign owners
• Track golden signals: request success rate, latency, hallucination proxy, cost per minute/tenant, guardrail violation rate
• Enforce strict deployment controls during incidents: freeze deployments, shift traffic to fallback model/provider, apply stricter rate limits
• Document postmortems with timeline, blast radius, missed signals, alert tuning actions, and hardening tasks with owners and due dates
• Regularly exercise playbooks and maintain up to date runbooks and baselines

Example Use Cases

• Model Outage: provider becomes unavailable; freeze deployments, shift to fallback, communicate ETA
• Quality Regression: post update leads to lower answer accuracy; rollback prompt/model version, re-run baseline evaluation
• Safety Spike: guardrails fail and outputs become policy-violating; disable risky optimizations and review prompts
• Cost Spike: sudden token spend surge; identify top tenants, enable cache and cheaper fallbacks
• Guardrail Violation Spike: spike in unsafe outputs; rapid triage and containment plus remediation

Frequently Asked Questions