_{🧠 图解记忆：SLO 从用户任务定义成功、可用性和延迟，告警再围绕预算消耗分层触发；点击图片可查看原图。}

**SLA 设计：**

展开 Yaml 代码示例（67 行）

# prometheus_alerts.yml
groups:
  - name: agent_sla
    rules:
      # SLA 1: 任务成功率 >= 95%
      - alert: AgentTaskSuccessRateLow
        expr: |
          (
            sum(rate(agent_tasks_total{status="success"}[5m]))
            /
            sum(rate(agent_tasks_total[5m]))
          ) < 0.95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Agent 任务成功率低于 SLA (95%)"
          description: "当前成功率: {{ $value | humanizePercentage }}"
      
      # SLA 2: P99 延迟 <= 30s
      - alert: AgentLatencyHigh
        expr: |
          histogram_quantile(0.99, 
            sum(rate(agent_task_duration_seconds_bucket[5m])) 
            by (le)
          ) > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Agent P99 延迟超过 30s"
      
      # SLA 3: Token 成本日增幅 > 20%
      - alert: AgentCostSpike
        expr: |
          (
            sum(increase(agent_tokens_total[24h]))
            /
            sum(increase(agent_tokens_total[24h] offset 7d))
          ) > 1.2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Agent Token 消耗日环比增长超过 20%"
      
      # 工具调用失败率 > 5%
      - alert: ToolCallFailureRateHigh
        expr: |
          (
            sum(rate(tool_calls_total{status="failure"}[5m]))
            /
            sum(rate(tool_calls_total[5m]))
          ) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "工具调用失败率 {{ $value | humanizePercentage }}，检查工具可用性"
      
      # 循环检测
      - alert: AgentLoopingDetected
        expr: increase(agent_loops_detected_total[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "检测到 Agent 循环，任务已自动熔断"

📚 参考：LangSmith：Monitoring 与告警

SLO 设计文档：

SLA 指标	目标值	告警阈值	测量方式
任务成功率	95%	< 93% PagerDuty	Prometheus counter
P50 延迟	< 5s	> 8s 告警	histogram
P99 延迟	< 30s	> 45s PagerDuty	histogram
TTFT	< 1s	> 3s 告警	histogram
日 Token 消耗	基线 * 1.2	> 1.5x 升级	counter delta
工具可用性	99.5%	< 99% 告警	availability check

面试话术：

"SLA/SLO 要从用户任务定义成功、可用性和延迟，而不是复制固定阈值。再根据错误预算设计多窗口燃尽率告警和严重级别，避免瞬时噪声叫醒值班人员。每条告警应关联负责人、影响范围、止血步骤、回滚和复盘入口。"

---