_{🧠 图解记忆：随机分流同类任务，固定版本与观测口径，用质量、延迟、成本和安全共同决策；点击图片可查看原图。}

**Agent A/B 测试架构：**

展开 Python 代码示例（76 行）

from scipy.stats import chi2_contingency
import random

class AgentABTest:
    def __init__(self, variant_a: callable, variant_b: callable):
        self.variant_a = variant_a
        self.variant_b = variant_b
        self.results = {"a": [], "b": []}
    
    def assign_variant(self, user_id: str) -> str:
        # 稳定的 hash 分桶，确保同一用户始终分到同一组
        bucket = hash(user_id) % 100
        return "a" if bucket < 50 else "b"
    
    async def run_test(self, test_queries: list, duration_hours: int = 24):
        """运行 A/B 测试"""
        for query in test_queries:
            variant = self.assign_variant(hash_user(query))
            
            start = time.time()
            result = await (self.variant_a if variant == "a" else self.variant_b)(query)
            duration = time.time() - start
            
            self.results[variant].append({
                "query": query,
                "result": result,
                "duration": duration,
                "success": self.evaluate_success(result),
                "token_cost": self.count_tokens(result)
            })
    
    def analyze(self) -> dict:
        """统计分析"""
        results_a = self.results["a"]
        results_b = self.results["b"]
        
        # 成功率检验
        success_a = sum(1 for r in results_a if r["success"])
        success_b = sum(1 for r in results_b if r["success"])
        
        _, p_value = chi2_contingency([
            [success_a, len(results_a) - success_a],
            [success_b, len(results_b) - success_b]
        ])[:2]
        
        return {
            "variant_a": {
                "n": len(results_a),
                "success_rate": success_a / len(results_a),
                "avg_duration": mean([r["duration"] for r in results_a]),
                "avg_cost": mean([r["token_cost"] for r in results_a])
            },
            "variant_b": {
                "n": len(results_b),
                "success_rate": success_b / len(results_b),
                "avg_duration": mean([r["duration"] for r in results_b]),
                "avg_cost": mean([r["token_cost"] for r in results_b])
            },
            "statistical_significance": {
                "p_value": p_value,
                "significant": p_value < 0.05,
                "confidence_level": "95%"
            }
        }

# 使用示例
ab_test = AgentABTest(
    variant_a=lambda q: agent_v1.run(q),  # 旧版本
    variant_b=lambda q: agent_v2.run(q)  # 新版本
)
await ab_test.run_test(test_queries, duration_hours=24)
analysis = ab_test.analyze()

# 如果 B 版本显著更好，则推广
if analysis["statistical_significance"]["significant"]:
    rollout_to_production("variant_b")

📚 参考：LangSmith：Experiments（在线实验/AB 评估）

Agent A/B 测试评估指标：

指标	测量方式	最小样本量
任务成功率	人工标注或 LLM-as-Judge	~1000/组
用户满意度	CSAT 评分（1-5）	~200/组
平均任务时长	埋点计时	~500/组
Token 消耗	API 日志	~500/组
工具调用次数	Agent 日志	~500/组

面试话术：

"Agent A/B 测试先定义任务级主指标、安全护栏、延迟和成本，再按用户稳定分桶，避免同一用户跨版本污染。检验方法取决于指标分布和实验设计，显著性也不等于业务价值；还要预先设定最小可检测效果、样本量、观察窗口和停止规则。"

---