ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

ALTK-Evolve, a learned-memory system, enables a basic ReAct agent powered by GPT-4.1 to achieve 67.5% Pass^4 on τ²-bench retail tasks, outperforming all retail entries in a leaderboard snapshot dated October 12, 2026. The approach relies on distilling behavioral rules from the agent’s training attempts into concise guidelines, which are then appended to the system prompt for subsequent conversations. No model weights are updated, and no human annotation is required, as the same GPT-4.1 model extracts and applies the guidelines.

τ²-bench, developed by Sierra, evaluates customer-service agents under realistic constraints, pairing them with simulated customers who have specific goals and domain tools. Tasks require strict adherence to written policies and correct execution of often irreversible actions, with success measured by matching the expected final database state. The benchmark includes retail and airline domains, using a two-phase ReAct loop with GPT-4.1 as the agent and GPT-5.5 as the customer simulator.

Guideline effectiveness varies by domain: in retail, more guidelines yield higher accuracy and reliability, while in airline, a compact high-support core (support ≥ 3) delivers the best reliability despite fewer guidelines. Mean accuracy improves by 4.4 points in retail (78.1% to 82.5%) and 7.5 points in airline (57.5% to 65.0%), with Pass^4 reliability rising by 10 points in both domains (57.5% to 67.5% in retail; 35.0% to 45.0% in airline).

Guidelines increase input token costs but reduce conversation length and latency. For example, retail’s 292-guideline prompt raises cost from $0.084 to $0.139 per conversation, yet trims assistant turns by 0.3–0.4 in both domains. The approach also scales to frontier models like GPT-5.5, where re-mined guidelines improve airline task performance despite lower support thresholds, demonstrating broader applicability beyond weaker models.