Long-horizon agent state and skill-file research

A paper identifies a systematic evaluation flaw in skill-enabled agents where overall gains can mask degraded performance on retrieval tasks; RAE proposed as a fix.

Data shows Google at 3.2 quadrillion tokens/month, agents 71% of OpenRouter traffic, Goldman projects 24x growth by 2030

Microsoft paper reported showing skill distillation from 35-50 trajectories recovers 55%-100%+ of reasoning model performance gap at 2.9-4.5x lower token cost

Harness-of-Harness scores 71.52 vs 58.24 for naive continuation on GameCraft-Bench

Google and Purdue's SKILL.state, published September 3, replaced growing agent transcripts with a structured state, cutting tokens 94% on a 100-step task while holding accuracy at 0.94 versus a baseline's 0.91.

ByteDance's Chain-of-Experience, published September 4, offered a direct contrast: keeping full attempt history in context raised accuracy from 66.8% to 79.3% across six benchmarks. Amazon and Microsoft's SPACE added a third angle, training agents to batch safe sequential actions, raising ScienceWorld success from 35.9% to 67.2% while cutting LLM rounds from 10.2 to 5.2.

ContextPilot paper described, using RL-trained context management

Three benchmarks published September 2 document frontier model struggles on extended tasks: 17 models averaged a 6.4% pass rate on multi-step terminal tasks, 10 of 17 solving zero; the best agent scored 51.3/100 after 12 hours; and the top setup on year-long decisions reached 27.3% of human earnings.

Investor Chamath Palihapitiya, at Stanford's AI Club, warned AI investment could reach a trough of disillusionment if these limitations are not addressed. A separate paper found agent traces compress into finite-state machines with 7 to 43 states.

A head-to-head experiment published September 1 comparing OpenClaw 2.0 and Hermes agents on a video generation task found that self-verification depth, not model capability, drove token consumption: OpenClaw used approximately 2.1M tokens with 10 self-fixes, Hermes approximately 2.9M with 20, the extra roughly 800K tokens spent on Hermes's render-checking harness measuring frame timing, seam visibility, and spec compliance.

The experiment frames a portion of agent token use as QA allocation, raising the question of how much compute agents should spend proving their work before returning results.

Multiple papers on how agents should manage state in long tasks, reported August 30-31, reached two conclusions: structured state cuts token use and raises accuracy, while skill file injection consistently lowers performance.

Google and Purdue's SKILL.state reduced tokens 94% on a 100-step warehouse task; Meta's EvoHarness-RL raised ALFWorld accuracy from 47.9% to 96.9%; Nvidia's ACES found skill document scores correlate near-zero with runtime gains; and WebDev-Skills-Bench found skill injection lowered pass rates across all tested models.

Paper testing eight leading models on year-long decision tasks finds best setup achieves 27.3% of human earnings

WebDev-Skills-Bench finds skill file injection lowers pass rates and raises token costs across all four tested models

Nvidia ACES finds skill document scores correlate near-zero with measured runtime improvement

Paper argues on-policy distillation outperforms GRPO for long-horizon agent training