$ cat wiki/papers/2026/2607.08964-long-horizon-terminal-bench.md
Long-Horizon-Terminal-Bench (LHTB)
TL;DR
A 46-task containerized terminal benchmark with dense reward grading; current best model achieves only 15.2% pass@1 — revealing a severe long-horizon capability gap in LLM agents despite consuming 9.9M tokens per task on average.
Authors & Org
- GitHub: zli12321 (primary)
- Affiliation: not yet confirmed in public sources (academic)
- Published: July 9, 2026 (arXiv: 2607.08964)
- HuggingFace Daily: July 14, 2026 (#1 paper, 25 stars)
Method
Benchmark design: 46 tasks in a stateful containerized terminal environment. Unlike short-horizon benchmarks where an agent writes one artifact and stops, LHTB drops the agent into a persistent, stateful environment and requires it to sustain useful work over hundreds of steps.
Grading: Dense reward-based scoring with hidden, rebuild-from-artifact verifiers. Self-reported progress does not count — the verifier independently recreates the artifact from scratch. Two thresholds:
- 0.95 (partial credit — most credit awarded)
- 1.0 (perfect)
Task categories:
- Interactive games & puzzles
- Multimodal analysis
- Software / reverse engineering
- Scientific computing
- Earth & energy systems
- Security & performance
- Research reproduction
- Professional APEX-style workflows
Results
| Metric | Value |
|---|---|
| Best model pass@1 (0.95 threshold) | 15.2% |
| Best model pass@1 (1.0 threshold) | 10.9% |
| Mean across all models (0.95) | 4.3% |
| Mean across all models (1.0) | 1.7% |
| Average tokens per task | 9.9 million |
| Average episodes per run | 231 |
| Average execution time per task | 85.3 minutes |
Significance
- Closes the measurement gap between "can it write a function" and "can it finish a real multi-hour task" for terminal agents
- The 9.9M token average means agents are generating a lot of text while succeeding very rarely — scale alone does not solve long-horizon coherence
- Dense reward (vs. binary pass/fail) prevents agents from exploiting partial credit without genuine progress
- Directly relevant to understanding the distance between current agents and production-grade agentic systems
- The 15% ceiling strongly motivates continued work on Agentic Reinforcement Learning and hierarchical planning
Open Questions
- What breaks at the 100+ step horizon? (planning coherence, context exhaustion, error recovery, goal drift?)
- How much of the failure is context-length vs. reasoning capability vs. tool use reliability?
- Do existing agentic RL training methods (like Self-Distilled Agentic RL, ASPIRE) change the LHTB ceiling?
Cite
arXiv:2607.08964. "Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading." July 2026. https://arxiv.org/abs/2607.08964
Related
- Agents (LLM Agents) — primary domain
- Agentic Reinforcement Learning — training approaches to improve long-horizon performance
- Self-Distilled Agentic Reinforcement Learning — related training method
- Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent — agent scaling (horizon-scaling)