The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

TL;DR

LLM RL implementations use separate inference and training engines for efficiency, creating a systematic training-inference probability mismatch (the "mirage"). The paper proposes MIPU, a two-step framework targeting the actual inference policy rather than the training proxy.

Authors & Org

• Jing Liang, Hongyao Tang, Yi Ma
• Affiliations: not confirmed as Tier-1 org (independent or academic)

Method

• Problem: Separate inference/training engines in LLM RL produce inconsistent probabilities for the same trajectories even after parameter sync — a structural training-inference mismatch that standard RL objectives silently optimize the wrong objective
• MIPU (Monotonic Inference Policy Update):
  1. Constructs sampler-referenced candidate updates
  2. Selectively accepts synchronized candidates via inference-side gap proxy
• MIPI (Monotonic Inference Policy Improvement): the resulting policy objective — designed to improve monotonically on the inference side, not the training proxy

Results

Tested on two model scales under high-mismatch conditions:

• Improved average reasoning performance vs. standard RL objectives
• Improved training stability under high-mismatch conditions

Significance

If the training-inference gap is as systematic as claimed, this affects all major RLHF/RLVR training pipelines — which run separate inference and training engines for throughput. The implication: GPT, Claude, and Gemini reasoning modes may be optimizing a proxy objective rather than the true inference-time behavior. MIPU is a drop-in correction, not a new architecture.

Open Questions

• How large is the mismatch in practice at frontier scale (GPT-5.6, Fable 5, Gemini 3.5 Pro)?
• Does the improvement hold at full frontier model scale (100B+)?
• Is this already addressed silently in closed-lab training stacks?

Cite

Liang, J., Tang, H., & Ma, Y. (2026). The Mirage of Optimizing Training Policies:
Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning.
arXiv:2606.29526.

Related

• Reasoning Models — affected training paradigm
• Agentic Reinforcement Learning — RL training pipeline context
• Test-Time Compute (Inference-Time Compute Scaling) — inference-time behavior being optimized