Weak-to-Strong Generalization via Direct On-Policy Distillation

TL;DR

Run RL on a cheap small model; transfer only the RL-induced policy delta (not the full policy) to a large model. Cuts the cost of bringing RL-style reasoning improvements to frontier-scale models.

Authors & Org

Shiyuan Feng, Huan-ang Gao, Haohan Chi, et al. (10 authors total) Tsinghua AIR (AI Institute) + ByteDance Seed

Method

Problem: RLVR (RL with verifiable rewards) improves LLM reasoning quality but requires many rollouts from the target model — very expensive for large models.

Observation: The "improvement" learned by RL is separable from the baseline policy. You can run RL cheaply on a weak (small) model, then extract and transplant just the improvement to a strong (large) model.

Direct-OPD (Direct On-Policy Distillation):

• Teacher runs RL training; extract the RL-induced delta: Δ = log π_T − log π_{T,ref}
• Distill Δ into the strong model using on-policy rollouts from the strong model (not the teacher's rollouts)
• On-policy component ensures the signal is grounded in the target model's own generation distribution

Key contrast with standard knowledge distillation:

• Standard KD: distill log π_T — copies the teacher's full reasoning style, including non-RL artifacts
• Direct-OPD: distill Δ — copies only what RL changed, not the teacher's baseline behavior

Results

(Per paper — specific benchmark numbers not in captured source; see arXiv for full tables)

• Outperforms standard KD on math/reasoning benchmarks when transferring from smaller RL-trained teacher to larger student
• On-policy component critical: removing it (using teacher rollouts instead) degrades performance

Significance

• Practical cost reduction for deploying RL improvements at frontier scale
• Formalizes the intuition that RL training has two separable components: (1) base policy, (2) RL-induced shift — and only (2) needs to be transferred
• Adjacent to Anthropic's AAR work (which runs RL on Claude Opus-class models — expensive); Direct-OPD provides a potential cost lever

Open Questions

• How much of the RL improvement is lost through the distillation step vs. native RL on the strong model?
• Does Direct-OPD generalize to non-verifiable reward settings (e.g., instruction following, not just math)?
• Works with RLVR (verifiable rewards) — does it extend to RLAIF or RLHF?

Cite

@article{feng2026directopd,
  title={Weak-to-Strong Generalization via Direct On-Policy Distillation},
  author={Feng, Shiyuan and Gao, Huan-ang and Chi, Haohan and Wu, Hanlin and Zhang, Zhilong and Jiang, Zheng and He, Bingxiang and Ma, Wei-Ying and Zhang, Ya-Qin and Zhou, Hao},
  year={2026},
  url={https://arxiv.org/abs/2607.05394}
}

Related

• Agentic Reinforcement Learning — RL for agents; this paper is in the RLVR family
• Self-Distilled Agentic Reinforcement Learning — Self-Distilled Agentic RL (related distillation approach)
• Automated Weak-to-Strong Researcher (AAR) — Anthropic's AAR uses RL on frontier-class models; Direct-OPD is a potential cost lever for this workload