Reinforcement learning has emerged as a powerful method for enhancing the reasoning capabilities of large language models, but the approach carries a significant practical cost. Training state-of-the-art models with verifiable rewards requires generating thousands of rollouts, making each training cycle computationally expensive and time-intensive. As models grow larger, this bottleneck increasingly limits how frequently companies can improve their systems.
Researchers at multiple institutions have proposed a solution that sidesteps this bottleneck entirely. According to arXiv, a team including researchers from leading tech organizations has developed Direct On-Policy Distillation (Direct-OPD), a method that transfers knowledge gained from reinforcement learning on smaller models to improve larger target models without repeating the expensive training process.
The core innovation addresses a fundamental limitation of traditional knowledge distillation. When a teacher model completes reinforcement learning, its final policy reflects both the valuable improvements gained from training and the inherent limitations of the smaller model itself. Simply copying this final policy to a larger model therefore transfers both useful and harmful characteristics.
Direct-OPD takes a different approach. Instead of distilling the teacher's final policy directly, the method analyzes the shift between the model before and after reinforcement learning. It calculates the log-ratio between these two versions and treats this difference as an implicit reward signal. This captures precisely which actions the reinforcement learning process made more or less likely, without needing the teacher model's size constraints.
The team applied this signal to a stronger student model training on its own data distribution. Rather than mimicking the smaller model's behavior or building an explicit reward model, Direct-OPD reuses the raw supervision signal from the original reinforcement learning run. The approach requires no additional sparse-reward training on the target model and no new reward model development.
Real-World Performance Gains
The practical benefits prove substantial. In testing, the method improved Qwen3-1.7B performance on the AIME 2024 benchmark from 48.3% to 62.4% accuracy in just four hours using eight A100 GPUs. This represents significant progress on a challenging mathematical reasoning task, achieved through knowledge transfer rather than direct training.
The technique also outperforms direct reinforcement learning on the larger model when both processes use the same computational budget. Additionally, Direct-OPD enables sequential composition, allowing researchers to stack multiple policy shifts from different weak teachers, further amplifying improvements.
Implications for AI Development
The findings demonstrate that reinforcement learning outcomes can be repurposed across different model scales in ways that transcend simple imitation. Rather than treating smaller models merely as reference implementations, the research shows that the learning signals they generate carry transferable value independent of their original architecture.
This approach could reshape how companies approach post-training at scale. Organizations could run expensive reinforcement learning experiments on smaller, more efficient models, then efficiently propagate those gains to production systems. The method potentially reduces the computational overhead of improving frontier models, making regular enhancement cycles more feasible.
