A team of AI researchers has identified and addressed a critical vulnerability in how large language models learn to reason through a technique called self-distillation. The problem, known as privileged information leakage, occurs when a model discovers shortcuts during training that won't be available when deployed in real-world scenarios, essentially allowing it to "cheat" rather than develop genuine reasoning capabilities.
Self-distillation has become a popular approach for training reasoning-focused language models. In this method, a single model serves dual roles: it acts as both teacher and student, with the teacher version having access to additional context or information. The student learns by mimicking the teacher's outputs. However, according to arXiv research from Yunhe Li, Hao Shi, Wenhao Liu, and colleagues, this setup creates unintended consequences.
The Core Problem with Current Methods
Existing self-distillation approaches require the student model to match the teacher's outputs at every token position. This dense, token-level supervision can backfire in several ways. Models become overfitted to patterns within their training domains, their exploratory capabilities diminish, and they struggle when confronted with out-of-distribution problems. Most critically, students absorb answer-dependent shortcuts that depend on privileged information, leaving them unable to function properly at test time.
The research team calls this phenomenon privileged information leakage, a fundamental limitation that undermines the entire premise of using self-distillation for reasoning tasks.
A Smarter Blending Strategy
To solve this, the researchers introduced DemoPSD, a framework built on the principle of selective teacher guidance adoption. Rather than forcing students to replicate the complete teacher distribution, DemoPSD aims for what mathematicians call a reverse-KL barycenter target: a weighted combination of both the teacher and student distributions.
The key innovation lies in how this weighting happens. Instead of applying uniform guidance across all tokens, DemoPSD measures the disagreement between teacher and student distributions and uses that disagreement to control how much the student should follow the teacher at each specific position. Where the models agree strongly, the student maintains independence. Where they diverge substantially, selective guidance kicks in.
This adaptive approach offers two provable benefits: it substantially reduces privileged information leakage while simultaneously preserving the student model's capacity for genuine exploration and reasoning.
Experimental Validation
The research team tested DemoPSD across multiple scientific domains using the SciKnowEval benchmark, which covers four different fields of scientific knowledge. The results showed consistent improvements over existing methods like GRPO and SDPO, while maintaining higher training entropy, a measure of the model's exploratory behavior. When tested on out-of-distribution problems using the GPQA benchmark, DemoPSD demonstrated robust generalization to scenarios unlike its training data.
These findings suggest that the path toward more capable reasoning models may require rethinking how models learn from each other. By preventing models from taking informatational shortcuts during training, researchers can build systems more likely to develop genuine reasoning abilities that transfer beyond their original domain.
