五度易链

菜单

lch
发布于 2026-05-12 / 0 阅读
0

Understanding LLM Distillation Techniques

#转载

Modern large language models are no longer trained only on raw internet text. Increasingly, companies are using powerful “teacher” models to help train smaller or more efficient “student” models. This process, broadly known as LLM distillation or model-to-model training , has become a key technique for building high-performing models at lower computational cost. Meta used its massive Llama 4 Behemoth model to help train Llama 4 Scout and Maverick, while Google leveraged Gemini models during the development of Gemma 2 and Gemma 3. Similarly, DeepSeek distilled reasoning capabilities from DeepSeek-R1 into smaller Qwen and Llama-based models.

The core idea is simple: instead of learning solely from human-written text, a student model can also learn from the outputs, probabilities, reasoning traces, or behaviors of another LLM. This allows smaller models to inherit capabilities such as reasoning, instruction following, and structured generation from much larger systems. Distillation can happen during pre-training, where teacher and student models are trained together, or during post-training, where a fully trained teacher transfers knowledge to a separate student model.

In this article, we will explore three major approaches used for training one LLM using another: Soft-label distillation , where the student learns from the teacher’s probability distributions; Hard-label distillation , where the student imitates the teacher’s generated outputs; and Co-distillation , where multiple models learn collaboratively by sharing predictions and behaviors during training.

Soft-Label Distillation

Soft-label distillation is a training technique where a smaller student LLM learns by imitating the output probability distribution of a larger teacher LLM . Instead of training only on the correct next token, the student is trained to match the teacher’s softmax probabilities across the entire vocabulary. For example, if the teacher predicts the next token with probabilities like “cat” = 70% , “dog” = 20% , and “animal” = 10% , the student learns not just the final answer, but also the relationships and uncertainty between different tokens. This richer signal is often called the teacher’s “dark knowledge” because it contains hidden information about reasoning patterns and semantic understanding.

The biggest advantage of soft-label distillation is that it allows smaller models to inherit capabilities from much larger models while remaining faster and cheaper to deploy. Since the student learns from the teacher’s full probability distribution, training becomes more stable and informative compared to learning from hard one-word targets alone. However, this method also comes with practical challenges. To generate soft labels, you need access to the teacher model’s logits or weights, which is often not possible with closed-source models. In addition, storing probability distributions for every token across vocabularies containing 100k+ tokens becomes extremely memory-intensive at LLM scale, making pure soft-label distillation expensive for trillion-token datasets.

Hard-label distillation

Hard-label distillation is a simpler approach where the student LLM learns only from the teacher model’s final predicted output token instead of its full probability distribution. In this setup, a pre-trained teacher model generates the most likely next token or response, and the student model is trained using standard supervised learning to reproduce that output. The teacher essentially acts as a high-quality annotator that creates synthetic training data for the student. DeepSeek used this approach to distill reasoning capabilities from DeepSeek-R1 into smaller Qwen and Llama 3.1 models.

Unlike soft-label distillation, the student does not see the teacher’s internal confidence scores or token relationships — it only learns the final answer. This makes hard-label distillation computationally much cheaper and easier to implement since there is no need to store massive probability distributions for every token. It is also especially useful when working with proprietary “black-box” models like GPT-4 APIs, where developers only have access to generated text and not the underlying logits. While hard labels contain less information than soft labels, they remain highly effective for instruction tuning, reasoning datasets, synthetic data generation, and domain-specific fine-tuning tasks.

Co-distillation

Co-distillation is a training approach where both the teacher and student models are trained together instead of using a fixed pre-trained teacher. In this setup, the teacher LLM and student LLM process the same training data simultaneously and generate their own softmax probability distributions. The teacher is trained normally using the ground-truth hard labels, while the student learns by matching the teacher’s soft labels along with the actual correct answers. Meta used a form of this approach while training Llama 4 Scout and Maverick alongside the larger Llama 4 Behemoth model.

One challenge with co-distillation is that the teacher model is not fully trained during the early stages, meaning its predictions may initially be noisy or inaccurate. To overcome this, the student is usually trained using a combination of soft-label distillation loss and standard hard-label cross-entropy loss. This creates a more stable learning signal while still allowing knowledge transfer between models. Unlike traditional one-way distillation, co-distillation allows both models to improve together during training, often leading to better performance, stronger reasoning transfer, and smaller performance gaps between the teacher and student models.

Comparing the Three Distillation Techniques

Soft-label distillation transfers the richest form of knowledge because the student learns from the teacher’s full probability distribution instead of only the final answer. This helps smaller models capture reasoning patterns, uncertainty, and relationships between tokens, often leading to stronger overall performance. However, it is computationally expensive, requires access to the teacher’s logits or weights, and becomes difficult to scale because storing probability distributions for massive vocabularies consumes enormous memory.

Hard-label distillation is simpler and more practical. The student only learns from the teacher’s final generated outputs, making it much cheaper and easier to implement. It works especially well with proprietary black-box models like GPT-4 APIs where internal probabilities are unavailable. While this approach loses some of the deeper “dark knowledge” present in soft labels, it remains highly effective for instruction tuning, synthetic data generation, and task-specific fine-tuning.

Co-distillation takes a collaborative approach where teacher and student models learn together during training. The teacher improves while simultaneously guiding the student, allowing both models to benefit from shared learning signals. This can reduce the performance gap seen in traditional one-way distillation methods, but it also makes training more complex since the teacher’s predictions are initially unstable. In practice, soft-label distillation is preferred for maximum knowledge transfer, hard-label distillation for scalability and practicality, and co-distillation for large-scale joint training setups.

Arham Islam
+ posts

I am a Civil Engineering Graduate (2022) from Jamia Millia Islamia, New Delhi, and I have a keen interest in Data Science, especially Neural Networks and their application in various areas.

机器人ETF万家（560630）5...
AI voice startup ...