Skip to main content
Back to timeline
arXivSource publication:

Few-step diffusion students lose directions, not variance: ReDiverse restores diversity at one intermediate block

Synopsis

Distilling a diffusion model into a four-step student makes different random seeds produce nearly identical images and videos for the same prompt. By comparing teacher and student activations block by block, ReDiverse finds that the student keeps its total activation variance but concentrates it into far fewer directions, at one intermediate block during the first denoising step. ReDiverse-FT then uses LoRA at that block to regress the student's first-step activations onto the teacher's, while ReDiverse-TF perturbs activations along teacher principal directions; across four teacher-student pairs on SDXL, SD3, Wan 2.1 and MiniMax-H3, both restore diversity while largely preserving generation quality.

Source-provided article image: Restoring Diversity in Distilled Image and Video Generators by Probing Activations
arXiv

Interpretation

What few-step distillation loses is the directional spread of activation variance, not its total amount: changing the random seed under one prompt barely changes the student's output, yet its internal total variance matches or exceeds the teacher's. Earlier explanations of diversity collapse pointed to distillation objectives, adversarial training or step budgets, and remedies either re-distilled the student or intervened at inference by optimizing noise or perturbing features, without locating where inside the network the variation disappears. Measured block by block on four teacher-student pairs spanning SDXL, SD3, Wan 2.1 and MiniMax-H3, students retain 1.03 to 2.92 times the teacher's total variance while keeping only 0.37 to 0.60 of its participation ratio, the effective number of directions the variance spans.

The repair only needs one intermediate block at the initial denoising step: putting the teacher's principal directions back into the student's activations lets downstream layers decode different layouts and poses. Steering the teacher along its own leading direction changes camera framing, subject pose and scene composition, whereas steering the student along its own leading direction changes only appearance-level details such as coat pattern and localized lighting, leaving global geometry fixed. Activation steering at the collapsed block injects the teacher's top principal direction into the student and recovers diverse poses and layouts without modifying any student weights; two students of the same teacher, FastVideo and Turbo, keep 61% and 28% of the teacher's participation ratio at that block, matching DINO diversity of 52.3 versus 35.8 against the teacher's 55.3.

ReDiverse-FT trains LoRA only in the layers up to and including the collapsed block, regressing the student's first-step activation onto the teacher's activation at the corresponding block, and is active only at the first step at inference; ReDiverse-TF needs no training and injects, at the first step, perturbations along teacher principal directions precomputed per prompt and scaled by the teacher's spread along each. The two interventions can be used independently or combined, and both act on publicly released four-step student checkpoints without re-distillation. On 553 GenEval image prompts with 100 seeds each and 300 VBench video prompts with 10 clips each, all methods share one noise bank for paired comparison; on SDXL the student's DINO is 59.4, ReDiverse reaches 65.0 and the teacher 66.3, while on Wan 2.1 the student is 31.6, ReDiverse 52.2 and the teacher 51.6.

Perspective

The findings rest on four teacher-student pairs: SDXL with DMD2, SD3-M with TDM, Wan 2.1 with lightx2v and MiniMax-H3 with Turbo, all publicly released four-step checkpoints that were not re-distilled; evaluation uses 553 GenEval image prompts and 300 VBench video prompts, with all methods sharing one noise bank for paired comparison.

Quality still drops at times, most visibly in reward-model scores on images and in video realism against the teacher, because ReDiverse corrects the student after distillation rather than during it; both variants also assume the student shares the teacher's architecture, so compressed students such as BK-SDM would need a learned projection between matched blocks.