Skip to main content
Search

# Knowledge Distillation

52 related articles

Subscribe to updates
  1. arXiv

    Pruned visual tokens keep the answer but shift the odds: DiFiT trains selection on the full model's response distribution

    Visual token pruning for multimodal large language models is usually judged by whether answers stay correct, yet pruning can change the probabilities of alternative responses even when greedy answers do not move. DiFiT learns which visual tokens to keep from both the full-token model's response distributions and reference answers, with one ranking serving several retention ratios. On Qwen2.5-VL-7B it cuts sampling-induced negative flips by 30.3% relative to FastV at 10% retention and preserves 98.74% of baseline performance on MME at 30% retention.

  2. arXiv

    Deletion requests stop touching most shards: IDSISA concentrates forget sets with influence scores, then distills from untouched shards to recover accuracy

    For machine unlearning under privacy rules, IDSISA adds influence-guided shard assignment and cross-shard distillation to SISA's shard-retraining framework. Influence scores are computed once before training and co-locate samples likely to be deleted together, so a deletion request touches fewer shards than under random partitioning; retrained shards also learn from unaffected shards' soft outputs to recover utility lost to isolation. On ResNet-18/CIFAR-10, BiLSTM/IMDB, TCN/IMDB and a fully connected Fashion-MNIST network, the deletion guarantee still comes from retraining itself, and distillation is only a separately quantified utility-recovery step.

  3. arXiv

    Blockwise distillation search breaks at a 55% latency budget: pairwise search leads by 3.3–16.6 points

    Blockwise distillation architecture search picks child networks by summing each replacement block's isolated score, an additive assumption rarely tested directly. On DeiT-S and DeiT-B, as the block-latency budget drops from 60% to 55%, the additive integer linear program begins to trail a pairwise integer quadratic program that models block interactions, by 3.3–16.6 top-1 points on ImageNet-100 after fine-tuning. The error splits into a metric part from the score's nonlinearity and a functional interaction part, dominant on DeiT-S and DeiT-B respectively; a hybrid search that ranks with single-block logits and measures only a few candidates needs no dedicated pair measurements and beats the pairwise search in 13 of 14 full ImageNet-1k settings.

  4. arXiv

    CallCleaner distills an audio foundation model into a 19M-parameter denoiser: field animal recordings run about 90 times faster than real time on a laptop CPU

    Field recordings of animal calls carry wind, rain, vehicles and other animals, while clean training data are scarce. CallCleaner distills the general-purpose audio source-separation foundation model SAM-Audio into an 18.9M-parameter waveform-to-waveform denoiser that needs no detector or prompt and runs about 90 times faster than real time on a laptop CPU. On the 47 clips of at most 30 seconds in the Biodenoising test benchmark, it leaves 11 dB less noise around the call than Biodenoising and is better on 44 of 47 clips.

  5. Personalized federated learning now tracks per-sample representation drift: 63.05% on CIFAR-100, 5.96 points above the strongest baseline

    Personalized federated learning must balance shared knowledge against client-specific adaptation, but non-IID data and partial client participation let shared representations drift across communication rounds. FedEAGD keeps a client-specific exponential moving average teacher, measures drift as the cosine discrepancy between each sample's current and historical shared representations, and uses it to set historical distillation strength per sample while a stage-wise schedule introduces personalized residuals. On CIFAR-100 with Dirichlet concentration α = 0.1 it reaches 63.05% accuracy, 5.96 percentage points above the strongest evaluated baseline.

  6. arXiv

    A 6U CubeSat trained a thermal segmentation model in orbit in about 13 minutes, reaching mIoU above 0.9, then used it to steer its attitude toward the Moon

    A 6U CubeSat trained and ran a thermal-infrared segmentation model onboard in low Earth orbit, then fed the model's output into the attitude determination and control system to point at the Moon in closed loop from a live thermal image stream. Training ran on the GPU of an NVIDIA Jetson TX2 NX for about 13 minutes across 30 plus 30 epochs and reached a final mIoU above 0.9; the two tracking experiments on January 26 and 30, 2026 brought mean pointing error down to 7.4 and 13.5 pixels, while earlier runs oscillated visibly.

  7. arXiv

    AutoCompress writes deployment limits into pruning search: up to 6.57 points higher average accuracy than Minitron++ across three GPUs

    LLM pruning usually targets parameter counts, but parameter count does not predict latency, throughput or memory, which depend on architecture, workload, hardware and serving engine. AutoCompress puts latency, throughput, parameter and memory constraints directly into differentiable architecture search, searching global dimensions and per-layer structure together and guided by cost models calibrated by measurement inside the target serving engine. Under the same deployment budgets and matched recovery training as Minitron++, it raises average accuracy across eight benchmarks by 6.57, 2.35 and 0.57 percentage points on RTX 5070, RTX 5080 and H100, using roughly one-half to one-third as many architecture-selection tokens.

  8. MongoDB | Blog

    MongoDB Research releases LEAF distillation framework: 23M-parameter embedding models keep up to 97% of teacher performance and run retrieval without a GPU

    MongoDB Research has released LEAF, a knowledge distillation framework, together with two 23M-parameter text embedding models, mdbr-leaf-ir and mdbr-leaf-mt. State-of-the-art embedding models have grown to hundreds of millions or billions of parameters, typically need GPU servers, and are not compatible with one another, so swapping models means recomputing every document embedding. LEAF aligns only the teacher and student output embeddings; the distilled models retain up to 97% of their teacher's performance while being 5–15× smaller and 7–24× faster, and can run offline on CPU-only servers, IoT and mobile devices.

  9. arXiv

    PRISM-Net splits product retrieval into line and color routes, lifting SwinV2 line mAP@10 by 5.74 points

    PRISM-Net splits a product image's holistic features into a line pathway and a color pathway, supervises the two routers zero-shot with frozen SigLIP text prompts, and fuses both with an image-adaptive meta-router. In 5-fold cross-validation over 275,454 e-commerce images in 18 categories and full-scale retrieval of 55,091 queries against a 220,363-image gallery, the SwinV2 backbone gains 5.74 points in line mAP@10, while the color-only configuration gains 9.12 points in color mAP@10; on ConvNeXt-V2 and DINOv3, product retrieval and attribute alignment trade off.

  10. PLaMo 3 Translate releases four weight sizes: the 0.8B model beats translation-specific models more than 30 times larger on Japanese-English and Japanese-multilingual pairs

    Preferred Networks has released weights for PLaMo 3 Translate, a translation-specific model, at 0.8B, 2B, 8B and 31B on Hugging Face, along with FP8, W4A16, NVFP4, GGUF and MLX formats for quantized and local inference. The smallest 0.8B model outperforms existing translation-specific models more than 30 times larger on Japanese-English and Japanese-to-multilingual translation, while the 2B model surpasses all existing translation-specific models on English-to-multilingual translation. Training uses two-stage supervised fine-tuning plus DPO, with smaller models distilled from the 31B teacher.