Skip to main content
Back to timeline
arXivSource publication:

Vision transformers quantize without original data: RankQ-ViT reaches 82.44% top-1 on ViT-B at 4-bit weights

Synopsis

Deploying vision transformers on edge devices usually requires original training data to calibrate quantization, and privacy constraints can make that data unavailable. RankQ-ViT instead generates calibration samples with a block-aware rank-promotion objective: rather than only maximizing patch-similarity entropy, it uses self-contrastive learning to spread patch features in different directions so that synthetic images approach the similarity-matrix rank of real images. On ImageNet, ViT-B under W4/A8 quantization reaches 82.44% top-1 accuracy, 6.15 points above RepQ-ViT, while data generation runs about 5 times faster than PSAQ-ViT v1.

Source-provided article image: Beyond Similarity Entropy: A Rank-Guided Data-Free Quantization Framework for Vision Transformers
arXiv

Interpretation

Data-free quantization can now generate calibration samples from a pretrained vision transformer alone, without original training images. The earlier PSAQ-ViT line generated calibration samples by maximizing patch-similarity entropy, which captures only how dispersed the similarity values are. On ImageNet with ViT-B under W4/A8 quantization, RankQ-ViT reaches 82.44% top-1 accuracy, 6.15 points above RepQ-ViT at 76.29% and 3.71 points above CLAMP-ViT at 78.73%.

The generation process pushes patch features in different directions, raising the rank of the synthetic images' similarity matrix. Computing and maximizing the rank of the similarity matrix directly would require matrix operations at every step, so a block-level self-contrastive loss raises the rank indirectly. Patches from the same region are pulled together and patches from different regions are pushed apart, with stronger rank-promotion weight on shallow blocks to match the drop in rank with depth seen in real images.

Generated samples are first filtered by rank and semantic consistency, then used to reconstruct the quantized model block by block. Using all generated samples without filtering can hurt learning of the quantized model because the generation process is unstable. In an ImageNet ablation with DeiT-S, combining JSD semantic consistency with rank filtering reaches 77.86% at W4/A8 and 79.59% at W8/A8, better than either criterion alone.

Perspective

The demonstrated scope is image classification, object detection, instance segmentation and point-cloud classification on ImageNet, COCO, ModelNet40 and ShapeNet, covering ViT, DeiT, Swin Transformer and Point Transformer; the authors propose future work on data-free quantization for large foundation models such as SAM.

On Swin-T at W4/A8, CLAMP-ViT reaches 80.28%, above RankQ-ViT's 79.24%; on ViT-B at W4/A4, RankQ-ViT reaches 17.77%, still far below the 84.53% full-precision figure; the authors attribute CLAMP-ViT's advantage on Swin-T to its cyclic adaptive data generation, which costs substantially longer runtime.