Skip to main content
Back to timeline
arXivSource publication:

ElasticDino upsamples DINO features with elastic deformation: sharper segmentation and depth at 0.71 GB peak memory

Synopsis

Self-attention in vision transformers has quadratic cost, so feature maps stay at patch resolution and carry positional-encoding artifacts and boundary bleeding. ElasticDino is a lightweight adapter that exploits the observation that these artifacts are consistent across channels, using image-guided elastic deformation fields to correct all channels together during progressive upsampling and output pixel-aligned high-resolution features. On DINOv2 features it leads on several semantic segmentation, open-vocabulary segmentation and depth estimation benchmarks, at 108.41 ms average time and 0.71 GB peak memory, below the compared methods.

Source-provided article image: ElasticDino: Dense Semantic Features Using Elastic Deformations
arXiv

Interpretation

When raising DINO features to high resolution, ElasticDino no longer edits feature values directly; it resamples features through deformation fields aligned to the input image structure, preserving sharp semantic edges while removing positional-encoding artifacts and boundary bleeding. Earlier task-agnostic upsamplers relied on attention layers or reconciled high- and low-resolution features through cropping-like mechanisms, letting semantics bleed at boundaries; task-specific dense heads instead needed large data and per-task training. On DINOv2 features, ElasticDino leads segmentation and depth results on all datasets except Cityscapes, at 108.41 ms average time and 0.71 GB peak memory, versus 545.42 ms and 1.94 GB for AnyUp and 418.89 ms and 8.31 GB for LoftUp.

Deformation is guided by the input image: at each stage the current-resolution features are concatenated with the same-resolution image, a depthwise-separable residual convolution network predicts a 2D deformation field, and features are sampled coordinate-wise. No high-resolution ground-truth features exist for training, so five linear heads with a 1×1 receptive field predict RGB reconstruction, monocular depth, semantic segmentation, surface normals and intrinsic decomposition, forcing per-pixel semantic and structural matching. Training alternates batches from Hypersim and ImageNet-22k, and the final model trained on a single A100 for 24 hours; variants trained without the depth or segmentation task remain strong on those held-out tasks.

Deformation rather than direct feature-value editing is the key design that keeps downstream task semantics from being baked into the feature maps. If the upsampler could rewrite feature values directly, the supervised task signal could be encoded into the features instead of requiring all channels to be corrected jointly. In the ablation that replaces deformation with convolutions only, feature values change almost imperceptibly and task semantics leak; with deformation, only local resampling is allowed and semantics are preserved.

Perspective

The demonstrated scope is a prototype on DINOv2 features, with some experiments on DINOv3, evaluated on semantic segmentation, open-vocabulary segmentation and monocular depth estimation; the authors note weaker relative performance on Cityscapes and list extension to video and 3D as future directions.

The authors attribute the Cityscapes shortfall to weaker capture of small objects, possibly from the low-frequency bias of the convolutional architecture, and note difficulty distinguishing holes smaller than the patch size; this explanation is supported by a supplementary comparison by object size and remains a hypothesis.