Feature upsampling stops redrawing the whole map: AnchorUp learns only the interpolation correction, at a quarter of NAF's parameters
Synopsis
Vision foundation models emit features on a grid 14 or 16 times coarser than the image, and dense prediction needs them per pixel; learned upsamplers have regressed the entire high-resolution feature map directly. AnchorUp keeps a parameter-free bilinear or bicubic interpolation in the output as an anchor, has a small guidance network predict only the correction to it, and blends the two through one learnable scalar gate. Trained once on DINOv3 with only the first of NAF's two stages, it is on par with two-stage NAF on Pascal VOC and Cityscapes and beats it on DAVIS-2017 across all six backbones, with 167,425 parameters, about a quarter of NAF's.
Interpretation
A learned feature upsampler no longer has to generate the whole high-resolution feature map itself; it only supplies what classical interpolation leaves out. Earlier learned upsamplers took the low-resolution features and the image and regressed the complete high-resolution feature map, so smooth regions were recomputed by the network. Across four datasets with a fixed backbone, JAFAR, FeatUp and AnyUp failed on average to outperform bicubic interpolation, indicating that most of the target map needs no learning.
The construction keeps a parameter-free classical interpolation as an anchor, lets the learned part output only a correction, and mixes the two with a scalar gate. The learned part is NAF's upsampler unchanged, in which only the guidance network encoding the input image carries parameters; the position embedding and the neighborhood attention have none. At guidance dimension 128 the model has 167,425 parameters, one more than NAF's reduced-size variant, the extra being the gate scalar; training uses ImageNet with DINOv3-B/16 features, taking as input features of the same images downsampled by 2.
The trained correction concentrates in high frequencies and at object boundaries, which is why the guidance network can be small. The anchor already covers the low-frequency content of the output, so gradients stay large only where the anchor is wrong. At the lowest measured frequency the correction carries over two orders of magnitude less Fourier power than the anchor, while between roughly 0.06 and 0.3 cycles per pixel it carries more.
Perspective
Evaluation stays within natural-image benchmarks: linear probing on Pascal VOC and Cityscapes, and first-frame mask propagation on DAVIS-2017 driven by feature similarity, upsampling to the full image resolution or by a factor of two on DAVIS frames. Cost figures come from a single forward pass of the upsampler alone on one RTX PRO 6000 Blackwell GPU, in FP32, at batch size 1 and fixed input size.
Only two classical interpolants were tried as the anchor, and which is better is benchmark-dependent: bicubic is better on the VOC mean, bilinear on Cityscapes. The gate is one scalar shared by every pixel and channel, so it does not distinguish where the anchor is more or less reliable, for example between flat regions and object boundaries. The learned part is fixed to NAF's upsampler, although the framework accepts any learned upsampler in its place; those substitutions were not tested. Domains that separate cleanly into flat and structured regions, such as remote sensing, are where the anchoring premise is expected to be strongest, but they were not tested either.
