Skip to main content
Back to timeline
arXivSource publication:

A frozen vision encoder gets a narrower view: vividRGB enlarges local crops and stitches them back, beating DINOv3 and SAM3 on edge F1 across five ecological image domains

Synopsis

vividRGB 0.1.0 packages an inference-time procedure as a Python library: without retraining a vision encoder, it restricts each input to enlarged overlapping local crops and maps their descriptors back into source coordinates to build a dense feature field. On grassland, fungal networks, airborne forest, coral reefs and phytoplankton microscopy, a frozen DINOv3 ViT-B/16 with this procedure reaches domain-mean RGB-edge boundary F1 of 0.290 to 0.596 at 5% focus and 0.244 to 0.441 at 10% focus, versus 0.028 to 0.076 for original DINOv3 and 0.020 to 0.090 for automatic SAM3. The cost is substantially higher extraction time and GPU energy, about 175.8 s per image at 5% focus.

Interpretation

Dense, fine structures in ecological photographs are easily smoothed away in whole-image features from general vision foundation models; vividRGB lets a frozen encoder see only enlarged local crops and then stitches the local descriptors back into a full-image field, preserving leaf, hypha and canopy edges and shapes in the feature map. The prior route fed the whole image to DINOv3 once, or let SAM3 generate masks automatically; on the same five ecological image domains both agreed poorly with visible brightness edges. On 50 working images, ten per domain, DINOv3-FFA5% reaches domain-mean RGB-edge boundary F1 of 0.290 to 0.596 and DINOv3-FFA10% reaches 0.244 to 0.441, against 0.028 to 0.076 for original DINOv3 and 0.020 to 0.090 for automatic SAM3; 5% focus exceeds 10% in every domain, and all 20 paired comparisons remain significant after Holm correction.

The procedure leaves encoder parameters untouched and changes only the context the encoder sees: each input is a square crop sized as a fraction of the image's short side, enlarged to the encoder input resolution and encoded on its own. Forced focus means an externally imposed field-of-view restriction, not a new attention layer, and attention coefficients are not manipulated; backbone weights are identical across conditions. Each crop is encoded in four orientations, original, horizontal reflection, vertical reflection and both, then inverse-aligned and averaged; each crop's feature map is bilinearly placed back at its source coordinates, and overlaps are weighted by a Hann taper and divided by total coverage weight to give one continuous image-aligned feature field. Experiments ran on one NVIDIA RTX 4000 SFF Ada GPU, with 5% and 10% focus taking about 175.8 s and 43.8 s of extraction time per image.

Perspective

The demonstrated scope is appearance-boundary recovery and computational cost on 50 working RGB images, ten per domain across five ecological domains, with a frozen DINOv3 ViT-B/16 and an exploratory five-image SAM3 image-encoder test; the authors propose task-specific biological annotation benchmarks and factorial comparisons to separate effective resolution, contextual scope and aggregation as the next step.

The authors state that the primary quantitative endpoint measures agreement between clustered-descriptor boundaries and RGB brightness edges, which may reflect texture, shadows or imaging artifacts and can miss biologically meaningful distinctions, so it needs validation against the intended biological task; the 50 photographs provide examples but do not establish generalization across representative datasets or acquisition conditions; and matched comparisons with feature-reconstruction methods such as FeatUp and AnyUp are missing.