Skip to main content
Back to timeline
arXivSource publication:

Encoding channels separately wins at scale: Meta FAIR's DINO-BoC beats joint channel encoding on microscopy and satellite imagery

Synopsis

Scientific images differ in channel count and semantics, and joint channel encoding was long assumed necessary to learn cross-channel relations. Meta FAIR and collaborators compared strategies on an extended training set of about 7.75 million images and found that DINO-BoC, which encodes each channel independently and concatenates features at inference, outperforms the joint-encoding Channel-ViT on most microscopy and geospatial tasks and exceeds the supervised HyperNet baseline on CHAMMI's mean linear-probe out-of-distribution score. The finding covers in-domain, cross-dataset and unseen channel combinations, though protein-localization tasks still need a linear probe to catch up under nearest-neighbor evaluation.

Source-provided article image: Scaling Channel-Adaptive Self-Supervised Learning
arXiv

Interpretation

On scientific images with heterogeneous channel composition, passing each channel separately through a shared backbone and concatenating channel features at inference outperforms joint channel encoding once training is scaled up. Earlier work argued that jointly encoding channels with learnable channel embeddings preserves cross-channel information and is therefore preferable when channel counts vary, while independent encoding was seen as giving up explicit channel interactions. On an extended training set of about 7.75 million images, DINO-BoC beats DINO Channel-ViT on all four tasks of HPA-FOV protein localization, HPA-FOV cell type, JUMP-CP perturbation classification and WTC cell-cycle stage, for example 61.7 versus 57.4 F1 on protein localization.

The recipe splits a multi-channel image into single-channel inputs, runs each channel through a shared vision Transformer encoder, and concatenates the resulting channel-level features into an image-level representation. Called Bag of Channels, the strategy resembles late fusion in multimodal learning: training draws one random channel at a time and performs no cross-channel computation inside the model. On the CHAMMI benchmark with a linear probe, DINO-BoC reaches a mean out-of-distribution F1 of 67.9, above DINO Channel-ViT at 59.8 and the supervised HyperNet baseline at 65.4; on Meter-ML with all channels it reaches 78.1 mAP.

Independent encoding is also fully compatible with arbitrary channel sets, because the model does not depend on the maximum channel count seen during pre-training and can process more channels than it was trained on. Joint-encoding methods can only handle inputs up to the pre-training maximum, since extra channels have no trained channel embeddings. On JUMP-CP, with pre-training capped at five channels, DINO-BoC scores 45.2 accuracy with five channels and 51.6 with eight, while Channel-ViT cannot be evaluated on eight channels.

Perspective

The conclusions rest on two channel-heterogeneous domains, fluorescence microscopy and geospatial imagery, with an extended CHAMMI training set of about 7.75 million images and evaluation on HPA-FOV, WTC, JUMP-CP, Cyclops and Meter-ML; the authors also report an ImageNet comparison in which, at a fixed amount of training data, DINO-BoC reaches 82.8 against 81.9 for the fixed-channel model, while at a fixed number of gradient steps the fixed-channel model is only 0.3 points higher (83.1 versus 82.8).

Under CHAMMI's nearest-neighbor evaluation, DINO-BoC trails Channel-ViT and the supervised baseline on HPA protein-localization tasks that require cross-channel spatial reasoning, and only catches up with a linear probe; the authors also note that the claimed unified-embedding-space advantage of joint encoding remains unclear in practice.