Skip to main content
Back to timeline
bioRxivSource publication:

Loki2 reads molecular state from H&E at single-cell resolution, lifting cell-typing weighted F1 from 0.608 to 0.806

Synopsis

Routine H&E slides show cell morphology but not molecular state, and earlier pathology foundation models worked at multicellular resolution, leaving open whether a single cell's molecular program can be read from its appearance. Loki2 distills molecular supervision into morphology using UniSeg, a resource of about 15 million cells across 15 tissue types and 22 major lineages, aligning each nucleus's morphology embedding with a transcriptomic embedding. On cell typing it reaches weighted F1 0.806 and weighted accuracy 0.807, against weighted F1 between 0.574 and 0.608 for PLIP, GigaPath and UNI used as frozen feature extractors, and the genes it retrieves skew toward morphologically distinguishable cell identities and functions.

Source-provided article image: A Biologically Interpretable Single-Cell Pathology Foundation Model for Molecular Decoding of H&E Images
bioRxiv · Page 5

Interpretation

The unit of pathology image analysis moves from a multicellular field to a single nucleus, with the model emitting an identity and a molecular-state embedding for each cell. Earlier pathology foundation models trained at gigapixel scale, and visual-omics models relied on spatial transcriptomics spots around 55 micrometers wide that mix several cells, so they learned averaged associations rather than the relation between one cell's appearance and its molecular program. On a held-out set spanning 15 tissue types and 9,573 test instances, Loki2 scored higher multi-class panoptic quality and multi-class F1 than HoVer-Net and CellViT (one-sided Mann-Whitney U test, P below 0.001), and also led on 7 unseen whole-slide images and 8 unseen patients.

The model encodes the image patch around each nucleus into a high-dimensional morphology embedding, then uses two decoders to predict cell type and reconstruct the transcriptomic embedding. Supervision comes from molecules rather than self-supervision or report text alone: a cell-type decoder maps visual features to transcriptome-defined identities, a transcriptomic decoder reconstructs each cell's position in the OmiCLIP gene-expression latent space, and a contrastive objective pulls morphology embeddings toward their molecular targets. The UniSeg training resource integrates 65 studies, 15 tissue types and about 15 million cells, with high-resolution ground truth from Visium HD and Xenium, Thor-inferred single-cell profiles for spot-level Visium, and expert-reviewed lineage annotations.

Latent dimensions of the morphology embedding correspond to nuclear phenotypes that can be measured independently, not just to class labels. Seven conventional pathology morphology descriptors reach only 0.41 accuracy and weighted F1 for cell typing, showing that this predefined feature set does not account for the information the embedding carries. In 18 esophageal cancer specimens with paired H&E and DAPI images, groups defined by chromatin-density-associated principal components differed in DAPI-measured chromatin density and groups defined by nuclear-enlargement components differed in nuclear size, with separations of 0.4 to 1.2 within-slide standard deviations and all but one comparison significant (two-sided Mann-Whitney test, P below 0.01).

Perspective

The demonstrated scope is single-cell segmentation, typing, morphology-to-transcriptome retrieval, virtual staining of selected protein markers and cell-level multiple-instance-learning tumor staging on routine H&E whole-slide images, validated on a 15-tissue held-out set, unseen slides and patients, and the TCGA and PLCO external cohorts. The authors propose next combining Loki2 with targeted genetic or pharmacological perturbations and paired morphology-molecular measurements, and integrating annotations of spatially defined tissue regions to connect to broader tissue phenotypes.

Retrieval-based molecular inference depends on the reference single-cell atlas containing the relevant cell types and disease states; when they are absent, cells may be assigned to biologically mismatched populations. Cellular crowding, weak within-cell-type morphology-molecular coupling and incomplete reference coverage all limit prediction accuracy. The study includes no perturbation experiments, so causality between morphology and molecular programs is not established, and patient-level prediction is constrained by the availability of large, well-annotated cohorts such as matched primary and metastatic samples.