Skip to main content
Back to timeline
发表出处待核验Source publication:

Pathology images and multilayer molecular profiles share one embedding: Multi-Embed lets morphology and molecules retrieve each other

Synopsis

Multi-Embed places multilevel tissue morphology and multilayer molecular profiles in a single interpretable embedding space, so pathology images and molecular profiles can be inferred from and integrated with each other in both directions. Multimodal learning had left methodological challenges in cross-modality inference and integration, and the framework reports superior performance across benchmark tasks including morphology-molecule inference, fine-grained tissue architecture identification and spatiotemporal trajectory modeling. Evaluation covers paired pathology images and RNA-seq from 12 cancer types in TCGA and adds external validation on independent datasets such as CPTAC.

Source-provided article image: Systematically decoding pathological morphologies and molecular profiles with unified multimodal embedding | Nature Methods
Fig. 1 · Nature Machine Intelligence

Interpretation

A pathology slide and a molecular profile can now be inferred from each other through one unified embedding space: morphology predicts molecules, and molecules retrieve morphology. Cross-modality inference and integration had been constrained by methodological challenges in multimodal learning, with no unified way to relate morphology to multilayer molecular profiles. On paired pathology images and RNA-seq from 12 cancer types in TCGA, morphology-related gene expression prediction was compared per cancer type against two benchmark models, HE2RNA and DeepPT, under five-fold cross-validation, with external validation on the independent CPTAC database.

The same space carries several molecular layers, so one pathology image can be tied to expression, protein abundance and mutation at once. Cross-modality inference had required separate predictions for molecular layers such as methylation, protein abundance and genomic mutation, with no shared embedding across them. On TCGA, morphology-related prediction of methylation, protein abundance and genomic mutation was compared against DEPLOY, WSI2RPPA, and CHIEF and MUSK respectively; on the spatial proteomics dataset ORION-CRC, 16 proteins were compared against OmiCLIP and ROSIE.

The embedding also supports spatial clustering and trajectory inference, reading out tissue architecture, tertiary lymphoid structures and spatial trajectories of malignant progression together. Spatial clustering and trajectory inference had been handled separately by multimodal models such as OmiCLIP, MISO, MUSE and SpaGCN and unimodal models such as GraphST, STAGATE and BayesSpace, with tissue regions and trajectories treated apart. On the HER2ST spatial transcriptomics dataset, adjusted Rand index was compared against the multimodal models OmiCLIP, MISO, MUSE and SpaGCN and the unimodal models GraphST, STAGATE and BayesSpace, while on the TNBC dataset the comparison was against unimodal models; on human tonsil spatial multi-omics data, F1 scores for germinal center identification were compared.

Perspective

The demonstrated scope is cross-modality inference and integration between cancer pathology images and molecular profiles, using paired images and RNA-seq from 12 cancer types in TCGA, the CPTAC external validation set, and spatial datasets including HER2ST, TNBC, ORION-CRC, human tonsil, stomach cancer and breast cancer; code and an online interactive platform are public.

The visible text summarizes benchmark results as superior performance without giving specific metric values; patient or sample counts for each dataset sit in a supplementary table not expanded in the preview. Reliability of cross-modality inference depends on the cancer types and modality combinations covered by training data.