RNAtomy trains RNA models on 3D geometry, not just sequence: with about 200,000 pretraining RNAs it leads the strongest baseline by 7 points on long-range contacts
Synopsis
RNAtomy is a self-supervised RNA foundation model whose pretraining input is no longer sequence or secondary structure alone, but inter-nucleotide distance and orientation maps plus per-residue descriptors of backbone torsions, the glycosidic torsion and phosphate geometry. A hybrid bidirectional Mamba and shifted-window attention backbone jointly reconstructs both feature sets, and in long-range contact prediction on the RNAcontact benchmark the frozen-backbone RNAtomy-FB reaches 0.73 precision at L, 7 percentage points above the strongest baseline. Pretraining used roughly 200,000 RNAs, fewer than the millions of sequences behind many compared foundation models.
Interpretation
RNA pretraining now learns three-dimensional geometry directly instead of inferring structural signals from sequence and secondary structure. Earlier RNA foundation models were pretrained mainly on sequence or secondary structure, and structural information appeared as paired states, dot-bracket annotations or attention biases rather than explicit inter-nucleotide distances, relative orientations and local backbone conformations. On long-range contact prediction in the RNAcontact benchmark, the frozen-backbone RNAtomy-FB reaches precisions of 0.94, 0.91, 0.84 and 0.73 at the L/10, L/5, L/2 and L cutoffs, ranking first at every cutoff and exceeding the strongest baseline by 5, 3, 3 and 7 percentage points.
The model reconstructs pairwise geometry maps and residue-level conformational descriptors together under one self-supervised objective. The pairwise maps come from distance and orientation channels predicted by trRosettaRNA, and the residue descriptors are 23 features extracted by DSSR: sine-cosine encodings of six backbone torsions and the glycosidic torsion, two phosphate-geometry quantities, and seven binary indicators of pairing and secondary-structure context. Pretraining data came from RNAcentral, sampled by RNA category and clustered at 95% sequence identity, yielding 391,845 non-redundant sequences across 16 RNA categories with lengths restricted to 50 to 1024 nucleotides.
The encoder uses bidirectional Mamba to propagate global context and shifted-window attention to restore local content-dependent interactions. Mamba reduces time and space costs from quadratic to linear in the number of tokens, but lacks the content-dependent token interactions that attention provides. The backbone has 16 blocks arranged as four repeated groups of three bidirectional Mamba blocks followed by one shifted-window Transformer block, giving 12 Mamba blocks and four window-attention blocks; pairwise geometry maps are split into non-overlapping P×P patches and projected into tokens.
Perspective
The demonstrated scope is evaluation on seven downstream tasks: solvent accessibility, 3D contact maps, long-range base pairs, miRNA-target interaction, 17 RBP binding datasets, 5′ UTR mean ribosome load and IRES identification, with the model pretrained on RNAs of 50 to 1024 nucleotides. The intended readers are researchers and engineers working on RNA representation learning and structure prediction.
The model currently represents each RNA with a single static structural description, whereas RNA is flexible and can occupy multiple conformational states whose populations shift with ions, ligands and molecular partners, so one pairwise map and one set of residue descriptors cannot fully express that heterogeneity. Across the 17 RBP datasets, RNAtomy's mean AUPRC is 0.855, below ERNIE-RNA's 0.865, although it performs better or equivalently on most individual datasets; in long-range base-pair prediction the PKF category still favors RNA-km at 0.61 over RNAtomy at 0.56.
