Skip to main content
Back to timeline
bioRxivSource publication:

ProtCOVER swaps per-variant conformal calibration for homology families: with no shift, pathogenic coverage on new families returns to the 0.90 target from 0.823–0.873

Synopsis

Zero-shot variant effect predictors calibrated conformally per variant let well-studied protein families set the error rate, leaving pathogenic coverage on new families at 0.823–0.873. ProtCOVER treats homology families rather than individual variants as the exchangeable unit and calibrates conditionally on label-free family descriptors; under the shift those descriptors define, its worse-class shortfall is 0.000, against 0.010 for group weighting and 0.029 for hierarchical conformal, at the price of larger sets. On 12,225 proteins outside the benchmark every method still falls short, with ProtCOVER at 0.027 and group weighting at 0.034, within the 0.01 margin.

AI-generated editorial illustration: ProtCOVER: Family-Aware Conformal Prediction for Variant Effect Predictors on Unseen Protein Families

Interpretation

The unit of error control for variant effect predictors moves from the individual variant to the homology family, making "a random variant of a new family is covered" a calibratable target. Variant-level calibration is valid for its own target, but families contribute between 2 and 741 labeled variants, so the largest families set the error rate and the 0.90 target describes variants of well-studied genes. On ProteinGym clinical substitutions, class-conditional (Mondrian) split conformal comes within 0.005 of its own variant-level target for four of five predictors, yet family-weighted pathogenic coverage is only 0.823–0.873 at a 0.90 target for all five, with no distribution shift at all.

The threshold no longer stays fixed across test families but varies with each family's label-free descriptors: sequence similarity, alignment depth and embedding distance. Family-level pooling already exists as hierarchical conformal prediction, but it gives every test family one threshold and cannot adjust for shifts the descriptors reveal. Under the descriptor-defined shift S2, ProtCOVER's mean worse-class shortfall is 0.000, against 0.010 for group weighting and 0.029 for hierarchical conformal; on orphan families and held-out embedding regions it is 0.002 against 0.003 for group weighting.

Calibration is a family-weighted quantile regression in which the test family's scores sit at positive infinity and the threshold is the largest optimal fitted value. The test family's labels are unknown, and the minimizer may be non-unique or the program unbounded, so a direct solve does not guarantee validity at the threshold actually computed. The program is solved through its dual linear program, where test families change only the right-hand side and HiGHS re-solves from the previous basis; the returned basis is certified optimal for the perturbed program in exact rational arithmetic and the fitted value is rounded up, so the computed sets keep every guarantee.

Perspective

The conclusions apply to five zero-shot predictors on the ProteinGym clinical substitution benchmark, to 2,081 families clustered at 30% identity and 80% bidirectional coverage, and to two ClinVar external evaluations run under pre-registered protocols; coverage is a marginal guarantee for a random variant of a random family, not for any particular protein.

The guarantees require family exchangeability, or a family-level shift whose density ratio is a nonnegative function in the span of the features with the ratio fitted on families independent of those it calibrates; under other shifts the loss is bounded only by total-variation distances that unlabeled data cannot estimate. In the S5 stress test, where predictor accuracy shifts, every family-level method's worse-class coverage drops to 0.761–0.825, and the effective sample size of the calibration weights is 0.98 of the number of families, so the ratio neither corrects nor flags such a shift. On proteins outside the benchmark the ratio barely departs from one (effective sample size 0.96), and the 223 families longer than any benchmark protein carry much of the shortfall, which neither the ratio nor the gate flags; these are plausible explanations from exploratory diagnostics, not proven causes.