LENS stops at an interaction score: a second model on held-out data must reproduce the pattern before a feature pair enters the model
Synopsis
A high interaction score can reflect a model fitting noise, and ranking methods leave the cutoff to the user. LENS trains two models with the same learning algorithm on disjoint data and accesses them only through predictions: it ranks pairs by interaction strength after removing main effects, then tests whether the second model reproduces the pattern at a meaningful strength, with one multiple-testing correction setting the retained set. On F1–F10 with XGBoost, LENS recovers 91% of material interactions against 45% for DIAMOND, without selecting false pairs.
Interpretation
Whether a feature pair is kept is decided by whether a second, independently trained model reproduces its interaction pattern, not by how high one model scores it. Ranking methods score each pair but leave the cutoff to the user; testing methods set a cutoff but rely on repeated model refits or on modeling the joint feature distribution. Across 16 synthetic designs, LENS ranks as accurately as model-specific scores such as TreeSHAP; on F1–F10 with XGBoost it recovers 91% of material interactions against 45% for DIAMOND in the same model family, without selecting false pairs.
The data are split into discovery, confirmation and evaluation parts, two models of the same kind are trained, and both are queried only through predictions. For each pair, a joint-effect surface is built and the main effects of the two features are subtracted, leaving a residual table whose energy ranks the pair. On 179,000 evaluation rows of French motor insurance data, VehAgeVehBrand has a replication ratio of 0.78 while the fifth-ranked VehAgeRegion reaches only 0.03; the first is kept and the second rejected.
A replication ratio alone is not enough: the reproduced interaction strength must also clear a floor. Two independent fits cancel their own fitting variation in expectation, but a systematic pattern shared by the same learning algorithm survives, so the ratio can approach one even when the shared pattern is too weak to matter. Without the strength floor, XGBoost still finds every material pair on F1–F10 but its false discovery proportion rises to 0.24 and it returns a list in every run on the additive designs; with the floor, only 7 of 120 XGBoost runs on the additive designs return a pair.
Perspective
The demonstrated scope is 16 synthetic designs plus French motor insurance and two credit datasets, with XGBoost and DIAMOND's multilayer perceptron as the model families; LENS tests the interactions of a fitted model, and both fits come from the same learning algorithm.
With strongly correlated features, proxy pairs are still kept: on the Pair-corr design, LENS with XGBoost has a false discovery proportion of 0.67. A replication ratio alone retains arbitrarily weak shared patterns, so the strength floor is required, and the authors note that a cross-learner replication variant has not been evaluated.
