Likelihood tempering in linear Bayesian neural networks makes predictive mean and variance transition at different temperature scales, so no schedule recovers both NNGP posterior moments
For single-hidden-layer linear Bayesian neural networks under NTK parameterization, the authors derive the limiting variational predictive distribution when the temperature decays as a power of width, and find that the predictive mean and variance undergo phase transitions at different temperature scales: the mean leaves its prior value below an explicit threshold and eventually equals the least-squares prediction, while the variance departs from the prior only at colder temperatures, and no schedule of this form recovers both the untempered NNGP posterior mean and variance.