Skip to main content
Search

# Size- and Dispersion-Corrected Two-Level Softmax Sampling

1 related articles

Subscribe to updates
  1. arXiv

    Two-level softmax sampling is shown to be systematically biased; S-2LS and SD-2LS corrections cut KL divergence by up to four orders of magnitude

    The work proves that two-level softmax (2LS) sampling systematically misweights clusters by ignoring cluster size imbalance and intra-cluster similarity dispersion, breaking softmax's invariance for equally similar items; the authors propose S-2LS (size-weighted cluster scores) and SD-2LS (adding a second-order covariance correction), which on five large-scale datasets match exact softmax more closely at comparable or slightly higher cost than 2LS.

You’re all caught up