Bilevel optimization stops flattening weight matrices: BiMuon adds orthogonalized updates and matches single-level Muon's convergence order
BiMuon brings Muon-style orthogonalized matrix updates into stochastic bilevel optimization, so upper-level parameters move along a matrix's singular spectrum instead of a flattened vector updated coordinate by coordinate. Nearly all prior bilevel methods used that flattened form, and the analysis shows lower-level curvature is squared through the bilevel composition, leaving upper-level hypergradients with more unequal singular values. BiMuon tracks the lower-level solution, estimates the hypergradient with a truncated Neumann series, then orthogonalizes the momentum; both the exact-polar and finite Newton–Schulz versions carry finite-time convergence rates, with sample complexity matching the best known for single-level Muon-type methods.