Implicit neural representations learn from one mesh cell at a time: one local recipe reaches 1.1–2.3% error on six conservation laws
Synopsis
Implicit neural representations are usually trained on mini-batches spread over the whole domain, because a generic network loses accuracy when a batch lies in one small region. Low-rank neural representations, whose weight matrices are products of shared bases and coefficients emitted by a small hypernetwork, can instead take each step on the points of one cell of an adaptive-mesh solution at one time snapshot. On six hyperbolic conservation-law problems one local recipe reaches 1.1–2.3% relative error with a tenth of the parameters of a dense network of the same width, and at matched size with the same local batches and steps a hypernetwork-SIREN, a DeepONet and a Fourier neural operator stay at 16–78% where the low-rank representation reaches 4.9%.
Interpretation
An implicit neural representation can be trained on spatially local, correlated mini-batches instead of a batch spread over the whole domain at every step. The prior practice keeps each batch spread over the domain and, for localized features such as shocks and contact discontinuities, moves collocation points toward large residuals; locality is usually built into the architecture rather than the batch. On six hyperbolic conservation-law problems solved with adaptive-mesh finite-volume solvers, one local training recipe reaches 1.1–2.3% relative error while the model stores a tenth of the parameters of a dense network of the same width.
Local steps work because inside one small cell the hidden states of a low-rank neural representation lie close to a low-dimensional subspace, and a gradient step reaches the shared bases only through each layer's coefficients. The low-rank representation writes every weight matrix as a rank product of shared bases and coefficients produced by a hypernetwork, which emits coefficients rather than weights. On periodic acoustics data, with the same local batches and training budget at matched size, the hypernetwork-SIREN reaches 16.3%, the Fourier neural operator 16.0% and 16.8%, and the DeepONet 77–78%, against 4.9% for the low-rank representation.
Local batches are not the more accurate way to train, since global batches are also more accurate for the low-rank representation, but its accuracy loss under local batches stays roughly constant as width grows. The same four-stage ladder with 4,096 random points per step instead of one cell's points reaches 8.4%, 2.1%, 1.6% and 1.4% after stages one to four, against 11.2%, 7.3%, 5.4% and 4.9% with local batches. At widths 500, 1000 and 2000 the ratio of local to global error is 4.7, 5.5 and 5.4 under adaptive sampling, while the hypernetwork-SIREN's ratio rises from 2.8 to 7.9 across widths 48 to 384.
Perspective
The demonstrated scope is two-dimensional hyperbolic conservation-law problems with time as the only parameter, using numerical solutions from Clawpack/AMRClaw with three refinement levels, with errors measured on the solver's adaptive points and weighted by cell area. The intended reader is a researcher or engineer training surrogates from solver output stored by region. The authors explicitly leave the actual cost saving of a local step relative to a global one as an unmeasured open problem.
Apart from the projector comparison, every result comes from a single training run; across five seeds the error of one configuration varies by about one percentage point at width 1000 and by several at width 500, so single-run differences of that size are not resolved. The baseline comparison is on one problem, only the hypernetwork-SIREN and the low-rank representation were trained at more than one width, and each sweep used two learning rates. The per-cell reduction over-fits on cells with few solver points, and the proposed remedy was identified but not tuned. The authors also state that local batches are not more accurate than global ones.
