Skip to main content
Back to timeline
arXivSource publication:

Diffusion models no longer need a reference to find memorized regions: JaRS reads Jacobian row sums

Synopsis

Diffusion models sometimes reproduce regions of their training images, and earlier localization methods read the diagonal of the denoiser Jacobian, which is also small in smooth regions and therefore needs a reference map to subtract; unconditional models have no null prompt and intermediate checkpoints are rarely released. JaRS instead takes the row sums of that Jacobian: local averaging keeps a row sum near 1 while memorized regions drive it toward 0, so no reference evaluation is needed. Paired with an early-step image-level detector as JaRS+, it reaches the best localization mIoU in seven of ten settings across Stable Diffusion v1.4, v2.1, Realistic Vision v5.1 and an unconditional DiT-B/2.

Source-provided article image: JaRS: Contrast-Free Localization of Memorized Regions in Diffusion Models via Jacobian Row Sums
arXiv

Interpretation

Localizing the training-image regions a diffusion model reproduces can now be done from the target model alone, without a null prompt or an earlier training checkpoint as a reference. Earlier work took the diagonal of the denoiser Jacobian and subtracted a reference map to cancel the response of smooth regions; unconditional models have no null prompt, and developers usually do not release intermediate checkpoints. On Stable Diffusion v1.4, v2.1, Realistic Vision v5.1 and an unconditional DiT-B/2 under the same balanced protocol, JaRS+ achieves the best localization mIoU in seven of ten settings, improving mIoU by up to several points on Stable Diffusion and from 0.666 to 0.869 on the unconditional DiT.

What separates memorization from local averaging is the sum of each Jacobian row's weights, not a single diagonal entry. In smooth regions the denoiser averages over neighboring pixels, so the diagonal shrinks as the neighborhood grows and becomes hard to tell from a memorized region; under averaging the whole row still sums to 1, while under memorization the row's weights vanish together. On DiT-B/2, explicitly forming the full Jacobian gives a median row sum near 1 on non-memorized samples, a clearly lower row sum inside the watermark and a value near 0 on whole-image near-copies, and the row sum separates watermark pixels from non-memorized pixels; on SD v1.4, the median row sum is near 1 on non-memorized images and outside the template, and drops inside the template, on matching verbatims and on retrieved verbatims.

JaRS+ first decides from an early denoising step whether an image contains memorized content, then outputs a pixel-level localization map at an intermediate noise step. Applying a single pixel threshold to every image produces false-positive masks on highly predictable backgrounds of ordinary images. On Stable Diffusion the detection score is available early in sampling, whereas the contrast methods need the full generation trajectory; on SD v1.4 the detection AUC is 1.000 on prompts that produce a verified template verbatim copy.

Perspective

Evaluation covers three text-to-image models, Stable Diffusion v1.4, v2.1 and Realistic Vision v5.1, plus an unconditional DiT-B/2 trained on FFHQ with a planted watermark and near-copy samples; localization accuracy is reported at an intermediate noise step chosen on a development split, and the detection step and threshold are also selected there.

On Realistic Vision, JaRS+ trails the contrast methods, scoring 0.849 mIoU on All against 0.951; the gap comes from the Stage 1 threshold rather than the localization map, and the same maps reach 0.945 with an oracle image-level gate. In prompt-level detection on SD v1.4, JaRS+ reaches an AUC of 0.946 against 0.969 for a score contrast, a gap that comes from retrieved verbatim prompts that copy under only some of the four seeds and from candidate prompts that produce no copy under any of the four seeds. The method evaluates the denoiser at an intermediate noise level, and extending it to single-step distilled generators is left for future work.