Skip to main content
Back to timeline
arXivSource publication:

Modify an open-weight model and its watermark evidence shifts: LoRA fine-tuning drops OpenStamp detection to 28.8% and 11.6%

Synopsis

Open-weight models keep generating after users modify their released weights, while an OpenStamp-style verifier still scores outputs against the original base and marked models. Treating each modification as a residual on next-token logits, the work derives an exact lower bound on expected log-likelihood-ratio evidence under a residual-span constraint, plus a high-confidence upper bound on sequence-level miss probability. On Llama-2-7B and Mistral-7B-v0.3, INT8 quantization and 30% pruning keep test true-positive rates at 85.6%–94.8%, while 500-step LoRA fine-tuning lowers them to 28.8% and 11.6%.

Source-provided article image: Watermark Evidence under Open-Weight LLM Modification: Exact Bounds for Low-Rank Output Watermarks
arXiv

Interpretation

Watermark evidence loss can now be bounded by the span of the residual a modification induces on vocabulary logits, without first knowing which modification it is. Earlier robustness discussion for open-weight watermarks centered on text editing, paraphrasing and adaptive evasion; how much evidence a verifier retains after weights are modified lacked a quantitative account. On Llama-2-7B and Mistral-7B-v0.3, INT8 quantization and 30% pruning give test true-positive rates of 85.6%–94.8%, close to the unmodified marked checkpoints at 94.8% and 85.4%; 500-step LoRA fine-tuning lowers them to 28.8% and 11.6%.

The approach represents the change at each shared prefix as a residual vector added to next-token logits, and the residual's span sets how far token probabilities can be redistributed. The covariance between that residual and per-token watermark scores gives the first-order change in expected evidence, with an explicit second-order remainder. Across 160,000 test prefixes, the covariance prediction matches the direction of measured mean LoRA evidence drift, with no remainder-bound violations; each model shows zero false positives among 500 base-model continuations.

Perspective

Experiments instantiate the OpenStamp verifier on Llama-2-7B and Mistral-7B-v0.3, with candidate modifications limited to INT8 quantization, 30% per-layer unstructured magnitude pruning and a 500-step OpenWebText LoRA update, and all generation uses raw temperature-1 softmax multinomial sampling for exactly 64 steps.

The familywise 95% holdout upper bound equals 1.0 for all eight model-candidate pairs, so the certificate is valid but numerically uninformative under the current sample and moment envelope; the authors note that larger certification sets, more seeds and more modification types are needed to assess when it becomes informative. The covariance remainder bound is conservative on these checkpoints, with mean absolute remainders far below the allowance under LoRA. The study uses one watermark seed and one prompt-pool repetition, and reported intervals quantify within-run sampling uncertainty only.