Skip to main content
Back to timeline
arXivSource publication:

Reward models stop returning a bare score: VRMaaS binds each reward to its request, weights and configuration in a checkable artifact

Synopsis

Reward models (RMs) increasingly act as service components in LLM alignment pipelines, yet a conventional interface returns only a score and cannot show whether it came from the intended request, checkpoint and numerical configuration. VRMaaS turns each returned reward into a proof-carrying artifact whose off-chain acceptance rule binds request identity, committed weights, verifier and configuration identifiers, proof-task order and export digest, so auditors can check a reward without rerunning the private model. On 1B, 3B and 8B reward models over 128 to 1024 tokens it keeps 93.5% to 99% pairwise agreement with the baseline RM and rejects every tested service-field substitution.

Source-provided article image: VRMaaS: Verifiable Reward-Model-as-a-Service for Trustworthy LLM Alignment
arXiv

Interpretation

A reward-model output becomes a service artifact that can travel across organizations rather than a bare number: the artifact binds the invocation to a specific request, registered model weights, verifier implementation and numerical configuration, so downstream audit, settlement or governance workflows can judge whether the score belongs to the service it claims. A conventional reward-model API returns a score without an accepted statement tying it to the request, checkpoint, numerical configuration and verifier logic, so a provider can substitute a cheaper model, replay a stale result, alter configuration, or graft a valid-looking output onto the wrong context. In a controlled comparison that isolates the accepted statement from prover scale, the GenericProof+Wrapper baseline, which wraps an inference proof for on-chain checking, remains valid after six classes of service-field substitution, whereas the corresponding VRMaaS checks reject them.

The service decomposes the Transformer reward-model forward pass into arithmetic and lookup-backed proof tasks, aggregates them into a composite proof, and links hidden weight operands to the committed model weights. The construction reuses standard sumcheck, lookup-argument, polynomial-commitment and Groth16 wrapper components; what changes is the accepted statement itself, from a raw inference proof to a reward-service output bound to request, model, configuration, verifier, task order and export digest. On a GPU prototype with Skywork 1B, 3B and 8B reward models over 128 to 1024 tokens, complete artifacts are generated in 49.8 to 206.1 seconds, auditors verify retained artifacts in 1.94 to 5.11 seconds, and optional partial public attestation costs about 217K gas.

Perspective

The demonstrated scope is a single-node GPU prototype on Skywork 1B, 3B and 8B reward models over 128 to 1024 tokens of repeated question-answer comparison scoring, with complete artifact generation at 49.8 to 206.1 seconds, auditor verification at 1.94 to 5.11 seconds and partial public attestation near 217K gas, aimed at delayed audit, settlement and governance workflows; the computation proof starts from field-encoded hidden states supplied in a canonical reward-model input package, and embedding lookup plus derivation of padding, segment and last-token position from raw text are outside the implemented proof.

Long-context quantization drift lowers fidelity, and the 8B/1024 setting reaches mean absolute error 2.326 and P95 absolute error 7.332, so it is flagged for a finer numerical configuration; the on-chain wrapper proves only selected fold, partial sumcheck, reward-consistency and public-field equality relations, neither deriving the complete export nor recursively verifying the full composite proof; off-chain weight binding still exposes per-request evaluation openings to auditors; optimized latency rows contain one to five completed runs per setting, and multi-tenant scheduling, cache invalidation and checkpoint rotation remain unvalidated.