Kuaishou's three-month logs become OmniBehavior: best model scores 44.55 overall against a 56.11 human baseline
Synopsis
Existing user-simulation evaluations are largely confined to isolated scenarios, narrow action spaces or synthetic data, and miss the cross-scenario, long-horizon structure of real behavior. OmniBehavior is built from three months of real Kuaishou logs, covering 5 scenarios and 22 actions across 200 representative users and 2.12 million interactions. On it, the best model, Claude-Opus-4.5, reaches an overall score of 44.55 against a human baseline of 56.11, and extending context from 16K to 128K tokens does not reliably improve performance.
Interpretation
The object of user-simulation evaluation shifts from short fragments in isolated scenarios to real behavior traces that span scenarios, run long, and mix action types. Earlier benchmarks mostly covered a single scenario such as video browsing, live streaming, e-commerce or customer-service dialogue, with action spaces limited to a few types like clicks and watches, so a purchase triggered by a video watched days earlier stayed invisible. OmniBehavior aggregates Kuaishou logs from September 1 to November 30, 2025, covering 5 scenarios and 22 actions, with 200 users' traces totaling 2.12 million interactions, an average sequence length of 8,143 actions and traces exceeding 100,000 steps.
Evaluation casts user simulation as conditional prediction: given a user profile, the historical behavior sequence and the current scenario context, a model must predict all of the user's behaviors in that scenario. Behaviors are scored in three types: binary behaviors such as likes and shares by F1, continuous behaviors such as watch duration by duration-normalized mean absolute error, and textual behaviors such as customer-service replies judged by Claude-Sonnet-4.5 on intent fidelity, persona mimicry, knowledge boundary and semantic alignment. The main experiment evaluates 11 models on 6,000 behavior prediction tasks with a global 32K context window, using the average of three annotators with master's degrees in engineering over 100 sampled tasks per scenario as the human baseline.
Simulated behavior carries a structural bias against real behavior: models overestimate positive interactions, collapse distinct users into one, and avoid negative emotion. Real behavior has a positive rate below 10%, while simulators such as Qwen3-235B and Gemini-3-Flash overestimate engagement by 40 to 60 percent; real users show far larger inter-user than intra-user variation, a ratio of about 0.29, whereas simulated users show ratios of about 0.7 to 0.87 with heavily overlapping distributions. In customer-service dialogues real users often express strong negative emotion while model utterances cluster around neutral or positive sentiment and use politer wording; the 19-dimensional action-rate comparison of intra- and inter-user distances shows the same trend across all evaluated models.
Perspective
The evaluation covers 200 representative users from three months of Kuaishou logs, with an offline conditional-prediction task that predicts behavior from a user profile, historical sequence and current scenario context, not closed-loop interaction with an environment.
The data come from a single platform, which the authors note cannot fully represent diverse cultural and linguistic contexts, and the offline formulation does not capture closed-loop interaction. Performance does not reliably improve as context grows from 16K to 128K tokens, and retrieval and summarization memory strategies yield only modest gains.
