StreamEmbed indexes screen history from pixels as it arrives: up to 3.7× faster than full-screenshot encoding while keeping over 98% of its accuracy
Synopsis
Screenshot retrieval has meant choosing between OCR text indexes that discard what only pixels show and visual encoding that takes 2.33 seconds per screenshot from scratch. StreamEmbed treats screenshots as a stream: it keeps only pixel blocks that differ from a neighbor, reuses cached keys and values for blocks unchanged across screenshots, and a small adapter pulls the result back toward the full-screenshot embedding. On its ScreenMemEval benchmark it encodes up to 3.7× faster than full-screenshot encoding on an M2 Pro GPU while retaining over 98% of that accuracy, and beats OCR-based indexing by up to 6.3 Recall@10 points at similar latency.
Interpretation
Screen history can now be encoded and indexed in its native visual form as each screenshot arrives, instead of choosing between fast but lossy OCR text and accurate but slow full-image visual encoding. Existing open-source tools convert each screenshot to OCR text before indexing, which is fast but drops layout and selection state that only pixels show; embedding the full screenshot directly takes 2.33 seconds each, occupying 28 minutes of GPU time per hour at one screenshot every 5 seconds. On ScreenMemEval, with the same Qwen3-VL-2B backbone and the same training data, StreamEmbed encodes a screenshot in 0.66, 0.84 and 0.63 seconds on SME-H1, SME-H2 and SME-Agent, 2.8–3.7× faster than the fine-tuned full-screenshot index at 2.33–2.34 seconds, with R@10 lower by 1.3, 1.2 and 0.6 points.
The speedup comes from computing only the parts of the screen that actually change, not from a smaller model or lower resolution. Screen content repeats heavily within and across screenshots: in screen recordings, 67–77% of the pixel blocks that become visual tokens either repeat a neighbor exactly or are unchanged from the previous screenshot, against 6–19% in natural video. Spatial pruning first drops blocks that repeat a left or upper neighbor within an 8-of-255 pixel tolerance, cutting computation by 33–41%; temporal reuse then carries over cached keys and values for blocks unchanged across screenshots, cutting the remaining computation by a further 34–57%; together they reduce per-screenshot computation by 57–71%.
Pruning and reuse shift the embedding away from the full-screenshot result, and a small adapter merged into the weights recovers that gap without adding inference latency. On SME-Agent, pruning every redundant block costs far more accuracy than on the two human histories, indicating that dropping all redundant blocks moves the embedding. A grid-keep rule retains one block per fixed tile, recovering most of the loss at about 0.15 seconds per screenshot; a consistency adapter distills the full-screenshot teacher into the pruned-input student and recovers more, leaving StreamEmbed within 1.3 R@10 points of the full screenshot on all three histories.
Perspective
The demonstrated scope is the three ScreenMemEval screen histories: two from human workers on Windows and one from a computer-use agent running GPT-6.1 Sol on Ubuntu; latency is measured on the GPU and CPU of an Apple M2 Pro, and the index-memory experiment simulates one to five years of screenshots by adding noise to copies of real embeddings.
The authors list learning what to recompute from the stream itself as future work, so current recomputation decisions still rest on pixel-difference thresholds and a fixed refresh rule; SME-Agent is an out-of-domain source on which StreamEmbed scores 1.0 R@10 point below the base full-screenshot index, and pruning costs noticeably more accuracy there than on the two human histories.
