Portrait video relighting stops assuming static light: FlashRelight splits delighting from relighting to follow frame-by-frame envmap sequences
Synopsis
Portrait video relighting has mostly targeted static or mildly varying illumination, and struggles when direction, intensity and color change over time. FlashRelight splits the task in two: a delighting model suppresses the source lighting to obtain an approximately illumination-neutral video, then a relighting model built on a DiT video generative prior synthesizes the target dynamic lighting. It accepts environment-map sequences, dynamic lighting text and lighting reference videos as controls; on a held-out test set of 10 subjects and 16 envmap sequences it beats the three listed baselines on lighting accuracy, LPIPS, identity similarity, PSNR and tLPIPS.
Interpretation
A portrait video can be relit while illumination direction, intensity and color vary over time, preserving identity, motion and temporal coherence. Earlier relighting methods mostly target static illumination or only mild lighting variation, and frame-by-frame processing introduces flicker. On a held-out test set of 10 subjects and 16 envmap sequences not used in training, FlashRelight reaches a lighting-accuracy error of 0.0584, LPIPS 0.1635, identity similarity 0.8097, PSNR 23.1099 dB and tLPIPS 0.1303, better than the corresponding values for Lite2Relight, Relighting-Cai and DiffusionRenderer.
The method first removes the original illumination from the video, then regenerates it under the target lighting on the neutral-light video. The delighting model only needs to provide a stable appearance representation that preserves identity and motion, not a physically exact intrinsic decomposition; the relighting model builds on the pretrained DiT video generative model Wan2.1-1.3B and concatenates the delighted video latent with the noisy target latent along the channel dimension at each denoising step. Training data mixes static OLAT data with generated dynamic-lighting portrait videos: the former from 60 FaceOLAT subjects with 2–4 expressions each, paired with 600 synthetic envmap sequences and 200 real environment maps augmented by rotation and temporal transforms, for 24,000 paired videos; the latter exceeds 10,000 pairs.
Three control modalities cover needs from frame-level precision to semantic description to cross-video transfer. Envmap sequences give explicit frame-level control, dynamic lighting text gives a semantic editing interface, and lighting reference videos give direct visual transfer cues. The envmap branch performs token-level temporal alignment before cross-attention, attending only to envmap tokens at the same temporal index; the text branch is encoded by T5 and injected through text cross-attention; the reference-video branch is fused by channel-wise concatenation at the latent input layer.
Perspective
The demonstrated scope is 49-frame portrait clips at a given resolution; the quantitative conclusions come mainly from the envmap-sequence setting on a held-out test set, while the text and reference-video settings have only partial quantitative metrics or qualitative demonstrations.
Delighting accuracy is the main limitation: strong projected patterns, fast head motion, facial paint, motion blur or severe image degradation make source-light removal ambiguous and may suppress useful appearance cues such as skin tone, facial detail or motion structure. The text branch is effective for a limited set of lighting descriptions and does not reliably follow general free-form lighting prompts; the current formulation primarily relights the portrait foreground without explicit background relighting, so foreground and background illumination may become inconsistent.
