Three practical defenses for face-recognition pipelines: catch poisoned identities during training, reject trigger-bearing inputs at inference
Synopsis
For face-recognition pipelines whose training data may come from compromised third parties while the model is trained in-house, a set of three backdoor defenses is proposed and tested on realistic workloads: a training-time check flags poisoned identities from anomalously fast-rising intra-identity accuracy, and two inference-time checks compare a second detector's boxes and landmarks or cut-and-paste inputs onto held-out faces. On RetinaFace detectors and five feature extractors, detector pairing reached about 7.1% average false negative rate on FGA and about 0.6% on LSA, Cut-and-Mix stayed below 1% false negative rate on well-learned backdoors, and the training-time defense failed on clean-label backdoors.
Interpretation
A face-recognition system is a chain of detector, alignment, antispoofing and feature-extraction models, so poisoning any one stage can make the whole pipeline produce false matches on trigger-bearing inputs, while most existing backdoor defenses are designed for closed-set image classification. Closed-set classification assumes a fixed label space, letting defenses run statistical tests on discrete output labels; a face feature extractor works in an open-set metric space trained with angular-margin losses, where an attacker manipulates embedding similarity rather than labels, so label-centric defenses do not apply. Across five feature-extraction architectures trained on MS1Mv2, all 12 All-to-One Poison-label attacks implanted a backdoor, against 3 successful All-to-One Clean-label attacks and 0 successful repurposed Master-Face Poison-label attacks, showing that learnability varies sharply with the poisoning regimen.
The training-time Early Poison Identification defense tracks intra-identity matching accuracy in early training batches and flags identities whose accuracy rises anomalously early. Poisoned samples are relabeled to a target identity and stamped with a trigger; the trigger is simpler than a face, so the model learns it first and the target identity's top-1 accuracy climbs earlier than any genuine identity's. It identified the backdoor identity in all 12 Poison-label cases, pruning at most 19 other identity classes, at roughly 1.5 times the training time, while missing all 3 Clean-label cases.
The two inference-time defenses guard the detector and the feature extractor: detector-level model pairing compares an off-the-shelf MTCNN against the main detector on box count, IoU and landmark drift and rejects any mismatch, while Cut-and-Mix splits an input into chunks, pastes them onto a few held-out faces and flags suspiciously high cosine similarity with the original. Earlier model pairing for feature extractors required a known-benign, equally large pre-trained extractor as a partner, whereas detectors follow standard supervised training and an off-the-shelf model can serve as the pair. Detector pairing averaged about 7.1% false negative rate on FGA and about 0.6% on LSA at about 0.8% average false positive rate; Cut-and-Mix, apart from two weak-backdoor models, kept false negative rate below 1% at false positive rates of 0.5% to 1.1%, adding about 0.02 s per image for MobileFaceNet and 0.04 s for ResNet50.
Perspective
Evaluation covers RetinaFace with MobileNetV1 and ResNet50 backbones and five feature extractors (GhostFaceNetV2, IR-SE-50, MobileFaceNet, RobFaceNet, ResNet50), trained on MS1Mv2 for extractors and WIDER-Face for detectors, with inference-time defenses tested on 2048 faces drawn from CelebA-Spoof. The defender is assumed to be the pipeline owner with white-box access to training and inference.
The training-time defense fails entirely on All-to-One Clean-label backdoors, which the authors attribute to clean-label poisoning supplying genuine identity information alongside the trigger and dampening the anomaly. Cut-and-Mix failed on the two models whose backdoors were weakly learned, which the authors attribute to lower cosine similarity between poisoned samples defeating the outlier check. The authors also note that collision attacks such as UniID, where only the insider carries the pattern, fall outside the defenses' assumption.
