One sentence picks the voice you want: Textraction shares one set of weights across single- and two-microphone audio, reaching 10.53 dB on spatial extraction
Synopsis
Textraction puts speaker extraction and speech enhancement into one model: a user states in natural language which speaker or noise to keep or remove, and the model outputs that target in a single forward pass without a separate-then-select stage. A lightweight text-retrieval encoder replaces the large language model, and the prompt plus the recording configuration condition every block, so single- and multi-channel inputs share the later network. On simulated reverberant data, attribute-based extraction reaches 11.11 dB (multi-channel) and 10.83 dB (single-channel) SI-SDR, spatially guided extraction 10.53 dB, and enhancement 18.66 dB (multi-channel) and 18.15 dB (single-channel).
Interpretation
Textraction lets a user name in one natural-language sentence what to pull out of a mixture, whether a speaker or a directional noise, and the same model accepts both single-microphone and two-microphone input. Earlier text-guided speaker extraction worked mainly in single-channel settings, while multi-channel schemes relied on predefined absolute regions or frozen text encoders and struggled with keep-versus-remove instructions. On simulated reverberant mixtures, attribute-based extraction raises SI-SDR from about -0.03 dB at the input to 11.11 dB (multi-channel) and 10.83 dB (single-channel), with STOI rising from 0.575 to 0.797 and 0.789.
The model feeds the prompt and the recording configuration as conditioning into every processing block, and attention decides which parts of the mixture to keep, so no stage first separates all sources and then selects the target. The common recipe separates first and then picks a target by cue, adding a stage that can fail; Textraction estimates the target directly from the mixture in one forward pass. Single- and multi-channel inputs each pass through a dedicated encoder, then share five TF-Locoformer blocks, with conditioning at six locations; the full network has about 52 million parameters, 33 million of them in the text encoder.
On the text side, a compact E5 retrieval model is contrastively fine-tuned instead of calling a large language model, so it separates opposite instructions such as keep and remove while staying stable under paraphrase. Prior work mostly used large language models for text embeddings, which are costly and often need parameter-efficient tuning such as LoRA, while frozen text encoders handled diverse prompt formulations poorly. After fine-tuning, an anchor prompt scores 0.9911 cosine similarity with a paraphrase and is clearly pushed away from the opposite instruction; across all, keep-only and remove-only prompt formulations, attribute-based extraction SI-SDR differs by less than 1 dB.
Perspective
The demonstrated setting is simulated reverberant mixtures: speech from LibriSpeech and noise from VGGSound, rendered as two-microphone recordings with an image-method room simulator at 8 kHz sampling; extraction scenes hold at most two concurrent speakers and two directional noise sources, and enhancement scenes one target speaker plus one or two noise sources.
All experiments use simulated reverberant mixtures, and behavior on real microphone-array recordings has not been evaluated; the extraction setting is fixed at up to two speakers and two directional noise sources, and scaling to more or an unknown number of sources remains open; extracting a single directional noise from strongly degraded input is still the hardest case, at about 3.8 dB SI-SDR; spatial descriptions come from a fixed set of relative relations and do not yet cover quantitative angles or distances.
