Skip to main content
Back to timeline
arXivSource publication:

Accent conversion drops the fixed neighbor count: AA-NN-AC reaches 8.35% WER with a 25-minute target-accent pool

Synopsis

Retrieval-based accent conversion takes acoustic features directly from target-accent speech instead of generating them, but earlier matching pools were built by sequential selection, the neighbor count was fixed, and style modulation was coarse, leaving rare phonemes underrepresented and spectra discontinuous. AA-NN-AC builds the pool by greedy phoneme-coverage selection, uses singular value decomposition to estimate local anisotropy and set the neighbor count per frame, and adds temporal up-sampling inside the AdaIN blocks. On L2-ARCTIC and VCTK it beats retrieval baselines such as NN-VC and NN-AC on objective and subjective metrics, with 8.35% WER and 4.04% CER.

Source-provided article image: Adaptive and High-Fidelity Matching Algorithm for Retrieval-Based Accent Conversion
arXiv

Interpretation

The matching pool for retrieval-based accent conversion is no longer cut in original chronological order; samples are chosen by phoneme coverage so rare phonemes enter the retrieval index even with limited target-accent data. Earlier retrieval frameworks such as NN-AC built the pool by sequential selection, and because phonemes are unevenly distributed in natural speech, common phonemes were overrepresented while rare ones could be absent. When target-accent data is 25 minutes or less, the coverage-selected pool consistently beats the sequential-selection version on WER and CER; a 25-minute coverage pool matches the sequential version built from 60 minutes.

The number of neighbors for each source frame is no longer fixed; it varies frame by frame with how directionally strong the nearby feature distribution is. A fixed neighbor count pulls in mismatched neighbors where features change rapidly and makes retrieval unstable where the distribution is uniform. On L2-ARCTIC speakers ASI and ABA and VCTK speaker P248, AA-NN-AC reaches WER of 8.35%, 15.72% and 5.86% and CER of 4.04%, 7.31% and 2.37%, all below NN-AC's 13.12%/6.59%, 17.52%/8.47% and 6.40%/2.71%.

Speaker style modulation moves from a single temporal resolution to two, injecting speaker statistics at 50 Hz and 100 Hz before the vocoder restores the waveform. The original NN-AC decoder applied AdaIN only at a coarse time scale and left fine harmonic structure to later HiFi-GAN layers, so style modulation and temporal reconstruction were mismatched in resolution. Removing the progressive AdaIN raises WER/CER from 8.35%/4.04% to 9.06%/4.42%, lowers SECS from 0.80 to 0.77 and raises PPG from 0.36 to 0.39.

Perspective

The demonstrated scope is English accent conversion: the target-accent pool is built offline from 25 minutes of CMU-ARCTIC American English recordings and held fixed across all conversions, source speech comes from L2-ARCTIC speakers ASI and ABA and VCTK speaker P248, objective metrics cover the full corpora of those three speakers, and the subjective MOS comes from 20 listeners rating 10 randomly selected utterances per test set.

The authors explicitly do not claim the WavLM layer-6/layer-23 pair is universally optimal, only a principled choice supported by layer-wise probing; the quality-function weights are empirically set and not claimed globally optimal. In the reverse American-to-other direction, PPG distances are generally higher than in the other-to-American direction, and the authors suggest the decoder and vocoder trained on American English may introduce an acoustic prior as one possible cause. Future work points to cross-lingual scenarios and low-latency streaming conversion.