HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer

Despite rapid progress in the voice style transfer (VST) field, recent zero-shot VST systems still lack the ability to transfer the voice style of a novel speaker. In this paper, we present HierVST, a hierarchical adaptive end-to-end zero-shot VST model. Without any text transcripts, we only use the speech dataset to train the model by utilizing hierarchical variational inference and self-supervised representation. In addition, we adopt a hierarchical adaptive generator that generates the pitch representation and waveform audio sequentially. Moreover, we utilize unconditional generation to improve the speaker-relative acoustic capacity in the acoustic representation. With a hierarchical adaptive structure, the model can adapt to a novel voice style and convert speech progressively. The experimental results demonstrate that our method outperforms other VST models in zero-shot VST scenarios.

 

We compare HierVST with several Voice Conversion models as:

1. AutoVC [K. Qian et al., 2019] : [Official Demo page]

2. VoiceMixer [S.-H. Lee et al., 2021]: [Official Demo page]

3. DiffVC [V. Popov et al., 2022]: [Official Demo page]

4. Speech Resynthesis [A. Polyak et al., 2021]: [Official Demo page]

5. YourTTS [E. Casanova et al., 2022]: [Official Demo page]