TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data

Although there has been significant advancement in the field of speech-to-speech translation, conventional models still require language-parallel speech data between the source and target languages for training. In this paper, we introduce TranSentence, a novel speech-to-speech translation without language-parallel speech data. To achieve this, we first adopt a language-agnostic sentence-level speech encoding that captures the semantic information of speech, irrespective of the language. We then train our model to generate speech based on the encoded embeddings obtained from a language-agnostic sentence-level speech encoder that is pre-trained with diverse languages. With this method, despite training exclusively on the target language's monolingual data, we can generate target language speech in the inference stage using language-agnostic speech embeddings from the source language speech. Furthermore, we extend TranSentence to multilingual speech-to-speech translation. The experimental results demonstrate that TranSentence is superior to other models.

Official implementation of TranSentence: https://github.com/sb-kim-prml/TranSentence_code

TranSentence was trained on the Common Voice 11 dataset and evaluated on the CVSS-C dataset. We used single-speaker unit vocoder for unit-to-waveform conversion.