STOR i C o: Storytelling TTS for H indi with Character Voice Modulation (2024)

Abstract: We present a new Hindi text-to-speech (TTS) dataset and demonstrate its utility for the expressive synthesis of children’s audio stories. The dataset comprises narration by a single female speaker who modifies her voice to produce different story characters. Annotation for dialogue identification, character labelling, and character attribution are provided, all of which are expected to facilitate the learning of character voice and speaking styles. Experiments are conducted using different versions of the annotated dataset that enable training a multi-speaker TTS model on the single-speaker data. Subjective tests show that the multi-speaker model improves expressiveness and character voice consistency compared to the baseline single-speaker TTS. With the multi-speaker model, objective evaluations show comparable word error rates, better speaker voice consistency, and higher correlations with ground-truth emotion attributes. We release a new 16.8 hours storytelling speech dataset in Hindi and propose effective solutions for expressive TTS with narrator voice modulation and character voice consistency.

## introduction speech synthesis has considerably evolved over the last few years, going beyond the goal of achieving understandable and natural speech. it now includes aspects such as expressiveness and other notable qualities of the desired speaking style. english has well-established speech datasets for read speech, such as ljspeech @xcite , m-ailabs (gmbh, 2019), blizzard 2013 @xcite , and the recently released storynory @xcite . while the blizzard 2013 and storynory datasets include expressive storytelling speech for children, there is a lack of such expressive tts datasets for hindi. hindi is the third most spoken language in the world after english and mandarin. although there are hindi tts datasets available, such as those released by indic tts @xcite and syspin foot_0 , these datasets primarily consist of neutral-toned read speech and lack expressiveness. however, storytelling for children involves more interactive and conversational speech. in storytelling speech, voice modulation by the storyteller for different characters is key to consider. using appropriate voices for dialogue and maintaining consistency in character voices are crucial in storytelling. children's stories can have characters of different species, ages, genders, and giving each character a distinct voice helps keep children engaged. previous works, like @xcite , aim to predict character voice for a text-to-speech (tts) system in storytelling. however, they only focus on retrieving the correct speaker voice from a given set of audio based on textual descriptions. xin et al. (2023) explores improving speech synthesis for audiobooks by considering acoustic and textual contexts. the study uses a multi-speaker japanese audiobook tts dataset @xcite , different from single-speaker storytelling speech. @xcite explore character acting in japanese audiobooks by predicting characterappropriate voices using character embeddings derived from the character's name, sentences and surrounding characters. however, the synthesized audio sample lacks expressiveness and does not evaluate the character's voice consistency when attempting to mimic the character's voice. @xcite focuses on synthesizing rakugo speech, a form of comic storytelling that only includes character dialogues. the authors create a database and annotate character descriptions based on the conversation. however, in storytelling speech, the character descriptions come from the stories themselves. moreover, storytelling end-to-end tts models, such as vae @xcite and gan-based models @xcite , have demonstrated the ability to generate high-quality speech using phonemes and audio as input. while many tts models can produce speech comparable to human speech, models utilizing gan and normalizing flows @xcite have shown improved expressiveness @xcite . @xcite analyses various kinds of neural tts for indian languages. due to its competitive performance for indian languages, we use vits tts @xcite in a multi-speaker setting. our contributions are a) a new, expressive single-speaker hindi storytelling tts dataset annotated with character information from the stories. the dataset is unique as the speaker modulates her voice for different characters within a story. b) a hindi storytelling tts system with the ability to modulate the voice according to different characters of the story while still maintaining character voice consistency within a story. ## dataset a youtube channel called storico foot_1 offers audio stories in hindi for children aged 7-12. the stories are collected from the internet, recorded specifically for kids, and narrated by a female speaker in hindi. the narrator enacts different characters by using multiple voices. the audio recordings include background music and hindi salutations at the beginning and end of each story. we sought permission from the channel owner to scrape her audio stories from youtube, with overall 169 stories totaling 19.5 hours. each story has an average duration of 7.3 minutes and a standard deviation of 3 minutes. the stories were segmented into 10-12 second clips based on silence in the audio. the clips contain background music, animal sounds, etc. but the speaker's speech is clear. the segmented clips were denoised using meta's denoiser @xcite . to ensure quality, 100 random clips were manually verified. all segmented clips were then processed through nvidia asr to obtain synthetic transcripts, which were manually corrected and annotated for speaker details. ## methodology the tts dataset is unique in two ways. firstly, it is the first expressive tts dataset for hindi that includes speaker information along with the transcript. the speaker information goes beyond just names and includes other details that affect the voice of the characters in the story. secondly, although the stories were narrated by a single speaker, she modulates her voice for different characters within each story. these characters have different characteristics such as age, gender, and species. however, the speaker can only modulate her voice for a limited number of voice types. neural speaker identification models treat this modulation as different speakers. we passed the speech of all character instances and randomly sampled 1000 narrator instances, across the stories, through a speaker encoder @xcite . the resulting speaker embeddings were visualized using tsne (fig. 1 ), and the plot revealed 4 clusters representing different kinds of voices. we also attempted to identify 3 our annotated dataset will be released upon publication. samples can be found at https://tinyurl.com/4zfxkmxj . a clustering based on the speaker descriptions using indicbert @xcite , but no such clustering was observed (fig. 2 ). to address narrator voice modulation and character voice consistency in storytelling speech synthesis, we trained a multi-speaker speech synthesis model on single-speaker data. this approach involved using k-means clustering on speaker embeddings to obtain cluster labels, which were then used as speaker labels in the training data for the textto-speech (tts) model. vits tts was utilized in a multi-speaker setting, with speaker embedding incorporated for conditioning. we fine-tuned in-dicbert to predict one of the 4 cluster labels. the fine-tuned model achieved 75% accuracy on the test set in predicting cluster labels and improved text embedding clustering compared to the pretrained model embeddings (figure 2 ). global conditioning is applied to incorporate the speaker embedding similar to @xcite . since hindi is a phonetic language, unlike english, we directly use graphemes instead of phoneme sequences as input to the text encoder.