The Future of Designing Spoken Dialogue Systems and Analyzing Written Conversations (2023)

Abstract: This is my position paper for YRRSDS 2023. In it, I write about the details of my research interests as well as past, current and future projects, talk about the status of spoken dialogue system research, include a short bio, and suggest topics for discussion.

## research interests my research interests lie generally in the area of natural language processing (nlp), text-to-speech (tts) and automatic speech recognition (asr), with a special focus on dialogue modeling with both supervised and unsupervised learning. with respect to spoken dialogues, i focus on creating representations that can model prosody, turn shifts, pauses, backchannels and improve the accuracy of twochannel speech generation, as well as solve downstream tasks like emotion and laughter detection. i am also interested in online asr and speech synthesis for dialogues. with respect to written dialogues, i aim to improve spoken dialogue representations with text and other modalities through multimodal learning, as is common in asr. i also try to improve language models using other modalities like speech, among others (e.g., vision and motion), and solve less-researched dialogue tasks using language models (like reference resolution (rr) and coreference resolution (crr)). ## spoken dialogue system (sds) research where do you think the field of dialogue research will be in 5 to 10 years? dialogue research is becoming more and more important as tts, speech-to-text (stt) and language models work fairly well on monologues, continuous text and short snippets (e.g., sentences) but do not usually take into account data more complex that these. dialogues can also provide context that is often missing from monologues (e.g., disambiguation of terms and affirmation) which can aid in better response generation when it comes to speech synthesis, for example. what i think will improve in the upcoming years is the connection between content and prosody as well as other non-verbal cues. what do you think this generation of young researchers could accomplish in that time? young researchers can improve the fine-grained details to make conversations more human-like, with respect to both content and the naturalness of non-verbal signs. another thing that is worth looking into is how to make use of long-term dependencies (similarly to how language models do) and representing dialogue history (previous dialogue sessions) by compressing previous information. what kind of questions need to be investigated to get the field to that point? i think it is really important to create high-quality representations similar to wav2vec and hubert but for dialogues and dialogue context, either with the inclusion of prosody or separately from it. what are the most important things for users of sdss? natural-sounding speech, informativeness and correctness of generated responses, identifying turn shifts in online systems, knowledge and visual grounding, correct reaction to human emotions, multimodal cues. is there a difference between sds research in academia and industry? industry and commissioned research are more profit-oriented than general academic research, and as such, they focus on hands-on and immediate applications. this narrows down the possible research questions, creates the need for patents and ndas and requires system integration (e.g., in video games). will sdss be more widely used in the future? how? in what scenarios? as tts and asr are considered to be solved for many use-cases, the areas where sds can be used becomes more and more niche. nevertheless, as mentioned before, the dialogue system of robots could greatly benefit from this field, as well as interactive gps systems, general-use chatbots, personal assistants, video game scripts and video transcriptions (especially for interviews and movie subtitles).