Semantic data augmentation for meaning maintenance on Task-Oriented Conversation with Large-size Language Model (2023)

Abstract: This paper presents our approach to building a generalized model for Track 5 in DSTC11: “Task-oriented Conversational Modeling with Subjective Knowledge” which addresses the challenge of generating responses to users’ utterances based on a variety of factual and subjective knowledge. To tackle this challenge, we first augmented the training data by leveraging contextual word embedding and back translation, thereby increasing the quantity of available data. Then, we utilized a large-size language model to enhance the acceptability of the augmented data and fine-tuned the model using augmented data. Specifically, we applied the DeBERTa-v3-large model for knowledge detection and selection, and the BART-large model for response generation. Our best model achieved the seventh rank in the objective evaluation and the second rank in the final official human evaluation. These outcomes serve as solid evidence that data augmentation and using a large-size model were highly effective for developing a conversational model system that incorporates objective and subjective knowledge.

## introduction over the past decade, ai technology has brought about many changes and has been used in various applications and devices that permeate our lives, such as chatbots, personal assistants, and smart kiosks. traditional studies on task-oriented dialogue systems have focused primarily on providing information and performing actions limited to specific databases or application programming interfaces (api). however, recent studies have focused on tasks beyond available apis and databases, incorporating relevant domain knowledge. knowledge-grounded task-oriented conversational modeling tasks are being introduced to task #1: knowledge-seeking turn detection task#2: knowledge selection task#3: knowledge grounded generation unstructured domain/entity knowledge deberta-v3-large bart-large original dataset semantic data augmentation contextual word embedding roberta-large dialogue history system response back translation fairseq wmt19 transformer augmented dataset fine-tuning ## method we try to improve the model in three ways based on the baseline code and dataset of dstc11 track 5 @xcite . first, we employ augmentation techniques, such as contextual word embedding and back translation, to enrich the training data. second, we test various large-size language models and fine-tune the model using augmented data. finally, we implement the contrastive search method for response generation to our model and check the feasibility. ## experiments and results although not all cases were tested, the performance was the best when the value of penalty alpha was 0.4 and the top-k value was 4. therefore, those values were used in subsequent tests related to the contrastive search. ## conclusion in dstc11 track 5, we applied data augmentation methods that retain meaning and used a large-size language model. as a result, we obtained the seventh place in the objective evaluation and second place in the final human evaluation. through various and gradual experimental innovations in the designs, it was found that data augmentation using contextual word embedding and back-translation technique play a major role in improving performance. using the deberta-v3-large and bartlarge language models was also crucial. it was also found that ontology-based augmentation and contrastive search were not very helpful for this task.