Exploring Back Translation with Typo Noise for Enhanced Inquiry Understanding in Task-Oriented Dialogue (2023)

Abstract: This paper presents our approach to the DSTC11 Track 5 selection task, which focuses on retrieving appropriate natural language knowledge sources for task-oriented dialogue. We propose typologically diverse back-translation method with typo noise, which could generate various structured user inquries. Through our noised back translation, we augmented inquiries by combining three different typologies of language sources with five different typo noise injections. Our experiments demonstrate that typological variety and typo noise aids the model in generalizing to diverse user inquiries in dialogue. In the competition, where 14 teams participated, our approach achieved the 5th rank for exact matching metric.

## introduction task-oriented dialogue (tod) systems have been developed to offer information and perform specific tasks such as making reservations. in general, tod systems depend on domain-specific apis and pre-defined databases (db), limiting their ability to handle a wide range of scenarios. @xcite . in order to enhance their capabilities and offer more valuable assistance, recent studies have incorporated unstructured textual information from the web into a dialogue system @xcite @xcite . however, in order to effectively integrate unstructured information, a more advanced comprehension of the user's utterances is needed. for example, when a user mentions having a backache and asks for hotel recommendations, the model should accurately comprehend the user's core intentions (e.g., "need a comfortable bed") and extract relevant information from the hotel review set * these authors contributed equally to this work. rather than rely only on a pre-defined db. by precisely understanding the user's question and leveraging others' comments, the model can respond with up-to-date information and diverse perspectives. unfortunately, precise methods for effectively utilizing these knowledge sources have not been thoroughly researched yet in tod systems. dstc11-track 5 selection task @xcite was introduced to address this challenge. unlike previous tod approaches that rely on a pre-defined db, this task offers an external natural language form knowledge source that contains reviews and q&as. participants are required to devise approaches that select the relevant information from the provided knowledge source. this track's objective is to improve the system's capacity to accurately address user inquiries by integrating external sources, going beyond the structured information. even if a user is asking a commonly asked question, the inquiry form can vary in their words or sentence structure, highlighting the importance of comprehending the core intent behind the user's utterance. to address this, a larger training dataset is essential to ensure robustness and generalize to a diverse form of inquiries. to tackle this challenge, we augment user inquiries by leveraging the back translation, which has been widely used for low-resource problems @xcite @xcite . however, previous studies have shown that machine translations may lack diversity compared to a human-generated dataset @xcite @xcite . to overcome this limitation, we conducted back translation across various typological languages, including fusional, isolating, and agglutinative languages. in total, we carefully selected 12 distinct languages for the back translation. another effective approach to generating diverse back-translated sentences involves introduc-ing noise into the translated data @xcite . moreover, applying typo errors to humanwritten datasets has been improve robustness across various tasks @xcite @xcite . building on these previous achievements, we propose adopting typo noises to the back-translated dataset to enhance the stability and performance when dealing with user-written inquiries. during the experiment, we compared our performance to the baseline and achieved an exact matching score of 0.448 using our augmentation methods, which surpassed the average score for this task. furthermore, in our analysis, we extensively examined the impact of typological diversity and typo noise in the back translation process. in the competition, our method achieved the 5th position in the exact matching score among the 14 participating teams. ## related work back translation was initially introduced to enhance neural machine translation performance through augmented monolingual datasets @xcite and has proven effective when language resources are limited. subsequently, various methods have been developed to enhance the augmented dataset. for the training method, iterative back translation method leverages two translation models and continues the training of each model with an augmented dataset until no further improvement is observed in both directions @xcite . alternatively, some researchers @xcite focus on selecting among the translated datasets based on model confidence scores to ensure dataset quality. similarly, distinguishing between back-translated and original datasets by adding tags to the training set @xcite and assigning weights based on the quality of translated sentences @xcite ) also helps in using the back translation method with the original dataset. in the context of our study, edunov et al. ( 2018 ) is particularly relevant. they enhanced back translation by incorporating noise during decoding with beam search, improving the machine translation performance. we, too, introduce noise to our backtranslated data, but in our case, we adopted a typo as the injecting noise method. many natural language applications such as re-triever @xcite , optical character recognition (ocr) @xcite , and language model pre-training @xcite have dealt with typos to impart robustness to a human-generated dataset. by injecting typo noise into the user inquiries, we also aim to be more stable to some erroneous conversation inquiries. ## dstc11-track5 in the dstc11-track5 selection task, participants are required to retrieve relevant knowledge from a provided review dataset in response to a user's utterance. the dialogue dataset consists of 32,604 dialogue data instances and 244,032 turns, which contain the system and user's utterances. the dialogue dataset is a revised version of multi-woz 2.1 @xcite , by specifically modifying several turns to be necessary subjective and factual knowledge. the knowledge data comprises 143 entities, representing individual hotels or restaurants. in total, there are 4,299 reviews and q&as available for these entities. it is worth noting that this knowledge data exhibits several unique characteristics compared to previous challenges. first, a single review can encompass multiple aspects. second, there are multiple review posts from diverse individuals, which may include conflicting opinions. during the training and testing of the selection model, participants are provided with user and system dialogue that includes the entity relevant index. participants can use this entity index when retrieving the knowledge set. for additional guidance, participants can access the baseline and evaluation scripts on the website. foot_0 . ## method we now introduce our back translation method with typo noise. we denote the conversation dataset as d, and i-th dialgoue in d is denoted as