Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues (2023)

Abstract: Robust state tracking for task-oriented dialogue systems currently remains restricted to a few popular languages. This paper shows that given a large-scale dialogue data set in one language, we can automatically produce an effective semantic parser for other languages using machine translation. We propose automatic translation of dialogue datasets with alignment to ensure faithful translation of slot values and eliminate costly human supervision used in previous benchmarks. We also propose a new contextual semantic parsing model, which encodes the formal slots and values, and only the last agent and user utterances. We show that the succinct representation reduces the compounding effect of translation errors, without harming the accuracy in practice. We evaluate our approach on several dialogue state tracking benchmarks. On RiSAWOZ, CrossWOZ, CrossWOZ-EN, and MultiWOZ-ZH datasets we improve the state of the art by 11%, 17%, 20%, and 0.3% in joint goal accuracy. We present a comprehensive error analysis for all three datasets showing erroneous annotations can lead to misguided judgments on the quality of the model. Finally, we present RiSAWOZ English and German datasets, created using our translation methodology. On these datasets, accuracy is within 11% of the original showing that high-accuracy multilingual dialogue datasets are possible without relying on expensive human annotations. We release our datasets and software open source.

## introduction tremendous effort has gone into the research and development of task-oriented dialogue agents for english and a few other major languages in recent years. a methodology that can transfer the effort to 1 https://github.com/stanford-oval/ dialogues other languages automatically will greatly benefit the large population of speakers of the many other languages in the world. underlying an effective tod agent is dialogue state tracking, the task of predicting a formal representation of the conversation sufficient for the dialogue agent to reply, in the form of slots and values. however, dst currently remains restricted to a few popular languages @xcite . traditional dst agents require large handannotated wizard-of-oz @xcite datasets for training, which are prohibitively labor-intensive to produce in most languages @xcite . large, multi-domain woz datasets are only available in english and chinese @xcite . the contributions of this paper are as follows: machine translation has been shown effective for localizing question-answering agents @xcite . it shows that for open ontology datasets, we need to use an alignment model to properly translate entities in the source language to entities in the target language. this paper shows that alignment is necessary even for closed ontology datasets and dialogues. furthermore, we improve alignment to address these challenging issues we discovered unique to dialogues: (1) translation errors accumulate and can prevent a correct parse for the rest of the dialogue; (2) there are logical dependencies between slot values across different turns; (3) utterances are generally longer and more complex carrying multiple entities. we found that alignment improves the accuracy on the risawoz benchmark by 45.6%. this technique eliminates the cost of human postediting used on all previous translation benchmarks, and can improve machine translation quality on other tasks too. using this methodology, we automatically trans-late the risawoz dataset to english and german, creating risawoz-en-auto and risawoz-deauto datasets respectively. bart-csp improves sota on ri-sawoz @xcite and crosswoz @xcite , two large-scale multi-domain woz dialogue datasets, by 10.7% and 17% in joint goal accuracy(jga). notably, bart-csp is more effective on translated data as evident by bigger performance improvement: on risawoz-en-auto and risawoz-de-auto datasets, automatically translated versions of risawoz, bart-csp improves sota by 32.4% and 52.5%. ## task setting we are interested in the dialogue state tracking task, in which the goal is to predict a formal representation of a conversation up to a certain point, also known as belief state, consisting of the slots that were mentioned in the dialogue and their value. at the beginning of the conversation, the belief state is empty, and it grows as the conversation progresses, accumulating the slots that were mentioned across all turns prior. formally, the problem is formulated given a predefined set of slots s 1 , s 2 , . . . s n (such as "restaurant name", "restaurant food", etc.). each slot has one value taken from the ontology v 1 , v 2 , . . . , v n . the ontology contains the legitimate values for the slot from the database (i.e. the list of restaurant names or restaurant cuisines), as well as the special values "none" indicating the slot was not mentioned, and "dontcare" indicating the slot was explicitly mentioned by the user but the user has csp input dst history input null ---can you help me find a place in the north where i can stay ? hotel @xmath0. when is your stay ? do any of them have a 4 star rating ? hotel @xmath1, actually they all have 4 star ratings , do you need parking ? yes , i would like to book 1 that offers parking please hotel @xmath2: visualization of the differences between inputs of csp and traditional dst models. while the latter encodes the full history of the dialogue, csp only encodes the current state and turn. b, a, and u indicate the dialogue state, agent utterance, and user utterance, respectively. t is the turn number. ˆindicates a predicted dialogue state. ## reducing translation noise key to the success of the dialogue state tracking is a precise and consistent annotation of the belief state at every turn. this is challenging in the multilingual setting, where we apply automatic translation to produce new training sets for dst: (1) the translation can be ungrammatical or incorrect, and it can introduce spurious information, and (2) the new annotation (in the target, translated ontology) can diverge from the translation of the sentence, such as referring to the same value in different ways in the target language. we refer to the error introduced in translation collectively as translation noise. we posit that translation noise is the reason for poorer performance in existing translated dialogue datasets, compared to the same model on the dialogue dataset in the source language. in this section, we describe our methods to reduce translation noise. ## experiments our experiments are designed to answer the following questions: 1) how well does csp perform on woz datasets compared to dst models that encode the full conversation history? 2) is our approach effective to reduce the translation noise? ## analysis and results as shown in furthermore, models that rely on predefined ontologies require changes in architecture for new datasets. on the other hand, bart-csp is a generative model that learns to copy slots from context and can be deployed for a new dataset as is. on multiwoz, we report results for two models: mintl which uses bart-large, and star foot_0 which uses bart as the context encoder. our model outperforms mintl despite using a smaller bart model, and achieves similar performance to star despite not having access to the full history. this shows bigger models do not necessarily yield better performance and model architecture and data representation are important too. ## data and error analysis a manual inspection revealed the following sources of errors on the csp model, showing some of the inference limitations and its susceptibility to misannotations. ## conclusion given a dialogue dataset in one language, this paper shows how to build contextual semantic parsers for a new language using automatically machinetranslated data. we propose an improved alignment approach for dialogues to ensure faithful translation of slot values. this removes the need for costly human-post editing used in all previous benchmarks. we show that the compounding effects of translation noise across turns can be mitigated with a csp model for dialogue state tracking. by leveraging pretrained seq2seq models such as bart, training with csp can outperform state-of-the-art results on risawoz, crosswoz, and multiwoz-zh, and remains competitive on multiwoz, despite not encoding any previous conversation turns or having access to a predefined ontology. we use our methodology to create risawoz english and german, the first automatically created high-quality translated datasets for dialogue state tracking with no human in the loop. we have implemented our methodology as a toolkit foot_1 which developers can use to create a new multilingual dialogue dataset as well as a contextual semantic parser for it.