SLDT : Sequential Latent Document Transformer for Multilingual Document-based Dialogue (2023)

Abstract: Multilingual document-grounded dialogue, where the system is required to generate responses based on both the conversation Multilingual context and external knowledge sources. Traditional pipeline methods for knowledge identification and response generation, while effective in certain scenarios, suffer from error propagation issues and fail to capture the interdependence between these two sub-tasks. To overcome these challenges, we propose the application of the SLDT method, which treats passage-knowledge selection as a sequential decision process rather than a single-step decision process. We achieved winner 3rd in dialdoc 2023 and we also validated the effectiveness of our method on other datasets. The ablation experiment also shows that our method significantly improves the basic model compared to other methods.

## introduction the advancements in neural models and the development of large-scale dialogue datasets have significantly propelled dialog generation research @xcite @xcite . open-domain dialogue systems strive to produce more informative and fluent responses @xcite @xcite , finding applications in a wide array of areas such as emotional companionship, mental health support, and social chatbots. despite demonstrating promising results, most existing dialogue generation systems @xcite @xcite depend on substantial data resources. in real-world scenarios, dialogue corpora for many languages are not readily available, thereby restricting the applicability of dialogue systems for low-resource or even zeroresource languages. consequently, it is crucial to develop methods capable of effectively transferring knowledge from a source language with ample resources to a target language. one such task is multilingual documentgrounded dialogue @xcite , where the system is required to generate responses based on both the conversation multilingual context and external knowledge sources, such as documents or databases @xcite . while various methods have been proposed to address the challenges of knowledge selection and response generation in this task @xcite , including sequential latent knowledge selection for document-grounded dialogue. there is a need for a novel approach that combines the advantages of these methods @xcite . in this paper, we propose a new method to address the problem of document dialogue by employing the sequential latent document transformer (sldt) to select the most relevant knowledge for conversation from a multilingual document set. the motivation behind focusing on multilingual document-grounded dialogue lies in its potential to provide more informative and engaging responses by leveraging external knowledge sources @xcite , thereby enabling the dialogue system to better assist users in satisfying their diverse information needs. traditional pipeline methods for knowledge identification and response generation, while effective in certain scenarios, suffer from error propagation issues and fail to capture the interdependence between these two sub-tasks. to overcome these challenges, we propose the application of the sldt method, which has shown promising results in knowledgegrounded dialogue, to the task of document dialogue. the use of sldt in document conversations is expected to bring several advantages, such as better modeling the diversity in document-knowledge selection, more accurate leveraging of response information, and the ability to work even when technologie -wikipédia paléolithique les plus anciens outils de pierre connus, regroupés sous le nom de pré-oldowayen ou d'oldowayen, datent d'il y a 2,3 millions d'années. kfr： multilingual documents monolingual document-grounded dialogue user:quelles sont les étapes d'évolution du néolithique à l'antiquité classique ? bot:une progression continuelle et qui amènera ultérieurement par exemple, au fourneau, et à sa ventilation, a fourni la capacité à fondre, et à forger, d'abord les métaux les plus accessibles. user:quels sont les plus anciens outils en pierre connus ? bot:les plus anciens outils de pierre connus, regroupés sous le nom de pré-oldowayen ou d'oldowayen, datent d'il y a 2,3 millions d'années. x fr ： y fr ： multilingual document-grounded dialogue user:quelles sont les étapes d'évolution du néolithique à l'antiquité classique ? bot:une progression continuelle et qui amènera ultérieurement par exemple, au fourneau, et à sa ventilation, a fourni la capacité à fondre, et à forger, d'abord les métaux les plus accessibles. user:quels sont les plus anciens outils en pierre connus ? bot:les plus anciens outils de pierre connus, regroupés sous le nom de pré-oldowayen ou d'oldowayen, datent d'il y a 2,3 millions d'années. x fr ： y fr ： bot: một số nhỏ bằng chứng trong nghiên cứu y khoa ủng hộ ý tưởng rằng kiểm tra thể chất trước khi đi làm thực sự có thể làm giảm sự vắng mặt, chấn thương tại nơi làm việc và bệnh nghề nghiệp có chất lượng thấp. user: một số nhỏ bằng chứng chất lượng thấp trong nghiên cứu y khoa ủng hộ ý tưởng gì? bot: một số nhỏ bằng chứng chất lượng thấp trong nghiên cứu y khoa ủng hộ ý tưởng rằng kiểm tra thể chất trước khi đi làm thực sự có thể làm giảm sự vắng mặt, chấn thương tại nơi làm việc và bệnh nghề nghiệp. x vi ： y vi ： khám sức khỏe -wikipedia tiếng việt khám sức khỏe trước tuyển dụng một số nhỏ bằng chứng chất lượng thấp trong nghiên cứu y khoa ủng hộ ý tưởng rằng kiểm tra thể chất trước khi đi làm thực sự có thể làm giảm sự vắng mặt, chấn thương tại nơi làm việc và bệnh nghề nghiệp. kvi： figure 1: introduction of multilingual document-grounded dialogue. our method achieved excellent results on dialdoc 2023 share task. obtained 208 points in online testing. we also validated the effectiveness of our method on other datasets. the ablation experiment also shows that our method significantly improves the basic model compared to other methods. ## related works document-grounded dialogue systems (dgds) categorize unstructured, semi-structured, and structured data in documents to facilitate the comprehension of human knowledge and interactions, thus fostering more natural human-computer interac-tions (hci) @xcite . the objective of dgds is to generate conversational modes based on information (utterances, turns, context, clarification) supplied by a document or documents @xcite . dgds are particularly advantageous in task-oriented and goal-oriented settings as they replicate the natural dialogue flow. a recent example of dgds, closely related to our work, is doc2dial, a multi-domain dgds dataset designed for goal-oriented dialogue that models hypothetical dialogue flows and scenes to simulate authentic interactions between a user and a machine agent in information-seeking contexts @xcite . in our proposed task, we adopt a similar approach, but we also permit users to pose clarification questions, the responses to which may not be directly grounded in the document. this aspect is crucial in the development of instruction-giving conversational agents, as the dialogue pipeline requires increased flexibility, as previously mentioned. multilingual dialogue tasks typically utilize a code-switching approach to achieve semantic alignment between various languages @xcite @xcite . this method of code-switching enables implicit semantic alignment without the need for parallel corpus pairs. drawing inspiration from these studies, we apply the code-switching technique to transfer knowledge from english dialogue history to other target languages lacking training examples. in line with previous work @xcite on multilingual representation, we implement code-switching at the utterance level, although code-switching at the word or span level is more prevalent @xcite @xcite . ## methodology we utilize xlm-r @xcite as our retrieval model, employing a representation-based bi-encoder consisting of a dialogue query encoder, denoted as q(•), and a passage context encoder, represented by p(•). for a given input query q and a set of passages {p i } m @xmath0, the encodings for the query and passage are computed as q(q) and p (p i ), respectively. the similarity between these encodings is determined by the dot product ⟨q(q), p (p i )⟩, with the model being trained to minimize the negative log likelihood of the correct passage among l in-batch and challenging negatives. subsequently, we pre-calculate the representations for all passages and index them offline. during inference, the top-k passages are retrieved using maximum inner product search (mips) in conjunction with faiss. we introduce a sequential latent document transformer tailored for multilingual documentbased dialogue, as illustrated in figure 2 . the objective of the model is to generate customized and informative responses by learning a probabilistic model p(r|c, k, p) that leverages passageknowledge and context flowing @xcite . we proceed by iterating through dialogue turns with 1 ≤ @xmath1, iterating over words in the utterances of the apprentice and wizard using 1 ≤ @xmath2, and denoting knowledge sentences in the pool with 1 ≤ @xmath3. here, t represents the dialogue length, m and n correspond to the lengths of the apprentice and wizard's utterances, and l denotes the passage-knowledge pool size. the input to the sldt at turn t comprises previous conversation turns, which include user utter-ances x 1 , ..., x t , system responses y 1 , ..., y t-1 , and the passage pool k 1 , ..., k t , where k @xmath4,l } = k t,1 , ..., k t,l . the model's output consists of the chosen sample passage-knowledge k t s and the response y t . we provide an in-depth explanation of sentence embedding, passage-knowledge selection, and utterance decoding. first, we consider passage-knowledge selection a sequential rather than a one-step decision-making process. due to the diversity of passage-knowledge selection in dialogue, we model it with latent variables. therefore, we can conduct joint inference for multiple turns of passage-knowledge selection and response generation, as opposed to distinct inference on a turn-by-turn basis. various studies have been conducted on sequential latent variable models. for instance, some have proposed a posterior attention model that represents the attention mechanism in seq2seq models as sequential latent variables. drawing inspiration from these works, we factorize response generation with latent document passage-knowledge selection and derive the variational lower bound as follows. the conditional probability of generating response y t given dialogue context @xmath5: eq.( 1 ) means that we first infer from the knowledge posterior which knowledge would be used up to previous turn t -1, estimate the knowledge for current turn t from prior knowledge distribution and generate an utterance from the inferred knowledge. figure 2 shows an example of this generation process at @xmath6. we parameterize the decoder network p γ , the prior distribution of knowledge π γ , and the approximate posterior q ψ with deep neural networks as will be discussed. ## experiments for the retrieval training stage, we utilized a batch size of 128 and a learning rate of 1e-4 and 5e-5 for post-training and fine-tuning, respectively. and retrieval passage number top-k is 25. during the generation stage, we used a batch size of 32 with a learning rate of 1e-4 and 1e-5 for post-training and fine-tuning, respectively. for r-drop, we set the dropout rate to 0.1, and the kl-divergence loss weight 0.02. ## conclusion in this paper, we present a novel sldt method for multilingual document-grounded dialogue , with a focus on addressing the challenges of selecting the most relevant documents for conversation and generating informative responses based on the selected knowledge. we then present an extensive experimental evaluation of our method, demonstrating its effectiveness in comparison to existing approaches.