Language-Agnostic Transformers and Assessing C hat GPT -Based Query Rewriting for Multilingual Document-Grounded QA (2023)

Abstract: The DialDoc 2023 shared task has expanded the document-grounded dialogue task to encompass multiple languages, despite having limited annotated data. This paper assesses the effectiveness of both language-agnostic and language-aware paradigms for multilingual pre-trained transformer models in a bi-encoder-based dense passage retriever (DPR), concluding that the language-agnostic approach is superior. Additionally, the study investigates the impact of query rewriting techniques using large language models, such as ChatGPT, on multilingual, document-grounded question-answering systems. The experiments conducted demonstrate that, for the examples examined, query rewriting does not enhance performance compared to the original queries. This failure is due to topic switching in final dialogue turns and irrelevant topics being considered for query rewriting.

## introduction english dominates as the most widely used language on the internet, and for communicating with virtual assistants 1 . however, the prevalence of english-centric content creates a language barrier for non-english speakers who wish to access information and services online. to bridge this gap, there is a growing need for multilingual knowledge-grounded question-answering dialogue systems that can enable individuals to access the internet and utilize virtual assistants in their native language. while the development of english document-grounded dialogue systems @xcite has been extensively explored, the exploration of other languages remains limited. in response to this, dialdoc 2023 shared task extends the task of document-grounded dialogue to include multiple languages with limited annotated 1 usage statistics and market share of content languages for websites, february 2023 -w3techs.com. the mdpr model employs a bi-encoder architecture, utilizing a pre-trained multilingual model to encode the questions and passages independently. the encoded representations are then compared using a maximum inner product search to identify relevant passages for a given question. in this study, we evaluate two paradigms for multilingual pre-trained transformer models as mdpr bi-encoders, namely, a language-agnostic paradigm and a language-aware paradigm. specifically, we consider two models for multilingual sentence embedding: language-agnostic bert sentence embedding (labse) @xcite and xlm-roberta (xlm-r) @xcite . labse combines masked language modeling with translation language modeling to produce language-agnostic sentence embeddings, while xlm-r is a cross-lingual version of roberta @xcite pre-trained on a large corpus of text in over 100 languages using a self-supervised approach. although both models are beneficial for multilingual sentence embeddings, based on our experiments, it has been observed that labse outperforms xlm-r. additionally, we examine the impact of query rewriting techniques using large language models (llms) such as chatgpt to summarize the conversational history more concisely and use transfer learning to generalize to french and vietnamese rewritten queries. therefore, in this study, we investigate the performance difference between the language-aware and language-agnostic paradigms, where we found that the language-agnostic labse retriever outperforms the language-aware xlm-r retriever. additionally, we explore the impact of query rewriting on the performance of such systems. while query rewriting has been proposed as a potential solution for improving performance, our results indicate that rewriting queries did not significantly improve performance for the considered sub-samples. our code is available on github foot_0 . ## dataset dialdoc 2023 shared task dataset consists of 797 vietnamese dialogues with an average turn count of 4 and 816 french dialogues with an average of 5 turns. these dialogues are grounded in multiple documents from nine different domains, namely technology, health, health care services, veterans affairs, insurance, public services, social security, department of motor vehicles, and student financial aid in the usa. each dialogue turn in the dataset contains role annotations for the conversation between a human and a conversational agent, with the turns in reverse chronological order, the latest turn first in dialogue history. the retrieval dataset includes query, dialogue data, positive passages, and negative passages. positive passages contain the answer to the given query and are within the document, while negative passages are closely related to the document but they do not contain the answer to the query in focus. ## methodology and experiments the prevailing paradigm for document-grounded question-answering models involves a retrieverreader approach that comprises of a document retrieval module, a reranker module, and an answer generation module. however, in this study, our main focus has been on the multilingual retriever component, while fixing xlm-r as reranker. however, we did experiment with the fusion-in-decoder (fid) approach @xcite to modify the mt5 model previously being used as the answer generator @xcite . ## results and discussion language agnostic retrievers outperform language-aware retrievers. in ## conclusion this paper investigates the effectiveness of language-agnostic and language-aware paradigms for multilingual pre-trained transformer models in a bi-encoder-based dense retriever. the paper also evaluates the impact of query rewriting on task performance. our findings indicate that the languageagnostic approach outperforms the language-aware approach. however, for the considered subsamples, query rewriting did not improve the performance over the original queries. furthermore, the observed topic switching in the conversations' last turns, and chatgpt's tendency to summarize nonrelevant topics and hallucination lead to less accurate rewritten queries compared to the original queries. the limitations of this study are primarily due to budget and credit constraints. consequently, our query rewriting observations are based on a sample size of 2000, leading to limited generalizability of findings. another limitation of the limited resources was the limited context size of chatgpt and the relatively long nature of the questions in our dataset. hence, we could not test prompting chatgpt with in-context examples for better query rewriting performance. hence one potential future work is testing the performance of query rewriting using in-context examples. finally, chatgpt architecture is not open source, preventing us from testing advanced prompting methods. hence another future work would be to test query rewriting using open source models and with advanced finetuning methods like prefix tuning (li and liang, 2021) and prompt tuning @xcite . the study is also subject to the risk of "hallucinations" in chatgpt's responses, which may lead to imprecision in query rewriting. the study suggests further investigation into these issues to improve the accuracy and reliability of the results. we recommend further investigation into these limitations and any potential societal biases present in our dataset to enhance the reliability and performance of query rewriting.