Multi-Task Learning of Query Generation and Classification for Generative Conversational Question Rewriting (2023)

Abstract: In conversational search settings, users ask questions and receive answers as part of a conversation. The ambiguity in the questions is a common challenge, which can be effectively addressed by leveraging contextual information from the conversation history. In this context, determining topic continuity and reformulating questions into well-defined queries are crucial tasks. Previous approaches have typically addressed these tasks either as a classification task in the case of topic continuity or as a text generation task for question reformulation. However, no prior work has combined both tasks to effectively identify ambiguous questions as part of a conversation. In this paper, we propose a Multi-Task Learning (MTL) approach that uses a text generation model for both question rewriting and classification. Our models, based on BART and T5, are trained to rewrite conversational questions and identify follow-up questions simultaneously. We evaluate our approach on multiple test sets and demonstrate that it outperforms single-task learning baselines on the three LIF test sets, with statistically significant improvements ranging from +3.5% to +10.5% in terms of F1 and Micro-F1 scores. We also show that our approach outperforms single-task question rewriting models in passage retrieval on a large OR-QuAC test set.

## introduction conversational question answering (qa), which simulates human dialogues in information-seeking tasks, necessitates resolving ambiguities in the user queries based on the conversation history @xcite . figure 1 exemplifies the possible ambiguities that might arise from a typical dialogue with a user, highlighting the importance of addressing these ambiguities based on the conversation history. for example, in the question q 3 , "was he the owner of the paper?", the "he" and "the paper" referents are context-dependent, typically clarified by prior questions (q 1 , q 2 ) and answers (a 1 , a 2 ). to address such ambiguities, a number of approaches have been explored, such as follow-up question identification @xcite @xcite and conversational question rewriting @xcite @xcite @xcite . indeed, the follow-up question identification approaches have been used in conversational qa to enhance the interpretation of the user's intent and context. the main objective of these approaches is to determine whether a follow-up question is linked to the previous conversation history or not. if it is determined that the follow-up question is related to the previous conversation, the system can leverage the context provided by the conversation to generate more precise and relevant responses. in figure 1 , a follow-up question (q 3 ) is classified as valid if it can be linked to the previous conversation (q 1 , a 1 , q 2 , a 2 ), else it is classified as invalid (e.g., q 4 ). for example, the three-way attentive pooling network approach @xcite predicts this continuity by analysing the candidate question and conversation history, outperforming other models such as bilstm, cnn, and bert @xcite . in this paper, we use the three-way attentive pooling network as our strongest baseline. on the other hand, conversational question rewriting approaches have been employed in conversational qa to enhance the accuracy of the retrieved information by reformulating the user's original question. these approaches usually involve modifying the original question or generating new queries that better represent the user's intent by transforming a concise conversational question into a fully-grown, contextualised ad-hoc query. for example, in figure 1 , the question q ′ 3 "was sheldon the owner of the israeli press?" is a rewrite of q 3 "was he the owner of the paper?", based on the conversation history (q 1 , a 1 , q 2 , a 2 ). a t5 @xcite model can be fine-tuned to automatically reformulate the question by injecting information that exists in the context into a fully defined query @xcite . previous work, such as @xcite @xcite @xcite , focused on conversational question rewriting; however, they did not address the task of follow-up question identification. in this paper, to alleviate ambiguities in conversational qa, we investigate the combination of both the follow-up question identification task and the conversational question rewriting task into a single framework, thereby also improving the effectiveness of both tasks. recently, multi-task learning (mtl) has emerged as an effective approach for simultaneously learning numerous related tasks @xcite . mtl can be used to increase a system's performance on the text generation task by leveraging classification tasks. for example, @xcite adopted an mtl approach using a text generation bart @xcite model, which jointly learns a classification task and a text generation task by sharing the learner, and showed an improvement in an emotion-aware dialogue response generation model. text generation models like t5 have also been effectively used for multi-task learning involving both text generation and classification tasks, such as passage ranking/re-ranking and answer generation/extraction @xcite @xcite . given their effectiveness on various tasks, we argue that text generation models -bart and t5 -can be tailored for the follow-up question identification and conversational question rewriting tasks. our intuition is that by combining follow-up question identification and conversational question rewriting, the system's response accuracy and relevance can be enhanced. indeed, by identifying connections between the user's questions, addressing ambiguities, and leveraging the conversation's context, the system can refine its understanding of the user intent and can provide more precise and relevant responses. our text generation models leverage the multi-task learning of the conversational question rewriting and classification tasks to identify whether a question is a follow-up to the previous question and, accordingly, reformulate a question using the dialogue context. to the best of our knowledge, no prior work has inherently combined both tasks to more effectively address ambiguity in conversational questions. our contributions are as follows: (1) we leverage multi-task learning with a text generation model to effectively address the tasks of follow-up question identification and conversational question rewriting; (2) using the recent lif dataset @xcite , we compare our models to two recent baselines from the literature, and show that our multi-task learning bart model yields the best f1 and macro-f1 performance improvements over the strongest baseline, three-way attentive pooling, with statistically significant improvements ranging from 3.5% to 10.5% on all lif test sets; (3) our proposed multi-task learning t5 model significantly outperforms the single-task learning of question rewriting models for passage retrieval on the or-quac test set. ## mtl: classification & generation we define the follow-up question identification and conversational question rewriting tasks in sections 2.1 & 2.2. an overview of the proposed text generation model follows in section 2.3. ## experimental setup we list our research questions in section 3.1, introduce the used datasets in section 3.2 and present our baselines in section 3.3 ## experimental results we now address rqs 1 & 2, and conclude with a qualitative analysis. ## related work we discuss related work in conversational qa. conversational qa: conversational qa, which involves the correct interpretation of questions within an ongoing dialogue, has been addressed by several datasets like quac @xcite , coqa @xcite , and trec cast @xcite @xcite . a common strategy is concatenating previous questions with the current one for passage retrieval, yet this may be suboptimal for methods like bm25, which favour concise queries. several studies @xcite @xcite have offered alternatives to improve retrieval performance. @xcite proposed detecting topic shifts through heuristics and dependency parsing, while @xcite and @xcite utilised bert models to predict historical questions enhancing the current one's retrieval. @xcite furthered this by predicting related past questions and reformulating the current one using albert. in contrast, our work focuses on using follow-up question identification to improve conversational question rewriting. follow-up question identification: this task has seen diverse methods, ranging from rule-based solutions @xcite to statistical machine learning models like logistic regression @xcite . @xcite proposed a standout three-way attentive pooling network, identifying follow-up questions related to the conversation history and the associated answer passage, outperforming rule-based methods, logistic regression models, and neural network-based models such as bilstm, cnn, and bert @xcite . in our study, we utilise both the three-way attentive pooling network and bert as baselines, given their proven effectiveness. conversational question rewriting: this task involves transforming conversational queries into context-independent queries suitable for information retrieval (ir) systems @xcite . approaches range from sequence-tosequence models like lstm and gru @xcite , to rule and self-supervised learning methods fine-tuning gpt-2 @xcite , and transformer++ trained on the canard dataset @xcite . vakulenko et al. (2021b) compared various methods on the cast 2019 and 2020 datasets, emphasising the effectiveness of transformer++. recently, lin et al. (2020) presented a t5 model for conversational question rewriting. t5 outperformed neural network-based models such as lstm, gpt-2, bert, and unilm on the canard and cast 2019 datasets. in this paper, we propose a t5 model for conversational question rewriting due to its overall effectiveness. mtl in conversational qa: mtl has been applied in conversational qa, primarily focussing on answer span prediction and auxiliary tasks @xcite @xcite . some studies have combined classification and text generation tasks to improve passage ranking and answer generation @xcite @xcite , often leveraging models such as t5 @xcite . similarly, @xcite combined the classification of relation types and the generation of sentences to express these relation types. moreover, @xcite aimed to improve the response selection in multi-party conversations using an auxiliary task, namely a topic prediction task, which classifies whether a follow-up question is relevant to the user. our work also embraces the mtl paradigm, but unlike prior studies prioritising answer generation, response selection, and passage ranking, we focus on improving retrieval effectiveness by rewriting the user's question using a classification task. other approaches addressing ambiguity in conversational questions: @xcite addressed the ambiguity of the conversational question by asking clarifying questions to the user. similarly, @xcite explored enhancing dialogue generation with conversational concept flows, using a conversation-aware knowledge graph and a novel relation-aware graph encoder. in contrast to these studies, our work focuses on addressing the ambiguity of the conversational question by reformulating the user's original question and classifying whether the conversation is still focused on the same topic to enhance the accuracy of the information retrieval system. ## conclusions we proposed a method for conversational qa, which learns to predict the follow-up question and rewrites the conversational question simultaneously. our proposed mtl method makes use of text generation models including bart and t5 by generating the first token for a classification task and the follow-up tokens for a conversational question rewriting task. for the follow-up question identification task, our experiments on the lif dataset showed that our proposed mtl bart model has the best effectiveness, yielding statistically significant improvements over the baselines. for the question rewriting task, our proposed mtl t5 model performed best both in terms of first stage retrieval and re-ranking. furthermore, we showed that the bart-based model can be employed for the follow-up question identification and the conversational question rewriting tasks in both dis-criminative+generative and generative mtl setups.