On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method (2023)

Abstract: Most work on modeling the conversation history in Conversational Question Answering (CQA) reports a single main result on a common CQA benchmark. While existing models show impressive results on CQA leaderboards, it remains unclear whether they are robust to shifts in setting (sometimes to more realistic ones), training data size (e.g., from large to small sets) and domain. In this work, we design and conduct the first large-scale robustness study of history modeling approaches for CQA. We find that high benchmark scores do not necessarily translate to strong robustness, and that various methods can perform extremely differently under different settings. Equipped with the insights from our study, we design a novel prompt-based history modeling approach and demonstrate its strong robustness across various settings. Our approach is inspired by existing methods that highlight historic answers in the passage. However, instead of highlighting by modifying the passage token embeddings, we add textual prompts directly in the passage text. Our approach is simple, easy to plug into practically any model, and highly effective, thus we recommend it as a starting point for future model developers. We also hope that our study and insights will raise awareness to the importance of robustness-focused evaluation, in addition to obtaining high leaderboard scores, leading to better CQA systems.1

## introduction conversational question answering (cqa) involves a dialogue between a user who asks questions and an agent that answers them based on a given document. cqa is an extension of the traditional single-turn qa task @xcite , with the major difference being the presence of the conversation history, which requires effective history modeling @xcite . previous work demonstrated that the straightforward approach of concatenating the conversation turns to the input is lacking @xcite , leading to various proposals of architecture components that explicitly model the conversation history @xcite @xcite @xcite . however, there is no single agreed-upon setting for evaluating the effectiveness of such methods, with the majority of prior work reporting a single main result on a cqa benchmark, such as coqa @xcite or quac @xcite . while recent cqa models show impressive results on these benchmarks, such a single-score evaluation scheme overlooks aspects that can be essential in real-world use-cases. first, quac and coqa contain large annotated training sets, which makes it unclear whether existing methods can remain effective in small-data settings, where the annotation budget is limited. in addition, the evaluation is done in-domain, ignoring the model's robustness to domain shifts, with target domains that may even be unknown at model training time. furthermore, the models are trained and evaluated using a ''clean'' conversation history between 2 humans, while in reality the history can be ''noisy'' and less fluent, due to the incorrect answers by the model @xcite . finally, these benchmarks mix the impact of advances in pre-trained language models (lms) with conversation history modeling effectiveness. in this work, we investigate the robustness of history modeling approaches in cqa. we ask whether high performance on existing benchmarks also indicates strong robustness. to address this question, we carry out the first large-scale robustness study using 6 common modeling approaches. we design 5 robustness-focused evaluation settings, which we curate based on 4 existing cqa datasets. our settings are designed to evaluate efficiency in low-data scenarios, the ability to scale in a high-resource setting, as well as robustness to domain-shift and to noisy conversation history. we then perform a comprehensive robustness study, where we evaluate the considered methods in our settings. we focus exclusively on history modeling, as it is considered the most significant aspect of cqa @xcite , differentiating it from the classic single-turn qa task. to better reflect the contribution of the history modeling component, we adapt the existing evaluation metric. first, to avoid differences which stem from the use of different pre-trained lms, we fix the underlying lm for all the evaluated methods, re-implementing all of them. second, instead of focusing on final scores on a benchmark, we focus on each model's improvement (δ%) compared to a baseline qa model that has no access to the conversation history. our results show that history modeling methods perform very differently in different settings, and that approaches that achieve high benchmark scores are not necessarily robust under low-data and domain-shift settings. furthermore, we notice that approaches that highlight historic answers within the document by modifying the document embeddings achieve the top benchmark scores, but their performance is surprisingly lacking in low-data and domain-shift settings. we hypothesize that history highlighting yields high-quality representation, but since the existing highlighting methods add dedicated embedding parameters, specifically designed to highlight the document's tokens, they are prone to over-fitting. these findings motivate us to search for an alternative history modeling approach with improved robustness across different settings. following latest trends w.r.t. prompting in nlp @xcite we design marcqap, a novel prompt-based approach for history modeling, which adds textual prompts within the grounding document in order to highlight previous answers from the conversation history. while our approach is inspired by the embedding-based highlighting methods, it is not only simpler, but it also shows superior robustness compared to other evaluated approaches. as marcqap is prompt-based, it can be easily combined with any architecture, allowing to fine-tune any model with a qa architecture for the cqa task with minimal effort. thus, we hope that it will be adopted by the community as a useful starting point, owing to its simplicity, as well as high effectiveness and robustness. we also hope that our study and insights will encourage more robustness-focused evaluations, in addition to obtaining high leaderboard scores, leading to better cqa systems. ## results and analysis we next discuss the takeaways from our study, where we evaluated the considered methods across the proposed settings. we further analyze the effect of the conversation history length in figure 1 , evaluating models from the standard setting with different limits on the history length. for instance, when the limit is 2, we expose the model to up to the 2 most recent turns, by truncating h k . foot_7 key findings a key goal of our study is to examine the robustness of history modeling approaches to setting shifts. this research reveals limitations of the single-score benchmark-based evaluation adopted in previous works ( §4.1), as such scores are shown to be only weakly correlated with low-resource and domain-shift robustness. furthermore, keeping in mind that history modeling is a key aspect of cqa, our study also demonstrates the importance of isolating the contribution of the history modeling method from other model components ( §4.2). finally, we discover that while existing history highlighting approaches yield high-quality input representations, their robustness is surprisingly poor. we further analyze the history highlighting results and provide possible explanations for this phenomenon ( §4.3). this finding is the key motivation for our proposed method ( §5). ## limitations this work focuses on a single-document cqa setting, which is in line with the majority of the previous work on conversation history modeling in cqa ( §2.3). correspondingly, marcqap was designed for single-document cqa. applying marcqap in multi-document settings @xcite @xcite may result in partial history representation, since the retrieved document may contain only part of the historic answers, therefore marcqap will only highlight the answers which appear in the document. 23 in §5.3 we showed initial evidence that mar-cqap prompts can encode additional information that can be useful for cqa. in this work we focused on the core idea behind prompt-based answer highlighting, as a proposed solution in light of our results in §4. yet, we did not conduct a com22 as annotated in canard @xcite . 23 we note that this limitation applies to all highlighting approaches, including hae and poshae @xcite . ## conclusion in this work, we carry out the first comprehensive robustness study of history modeling approaches for conversational question answering (cqa), including sensitivity to model and training data size, domain shift, and noisy history input. we revealed limitations of the existing benchmark-based evaluation, by demonstrating that it cannot reflect the models' robustness to such changes in setting. in addition, we proposed evaluation practices that better isolate the contribution of the history modeling component, and demonstrated their usefulness. we also discovered that highlighting historic answers via passage embedding is very effective in standard setups, but it suffers from substantial performance degradation in low data and domain shift settings. following this finding, we design a novel prompt-based history highlighting approach. we show that highlighting with prompts, rather than with embeddings, significantly improve robustness, while maintaining overall high performance. our approach can be a good starting point for future work, due to its high effectiveness, robustness, and portability. we also hope that the insights from our study will encourage evaluations with focusonrobustness,leading to better cqa systems.