Q uality A dapt: an Automatic Dialogue Quality Estimation Framework (2022)

Abstract: Despite considerable advances in open-domain neural dialogue systems, their evaluation remains a bottleneck. Several automated metrics have been proposed to evaluate these systems, however, they mostly focus on a single notion of quality, or, when they do combine several sub-metrics, they are computationally expensive. This paper attempts to solve the latter: QualityAdapt leverages the Adapter framework for the task of Dialogue Quality Estimation. Using well defined semi-supervised tasks, we train adapters for different subqualities and score generated responses with AdapterFusion. This compositionality provides an easy to adapt metric to the task at hand that incorporates multiple subqualities. It also reduces computational costs as individual predictions of all subqualities are obtained in a single forward pass. This approach achieves comparable results to state-of-the-art metrics on several datasets, whilst keeping the previously mentioned advantages.

## introduction open-domain neural dialogue systems have increasingly drawn attention in natural language generation (nlg). these systems, colloquially known as chatbots, take advantage of large-scale training of complex models, making them increasingly more humanlike @xcite @xcite . a crucial step in the development of a dialogue system is its evaluation. the community has identified multiple characteristics of what constitutes a high-quality dialogue. these include comprehensible, fluent, empathetic, relevant and interesting, among others. the precise definition is often challenging to define and is application dependent. the current trend is to train models to evaluate responses under various aspects. these learningbased metrics either (1) map overall quality to a * corresponding author single defined aspect such as sensibleness (is the response adequate given the context) or (2) leverage several individual models to cover a wider range of quality aspects (subqualities). both have their drawbacks: in the first approach, the use of a single notion of quality limits the overall understanding of model performance and consequently its applicability to other domains; in the second approach, the need to individually train several models is both time and resource consuming, possibly duplicating model parameters that could be shared, such as feature representations. this paper proposes qualityadapt foot_0 , an automatic dialogue quality estimation framework that leverages the adapter paradigm @xcite to train individual adapters on different dialogue subqualities. then, adapterfusion @xcite combines the knowledge of the individual adapters for the downstream task of overall quality estimation. this allows for a system that is both extensible (by including different subqualities) and less resource-intensive (by sharing most of the pretrained model parameters). experimental results show that qualityadapt achieves comparable correlations with human judgements when compared to other state-of-the-art metrics. ## qualityadapt qualityadapt trains individual adapters for each subquality and composes them using adapterfusion for the task of overall quality estimation. in both the subquality and overall quality tasks, it returns a score that is obtained by combining a transformer encoder with a regression head on top. during inference, individual subquality predictions can be obtained in a single forward pass by parallelising their respective heads. encoder in our experiments, roberta-large @xcite is used to encode the contextresponse pair. in the tokenization step, we add for each utterance a token representative of the speaker. this added information lets the network identify the response's speaker, which in turn allows it to pay more attention to utterances from this speaker in the context if needed. compositionality training adapterfusion for the downstream task of overall quality estimation is a supervised task. as such, quality annotated data in terms of overall quality is required. however, the amount of annotations required for the fusion training step is much smaller when compared to fully fine-tuning a language model with this data. as a proof of concept, we composed two adapters in this paper: u-adapter, for understandability, and s-adapter for sensibleness. ## prediction compute one of the motivations of the qualityadapt framework is it's computational efficiency. we present average sample predictions per second on the test set using a single rtx 3070ti 8bg gpu, together with size of the metric's unique parameters on as expected, the forward pass on several transformer models decreases runtime performance when compared to a single forward pass, even when using larger models (usr and usl-h metrics are based on the roberta and bert-base models, respectively). when comparing between the different larger models, we can see that the inclusion of the adapter model decreases run-time performance by 25%. however, both the fusion module and the inclusion of more adapters does not significantly affect performance. ## conclusions this paper presents qualityadapt, a framework for automatic dialogue quality estimation. we show the composition of sensibleness and understandability adapters for the downstream task of quality estimation outperforms, on average, the performance of robust baselines, including those that take advantage of subquality composition. however, qualityadapt only requires a single forward pass on a language model to produce predictions for overall quality, thus reducing computational complexity. 2 current research in dialogue focuses mostly on monolingual chatbots, typically in english. multilingual lms such as xlm-roberta @xcite can be used to extract utterance representations directly in the target language after fine-tuning. however, this approach would still be somewhat limited by the lack of multilingual annotated data. @xcite proposes leveraging adapters for transfer learning in low resource settings by training a stack consisting of the source-language adapter with a task adapter. then, during inference, the source-language adapter is replaced with the target-language one. we leave these experiments for future work.