Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach (2023)

Abstract: Evaluating conversational information retrieval (CIR) systems is a challenging task that requires a significant amount of human labor for annotation. It is imperative to invest significant effort into researching more labor-effective methods for evaluating CIR systems. To touch upon this challenge, we take the first step to involve active testing in CIR evaluation and propose a novel method, called HomCoE. It strategically selects a few data for human annotation, then calibrates the evaluation results to eliminate evaluation biases. As such, it makes an accurate evaluation of the CIR system at low human labor. We experimentally reveal that it consumes less than 1% of human labor and achieves a consistency rate of 95%-99% with human evaluation results. This emphasizes the superiority of our method over other baselines.

## introduction as evidenced by various studies @xcite , the conversational information retrieval (cir) system has shown its effectiveness in domains such as conversational question answering @xcite and conversational recommendation @xcite . however, accurate evaluation of cir systems continues to be a rapidly evolving research area due to the multi-turn nature of user-system conversations @xcite @xcite . it often requires the user to constantly interact with the cir system and then evaluate the system based on its real-time responses, which are labor-expensive and generally only feasible in an industry lab @xcite . to alleviate this problem, current practices prepare a set of input data and employ humans to either manually annotate the answer to be compared with the system-generated results or directly score the corresponding system output (larson et al., 2019; † correspondence to wenqiang lei. @xcite . for example, @xcite ; @xcite require the human to answer the questions in a pre-collected question set and use these answers as ground truth to evaluate the modelgenerated results. @xcite requires the human to validate and re-annotate the pre-collected conversation dataset to provide finer-grained labels for the input and characterize the system performance based on such labels. moreover, @xcite employs the human to directly rate each response of the system on relevance and naturalness. while these methods show their effectiveness in cir evaluation, they still require traversing the entire dataset, resulting in significant human labor costs @xcite @xcite . consequently, it is imperative to invest significant effort into researching more labor-effective methods for evaluating cir systems. to this end, we take the first step to involve active testing 1 for the label-effective cir evaluation, proposing a novel method calledhumanmachine collaborative evaluation (humcoe, for short). the idea is to strategically select a few data for the human to annotate the system outputs (e.g., provide an answer to the question), and then calibrate the evaluation results to eliminate evaluation biases. specifically, given an affordable number of human interactions, humcoe starts by selecting a subset of representative data and requesting human annotation or scoring. in this process, the representative data is measured based on the probability of inconsistency between cir's prediction and a pseudo-label that is generated by a surrogate machine. after that, humcoe assigns an importance weight to each selected data and calibrates the evaluation results. as such, the sampling bias is eliminated, and hence the evaluation accuracy is largely improved. we demonstrate the effectiveness of humcoe on typical cir tasks, including asking clarifying questions and the conversational question answering. in our experiments, we assess the task success rate and quality of system responses using automatic evaluation metrics on pre-collected conversational datasets. additionally, we employ the human to assess the relevance and naturalness scores of system responses. our experiments indicate that humcoe produces results that are extremely comparable to those of a human-based evaluation yet require very little labor. it consumes less than 1% of human labor and achieves a consistency rate of 95%-99% with human evaluation results on all tasks. compared to other classical active testing methods, our approach achieves a higher evaluation consistency rate by an average of 11.75%. we also explore our method in the context of chatgpt. we notice that chatgpt does not provide satisfactory evaluation consistency with human evaluation, with an average consistency rate of 88.24%. but, after introducing a small number of labor costs (0.68%), humcoe improves the evaluation consistency by 9.59%. this highlights the importance of humanmachine collaboration for efficient cir evaluation. evaluating any highly interactive process is challenging. in this paper, we experimentally reveal our effectiveness and efficiency on the cir evaluation task. we believe that our work could provide a landmark for future human-machine-based cir evaluation research. to sum up, we claim the following contributions. we set a landmark for future human-machinebased cir evaluation research. ## related works as our focus is on utilizing both human and machine in evaluating the cir system via active test-ing, we offer literature reviews on cir evaluation and active testing. for more discussion on humanmachine evaluation, please refer to appendix e. ## task definition given an evaluation metric em and a test conversational dataset 2 d, the evaluation τ is completed by calculating the metric value on the given dataset, described by the following equations. ## methodology there are two steps in humcoe: 1) it starts by selecting a subset of representative data d s and requesting human annotation or scoring y ds , where the representative data d s are selected based on a surrogate machine. 2) after that, it assigns an importance weight to each selected data and calibrates the evaluation results. step 1: surrogate machine based data sampling. we aim to select representative data for human annotation. in this process, the representative data is measured based on the probability of inconsistency between cir's prediction and a pseudo-label that is generated by a surrogate machine. before human annotation, we involve a surrogate machine g as a user proxy to generate pseudo-labels. taking the conversational questionanswering task as an example, the surrogate machine is required to answer the input questions. the output answers are stored as pseudo-labels. formally, given the surrogate machine-annotated dataset d 1 , we denote d @xmath0, v i } ∈ d 1 and v i is a list of proxy responses. usually, a surrogate machine g is trained or fine-tuned based on the same training data as f . meanwhile, designing g with a different architecture from f is always preferred to encourage response diversity between f and g. for example, one could set gpt-2 as f and bart as g to generate responses during the conversations. borrowing the idea of active testing methods @xcite , we aim to select a subset of representative data d s that are hard for the cir system to properly handle for human annotation y ds . the "hardness" in this paper is measured by the probability of the cir system making mistakes on pseudo-labels ŷd 1 , rather than the model uncertainty @xcite . namely, we prefer data with a high probability of inconsistency between cir's predictions and pseudo labels. technically, we measure the hardness as follows. large q(d i ) means that the performance of cir f on d i in terms of a metric em may not be satisfying. finally, given the maximum number of human interactions t , we sample a subset d s with the size |d s | = t according to probability q. the selected data are assigned to the human for evaluation and obtaining the ground truth labels. ## conclusion evaluating any highly interactive process is a challenging task when taking human labor into account. to touch upon this challenge, we take the first step to utilize active testing to evaluate the cir system. we tentatively give a successful implementation to explore the potential of human-machine collaborative evaluation in the cir scenario. empirical experiments are conducted to testify to the effec-tiveness of our method. in a consistent manner, our method produces results that are extremely comparable to those of a human-based evaluation yet require very little human labor compared to other baselines. our future work may design a more sophisticated active testing method and see the possible outcomes of cir evaluation.