A View From the Crowd: Evaluation Challenges for Time-Offset Interaction Applications (2021)

Abstract: Dialogue systems like chatbots, and tasks like question-answering (QA) have gained traction in recent years; yet evaluating such systems remains difficult. Reasons include the great variety in contexts and use cases for these systems as well as the high cost of human evaluation. In this paper, we focus on a specific type of dialogue systems: Time-Offset Interaction Applications (TOIAs) are intelligent, conversational software that simulates face-to-face conversations between humans and pre-recorded human avatars. Under the constraint that a TOIA is a single output system interacting with users with different expectations, we identify two challenges: first, how do we define a ‘good’ answer? and second, what’s an appropriate metric to use? We explore both challenges through the creation of a novel dataset that identifies multiple good answers to specific TOIA questions through the help of Amazon Mechanical Turk workers. This ‘view from the crowd’ allows us to study the variations of how TOIA interrogators perceive its answers. Our contributions include the annotated dataset that we make publicly available and the proposal of Success Rate @k as an evaluation metric that is more appropriate than the traditional QA’s and information retrieval’s metrics.

## introduction time-offset interaction applications (toias) @xcite are a sort of chatbot applications that lie between question answering (qa) and information retrieval (ir). they differ from qa in that a toia's task is not about demonstrating comprehension of a text span @xcite but selecting a single (one-shot) appropriate answer from a restricted set of answers, a problem also known as answer retrieval (ar) or retrieval-based dialogue @xcite . ideal toia interactions are expected to mirror a dialogue with a real person, including all the pos-figure 1 : initial ten turns of a casual dialogue between an interrogator and a toia avatar using our best performing retrieval model. we explore this question using a publicly available dataset that was manually annotated by its avatar maker -the margarita dialogue corpus (mdc) @xcite . the best performing ir model we could produce had a low recall@1 -24% on the development (dev) set and just below 10% on the test set. when trying to interact with this avatar, one would expect to get a wrong answer about 1/4th of the times or less. however, when chatting with the avatar using the best performing model (figure 1 ), we could see that the system wasn't so bad in entertaining and holding a conversation. hence we asked a 'crowd' of human annotators to give their opinion, and we learned that the task to define the correct answers isn't straightforward, primarily due to misaligned expectations about answer relevance. we explore a number of metrics and single out success rate @k (sr@k) as the most pertinent metric for optimizing toias. our contributions include the annotated dataset that we make publicly available, and the introduction of sr@k as the best metric for evaluating toais. we present previous work on toias and related datasets in section 2. sections 3, 4, and 5 introduce the corpus, retrieval models, and annotation process we used, respectively. we present our results and analysis in section 6, and discuss further in section 7. in section 8, we share how to apply this work's conclusions to develop better avatars. ## related work we present a number of recent toias, and data sets relevant for their study and development. while most of the related work focuses on large corpora, working with small datasets and addressing evaluation issues of toias are interesting, practical problems both for the ir (what is 'relevant'?) and the nlp communities (transfer learning and low resources corpora). ## retrieval models we used five models for retrieving answers for the questions in the mdc dialogue dataset, and for shortlisting the top candidate responses for the 'crowd' annotation task. let a be an answer in the kb. for every q-a pairs in the kb, we labeled them as 1's to indicate a relevant match. we then sampled a number of irrelevant (or 'wrong') matches for every question, and labeled them as 0's. we tried different sampling ratios, namely drawing one wrong match for every correct one (1:1), ten wrong ones (1:10), a hundred (1:100) and using all the available utterances (1:all). to increase the data size further and better generalize for questions phrased differently, we augmented the train data by sampling synthetic questions using the methodology proposed by @xcite and their python implementation. 5 we fine-tuned bert for 3 more epochs (we chose a few epochs as advised by dodge et al. (2020)) using wolf et al. (2019)'s transformers library. we only report on bert q-a 1:100 and bert q-a 1:all as they were the best performing. ## crowd annotations we developed a web interface (figure 3 ) for collecting the annotations from the 'crowd' using the crowdsourcing platform amazon mechanical turk (amt). full anonymity of the users were maintained and the erb review of the host institution didn't raise ethical concerns. for each question in the mdc dialogue dataset, we took the union of the top-10 answers retrieved by the five different retrieval techniques described above. on average, each question has about 24 selected answers. using a sliding window on all the dialogue questions, we selected three conversation turns, and appended the prediction as a fourth turn (interrogator-avatar-interrogator-predicted avatar response) without specifying who was whom, and always starting the dialogue snippet from an interrogator's question and ending with the avatar's answer. we chose four turns because it seems to give an optimal context size by looking at the annotations performed on the humod dataset @xcite ). so we have 339 dialogue snippets for the dev set and 341 for the test set. each human annotator could rate as many snippets as they wanted in one task. on average, they rated 23 sampled dialogue conversations. they were asked to rate the last reply of the dialogue snippet on a 1-5 scale according to the dialogue context (where 1: clearly not a good match; 5: perfect match for the context). each dialogue-reply pair is rated by three different annotators. for each dialogue context, there are on average 72 annotated answers (24 times 3), resulting in a total dataset size of 24,291 annotations for the dev dialogues and 24,555 for the test dialogues. in order to maintain high quality responses in the data, we defined a blacklist of annotators who gave poor quality annotations as follows. we forced each annotator to give a rating for the 'gold answer' given by the avatar maker in the dialogues data. if the annotator gave a rating lower than 4 (i.e., 1, 2 or 3) to the gold answer, we removed them from the annotations. while this blacklisting methodology is quite restrictive (we lose about 36% annotations), we have a large enough number of annotations left for our purposes. ## results and analysis we analyze our annotations in terms of interannotator agreement and the relationship between the crowd's opinion and the best retrieved answers by the models. we then report the ir metrics on the models we decided to study. ## conclusion and future work we explored the challenge of defining what a 'good' answer is in the context of a toia by annotating a dataset used for creating an avatar, and evaluat-ing human-avatar dialogues. we learned that the perceived 'right' answer for avatar interrogators differs from the avatar maker expectations partly because some questions and answers are too versatile, i.e., they go well with many answers and questions, respectively. additionally, yes and no answers are equally perceived as relevant by users interrogating an avatar but would be right or wrong for a given avatar maker. we make all the human annotations we collected available to the research community. we challenged classical retrieval metrics and proposed that toia's dialogue managers should optimize success rate @1. success rate @k for different levels of k can help identify how to improve retrieval techniques. our future work includes recognizing versatile questions and answers, designing methods to elicit more precise answer recordings at the avatar creation stage, and forcing yes/no answers with acceptable degrees of confidence. we plan to use transfer-learning and one-shot learning for leveraging state-of-the-art results of deep neural models in the context of a toia. addressing misaligned expectations between different user needs and picking the right metric are essential to improving the design, usability, and answer retrieval methodology of time-offset interaction applications.