B ot E val: Facilitating Interactive Human Evaluation (2024)

Abstract: Following the rapid progress in natural language processing (NLP) models, language models are applied to increasingly more complex interactive tasks such as negotiations and conversation moderations. Having human evaluators directly interact with these NLP models is essential for adequately evaluating the performance on such interactive tasks. We develop BotEval, an easily customizable, open-source, evaluation toolkit that focuses on enabling human-bot interactions as part of the evaluation process, as opposed to human evaluators making judgements for a static input. BotEval balances flexibility for customization and user-friendliness by providing templates for common use cases that span various degrees of complexity and built-in compatibility with popular crowdsourcing platforms.We showcase the numerous useful features of BotEval through a study that evaluates the performance of various chatbots on their effectiveness for conversational moderation and discuss how BotEval differs from other annotation tools.

## introduction as natural language processing (nlp) models become more versatile with the recent advances of language models and their instruction-tuned counterparts @xcite , it is becoming more common to create language agents @xcite and apply them to complex interactive tasks, such as negotiations @xcite , conversational moderation @xcite , reasoningguided response generation @xcite , and personalized response generation @xcite . as noted by @xcite , the evaluation methodology plays a critical role in accurately comparing models. for example, rankings between dialogue models can change depending on whether they are evaluated based on single-turn responses or full conversations. in addition, @xcite found the evaluators point of view when evaluating a model is also an important factor. they showed that human evaluators perceived conversational moderators as more effective in making the evaluators become more cooperative and respectful when the evaluators directly interacted with the moderators while acting as the moderated user (first person point of view) compared to when they evaluated a completed interaction between a moderator and a moderated user as a bystander (third person point of view). however, these factors are overlooked in previous approaches that have focused on a simplified evaluation, such as comparing two complete conversations or individual responses @xcite , or specific dialogue applications such as task-oriented dialogue @xcite . therefore, it is important to develop evaluation tools that enable an environment that evaluates models in a setting that best encapsulates how humans actually interact with models. to facilitate accurate human evaluations of complex interactive tasks, we developed boteval, 1 a comprehensive evaluation toolkit that focuses on enabling human -bot foot_0 interactions as part of the human evaluation process. for flexibility, it is dynamically configurable to accommodate as many human agents and model agents to interact with each other simultaneously with a custom dialogue manager. it is also designed with modular components, such as the interaction interface, instructions, and survey, so that they can be individually adapted to accommodate various use cases. while main-1 source code and documentation for boteval can be found at https://github.com/isi-nlp/boteval . we make the demo video of boteval available at https: //justin-cho.com/boteval . in addition, a live demo of boteval is also available at https://spolin.isi.edu/ boteval-dev1 where reviewers can complete a sample human evaluation task. in summary, boteval's main contributions are: ## boteval system overview boteval is a web application that provides an evaluation interface, what the human evaluators (i.e., crowdsource workers) see (section 2.1), and an administrator dashboard, what the administrator uses to manage the evaluation task and evaluators (section 2.2). we recommend that the bots that evaluators interact with are provided as separate apis that boteval can make queries to, as this isolates the management of the bot deployment and boteval (section 2.3). human evaluators can be flexibly set to crowdsource workers from amt or prolific or any other evaluators with internet access by having them create an account directly for a deployment of boteval using a public external link. an evaluation task is configured with a central yaml config file that identifies the frontend components to use, the deployment environment, and the crowdsourcing platform to use. ## system architecture an overview of boteval's system architecture is shown in figure 3 . boteval is a web application (i.e., a client-server model). we describe the frontand back-end technology stacks in the following sections. ## related work in ## conclusion we presented boteval and its usefulness in collecting human evaluations for interactive tasks that require live human-bot interactions through a case study of evaluating various language models on their ability to conversationally moderate online discussions. boteval provides a customizable interface that can be adapted for various evaluation and annotation use cases while also providing integration with popular crowdsourcing platforms and task management features. we hope that this work will serve as an important foundation for setting up custom interactive human evaluation tasks that facilitate our understanding of more complex nlp systems as they become increasingly sophisticated and capable.