LEGOE val: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing (2021)

Abstract: We present LEGOEval, an open-source toolkit that enables researchers to easily evaluate dialogue systems in a few lines of code using the online crowdsource platform, Amazon Mechanical Turk. Compared to existing toolkits, LEGOEval features a flexible task design by providing a Python API that maps to commonly used React.js interface components. Researchers can personalize their evaluation procedures easily with our built-in pages as if playing with LEGO blocks. Thus, LEGOEval provides a fast, consistent method for reproducing human evaluation results. Besides the flexible task design, LEGOEval also offers an easy API to review collected data.

## introduction as dialogue systems are becoming an increasingly trending topic, the need for standardized and reliable evaluation procedures has grown significantly. typically, the evaluation of dialogue systems is accomplished by the use of both automatic metrics @xcite @xcite and human evaluation @xcite . automatic metrics are reliable measurements, but common automatic metrics correlate weakly with human judgment @xcite @xcite . thus, human evaluation has become a primary method for dialogue system evaluation. previously, researchers invited participants to the lab to physically interact with dialogue systems; recently, the popular approach is crowdsourcing using platforms such as amazon mechanical turk (amt) @xcite . however, human evaluation via crowdsourcing presents its own challenges, being both expensive and time-intensive. specifically, human evaluation requires a huge engineering effort to develop the interface and deploy the task on crowdsourcing platforms. the front-end interfaces can be difficult to set up: the crowdworkers need to be properly instructed, and the tasks need to be prepared to reflect real-world environment as closely as possible. furthermore, one needs to take into account the high variability of user behaviour especially in crowdsourced environments @xcite . it was shown that even different phrasings can result in weaker levels of agreement @xcite . thus, it is not trivial to reproduce the human evaluation results from scratch. to address these problems, we present legoeval, an open-source toolkit that enables researchers to easily build and deploy their human evaluation tasks on amt in one click. legoeval supports representative human evaluation tasks, such as static evaluation, where crowdworkers are asked to rate sampled dialogues, and interactive evaluation, where crowdworkers interact with two systems and evaluate their responses @xcite . furthermore, researchers are also able to customize their own human evaluation procedures easily with legoeval. existing tools typically provide rigid human evaluation templates. for example, dialcrowd @xcite follows the speech synthesis evaluation toolkit @xcite and provides a small number of standard evaluation experiments, however, researchers have to manually create the web services and then post the evaluation task on amt. @xcite developed chateval, which posts a response comparison task @xcite on amt. it is only effective for specific dialogue systems and is not generalizable. the widely used toolkit parlai @xcite supports crowd-toolkit legoeval dialcrowd @xcite parlai @xcite ## legoeval toolkit legoeval is an open-source python-based toolkit. as shown in figure 1 , legoeval includes three modules: (1) a task flow builder (section 2.1) for designing the human evaluation task, (2) an amt manager (section 2.2) that automatically deploys the evaluation task on amt, and (3) a data reviewer api (section 2.3) that retrieves and formats data collected on amt. we describe each module in the following sections. ## toolkit usage in this section, we provide a simple example of what the researcher needs to do to create a task. typical usage researchers would first design their high-level task flow on paper, (e.g, instruction page, then a pre survey, ..., and finally, a post survey). after this, researchers can add their different pages to their task flow by editing the build.py file and initializing a list of our provided python classes. some pages, such as the survey page, have a high level of customization, where developers can specify what types of questions they want to display on the survey, and the questions' relative order. next, researchers can test their task locally by running server.py and navigating to their localhost. if they are happy with the results, researchers can launch their task on mturk with one command: launch hits.py. if researchers want to make a few tweaks, such as using a custom font for the instructions, they can easily edit a single react.js file and override any necessary css in typically one line of code. a strong benefit of our platform is that it is very fast for researchers to plug and play typical evaluation tasks. for further detail, see figure 2 for an example. advanced usage as previously mentioned, it is possible to add branching logic via the use of lambda functions. firstly, the researcher must define a function that takes as an input the state dictionary. the state dictionary contains information representing the current state of the task and any persisted data. thus, based upon the data collected so far, the researcher can decide which page to instantiate and return. the researcher adds the lambdafunction object passing in their lambda function. during run time, when the lambdafunction object is popped from the task flow list, our framework will call the function by passing in the current state dictionary and return the determined page to display. ## conclusion and future work we introduce legoeval, an open-source pythonbased toolkit that allows researchers to easily develop human evaluation tasks for dialogue systems on amt in a lego plug-and-play fashion. lego-eval provides a variety of commonly-used react.js components as building blocks for researchers to use. we have shown that it is straightforward to customize different types of human evaluation procedures for dialogue systems in a few lines of code, create new components by leveraging the shared state between the front-end and back-end, and reproduce human evaluation results in other works effortlessly. with legoeval, we hope to provide a simple and flexible way to evaluate dialogue systems. for future work, we plan to add more human evaluation procedure templates and react.js components. we further want to extend the tool to support human-human interaction on amt, making it work not only for evaluation, but also for dialogue data collection.