meta CAT : A Metadata-based Task-oriented Chatbot Annotation Tool (2020)

Abstract: Creating high-quality annotated dialogue corpora is challenging. It is essential to develop practical annotation tools to support humans in this time-consuming and error-prone task. We present metaCAT, which is an open-source web-based annotation tool designed specifically for developing task-oriented dialogue data. To the best of our knowledge, metaCAT is the first annotation tool that provides comprehensive metadata annotation coverage to the domain, intent, and span information. The data annotation quality is enhanced by a real-time annotation constraint-checking mechanism. An Automatic Speech Recognition (ASR) function is implemented to allow users to paraphrase and create more diversified annotated utterances. metaCAT is publicly available for the community.

## introduction the progress of the development of multi-turn task-oriented dialogue systems has been largely constrained by the availability of large-scale high-quality data @xcite . due to the complexity of annotating task-oriented dialogue data, a large number of errors have been found in existing benchmark datasets, e.g., multi-woz @xcite . despite ongoing efforts such as re-annotating the state tags based on the original utterances in multiwoz 2.1 @xcite and introducing slot span annotations in mul-tiwoz 2.2 @xcite , there is still much room for improvement. creating high-quality annotated dialogue corpora is highly challenging thus necessitates a high level of human engagements. it is essential to develop practical annotation tools for supporting experts in this time-consuming and error-prone task. although many nlp annotation tools exist, lida @xcite is the only one designed specifically for annotating multi-turn task-oriented benchmark dialogue datasets like multiwoz. as a lightweight interactive dialogue annotator, lida provides an end-to-end pipeline for converting raw text to structured conversation data. it claims to support machine learning-based annotation recommenders and provide an interface resolving interannotators disagreements. lida is an easy-to-use tool for tagging boolean and string type slot values. however, it lacks functionality for annotating complex data. these data include enumeration, span information, and multi-valued slot values for utterance and other metadata such as domains, intents, etc. a critical step in ensuring data quality when annotating is to perform real-time constraintchecking. retrospective inter-annotator disagreement resolution (i.e., after an annotation task is completed) is an option offered by lida. it is desirable to design a function which provides realtime constraint-checking to eliminate human errors. for example, the slot value "dontcare" should be avoided in the system's annotation, because it can only be applied to a user's annotations. this highlights the need to introduce metadata to define the scope of the annotation and, in the meantime, enforce consistency between human inputs and underlying constraints. data annotation task is time-consuming as there are large amounts of typing actions involved. to improve the efficiency of the annotation task, we identify the need to integrate the automatic speech recognition (asr) function to the annotation process. by automatically converting speech to text, asr can speed up the data input process otherwise done by keyboard typing. asr can also be used to create paraphrases of utterances, and as a consequence, enhance the diversity of the spoken language. this is especially useful in scenarios where manually collected dialogue corpora are rare and expensive. providing such diversity is regarded as the key to increasing the robustness of dialogue models trained with these data. in this paper, we propose metacat, which is a web-based task-oriented chatbot annotation tool with complete management of users, tasks, datasets, and dialogues. metacat extends lida by contributing additional key useful features including: various annotation tools have been developed, targeting nlp tasks in recent years. as depicted in @xcite , these tools are built for disparate nlp tasks ranging from general text processing (i.e., gate @xcite ) to entity linking (e.g., inception @xcite ). among them, a number of dialogue annotation tools are discussed below. dialogueview @xcite is a dialogue annotation tool developed for segmenting recorded dialogue into utterances, annotating speech repairs, tagging speech acts, and segmenting dialogue into hierarchical discourse segments. the focus is on dialogue segmentation; therefore features for generating task-oriented dialogue data, for example slot-value labeling, are not available. twist @xcite supports turn segmentation and content feature annotation. after highlighting and creating new turn segments, users can assign predefined content features to the targeted turn segment. the content feature includes descriptions like "objective", "subjective", etc. however, there is no means to support label classification and slotvalue annotation. dart @xcite stands for dialogue annotation and research tool, which is a research environment enabling users to annotate and analyze dialogues automatically. dart was developed for linguistic research and analysis. dart provides a convenient means for leveraging linguistic resources to improve the annotation results and test hypotheses. only intent and enumeration slot can be labeled via this tool. it is not designed to create task-oriented dialogue data for training dialogue models, therefore lacking most of the related features. lida @xcite claims to be the first annotator capable of providing a full end-to-end dialogue annotation pipeline. it is a web-based tool designed specifically for task-oriented dialogue systems. it provides basic features to annotate the boolean or string type slot-value of each turn of the dialogue. lida integrates with back-end machine learning models as annotation recommenders to achieve semi-automatic annotating. it contains a dedicated interface to resolve inconsistent annotations between different annotators. despite implementing several key features for annotating task-oriented dialogue data, lida lacks functionality supporting users to handle complex data annotation, such as enumeration, span information and multi-valued slot labeling. lida can only handle inter-annotator disagreement resolution at the postediting stage. real-time annotation error-checking is out of the scope. in this paper, we present metacat, which is designed to fill the above gaps. the user interfaces for the administrator and annotators are developed independently as the roles and privileges are different. the administrator is responsible for uploading metadata and dialogue data, assigning and monitoring the annotation tasks, and downloading annotated data. annotators are responsible for the execution of annotating tasks and paraphrasing tasks. all tasks are managed in batches with each containing several to dozens of dialogues. the intermediate results of annotation are saved to mongodb. the slots can be classified into enumeration type and string type. an enumeration slot contains a list of specific slot values without any span information. a string slot is usually one word or short segment of text taken from the utterance. some special values of a string slot exist without span information, e.g., "dontcare". a slot usually corresponds to one value but there are some slots containing multiple values, such as names of two restaurants recommended by the system. in addition to offering a comprehensive metadata annotation ontology, metacat also supports a dialogue paraphrasing feature, which requires the annotators to paraphrase the utterances via keyboard or asr before the annotation task. figure 1 depicts the journey of the administrator and annotators in using metacat. a short video clip is available in our youtube channel foot_0 demonstrating the system. ## conclusion and future work we present metacat, an open-source web-based annotation tool designed specifically for taskoriented dialogue datasets. it is the first annotation tool providing a comprehensive metadata annotation to cover service domain, intent and span information. metacat ensures data quality with a real-time constraint-checking mechanism. an asr-inclusive paraphrasing function enables users to generate more diversified utterances with annotations. metacat can be used to enhance taskoriented dialogue datasets (i.e., multiwoz 2.1) with easy-to-use functionality. existing dialogue annotation tools focus on the current turn of the dialogue, lacking inclusion of related contexts. for instance, there is a need to refer to the information from the previous turns to annotate the term "the hotel" in a typical hotel booking dialogue scenario. apart from the abovementioned co-referencing annotation function, future work will focus on designing other annotation features to meet the community's practical needs, for example, annotations for sentiment and multimodal intent. many open-domain conversations are goal-driven with the intent to solve real-world problems, for example, for recommendation and search purposes. developing knowledge-grounded opendomain conversation datasets becomes an emerging trend @xcite to regulate the free-form data with facts and knowledge. meta-cat can be customized to annotate open-domain dialogue datasets from this perspective.