MATILDA - Multi- A nno T ator multi-language I nteractive L ight-weight Dialogue Annotator (2021)

Abstract: Dialogue Systems are becoming ubiquitous in various forms and shapes - virtual assistants(Siri, Alexa, etc.), chat-bots, customer sup-port, chit-chat systems just to name a few. The advances in language models and their publication have democratised advanced NLP.However, data remains a crucial bottleneck. Our contribution to this essential pillar isMATILDA, to the best of our knowledge the first multi-annotator, multi-language dialogue annotation tool. MATILDA allows the creation of corpora, the management of users, the annotation of dialogues, the quick adaptation of the user interface to any language and the resolution of inter-annotator disagreement. We evaluate the tool on ease of use, annotation speed and interannotation resolution for both experts and novices and conclude that this tool not only supports the full pipeline for dialogue annotation, but also allows non-technical people to easily use it. We are completely open-sourcing the tool at https://github.com/wluper/matilda and provide a tutorial video1.

## introduction as a community, we have observed great advances in the last decade that include word-embeddings @xcite , seq-to-seq models for a variety of tasks @xcite and pretrained, transformer-based language models @xcite . relying on these seminal works, a plethora of downstream tasks (e.g. nmt, q&a, dialogues, summarisation, etc.) have seen no in the field of annotation tools and data generation, recent advances such as @xcite show similar promise by open-sourcing technology and developing it with modern usability-related principles in mind. following in the spirit of such similar research, we present matilda (a full dialogue annotation tool specifically focused on the inclusivity for all languages and facilitating multiple annotators). we evaluate it on a variety of usability aspects, both with experienced and untrained users, and conclude that both our dialogue annotation and creation tools are easy-to-use. furthermore, matilda offers more features than any comparable tool in the research community; comfortably supporting multiple annotators as well as multiple languages during the annotation process. therefore, we have open-sourced it and provided precompiled docker images for easy setup. matilda's main contributions are: 1) a native annotation tool that is quick-to-adapt foot_0 for multilanguage support; 2) a user-friendly interface to simply and intuitively manage multiple users as well as easily distribute datasets to crowd-workers for annotation; 3) task-oriented multi-speaker annotation capabilities (in the style of multiwoz and taskmaster); 4) inter-annotator resolution; and 5) integrated recommendations to assist annotators. twist @xcite is a dialogue annotation tool which consists of two stages: turn segmentation and content feature annotation. turn segmentation allows users to create new turn segments from raw text. after this, users can annotate sections of text in a segment by highlighting them and selecting from a predefined feature list. however, this tool doesn't allow users to specify custom annotations or labels and doesn't support classification or slot-value annotation. this is not compatible with modern dialogue datasets which require such annotations @xcite . incep-tion @xcite is a semantic annotation platform for interactive tasks that require semantic resources like entity linking. it provides machine learning models to suggest annotations and allows users to collect and model knowledge directly in the tool. gate @xcite is an open source tool that provides predefined solutions for many text processing tasks. it is powerful because it allows annotators to enhance the provided annotation tools with their own java code, making it easily extensible and provides a great number of predefined features. however, gate is a large and complicated tool with a significant setup cost -its instruction manual alone is over 600 pages long foot_2 . despite their large feature sets, inception and gate are not designed for annotating dialogue and cannot display data as turns, an important feature for dialogue datasets. brat @xcite and doccano foot_3 are webbased annotation tools for tasks such as text classification and sequence labelling. they have intuitive and user-friendly interfaces which aim to make the creation of certain types of dataset such as classification or sequence labelling datasets as fast as possible. brat also supports annotation suggestions by integrating ml models. however, like inception foot_4 and gate foot_5 , they are not designed for annotating dialogues and do not support the gen-eration of formatted conversational data from a raw text file such as might be outputted by a transcription service. lida @xcite provides an easy-to-setup annotation tool for modern taskoriented dialogues and also supports the integration of recommendations. however, lida is not accessible for multiple users and is only intended for the english language. matilda addresses these shortcomings and adds features such as: annotation styles compatible with modern dialogue datasets, inter-annotation resolution, customisable recommendations and user administration. dia-logueview's @xcite main use-cases are focused on segmenting recorded conversations, annotating audio files and discourse segmentation. granular labelling of the dialogue, recommenders, inter-annotator agreement, and slot-value labelling are not possible. ## system overview we introduce an annotator service that extends previous successful experiences, like lida, by introducing features that address large-scale, taskoriented dialogue annotation projects. in particular, we allow for distributed multi-annotators, multilanguage support, interannotator resolution and custom recommenders to assist the annotation process. furthermore, our modern and modularised implementation simplifies extension to additional languages, use-cases and annotation styles. a typical use-case follows this workflow: creation of a dataset we envision two main ways to create a corpus: either interactively or by uploading existing data. we adopt data representations that allow backward compatibility with other tools based on text files with a simple syntax, and a json format that is easy to operate. user administration once a corpus consisting of several collections is created, administrators can then proceed to assign those collections to one or more different annotators. the assigned partition will then be shown to the designated annotators in their "collection" view, ready to be annotated. according to the typical use case, we need two roles for the users, which we call annotators and administrators. we want our system to include user management with a simple interface for creation, editing and removal. annotation and supervision each annotator has access only to the subsets of dialogues assigned to them to add/modify annotations and monitor multi-language support: the ability to localise the annotation tool for different languages. crowd multi-annotator support: the possibility to manage users and easily deploy to many annotators in different locations. recommenders: ml models to suggest annotations. inter-annotator disagreement resolution: whether the system has an interface to resolve disagreements between different annotators. language: what programming language the system uses work progress. figure 1 shows a screenshot of the annotation interface and highlights the slot-filling functionality. administrators inspect annotators' work and resolve conflicts in the interannotation interface. when annotators provide diverging annotations, a designated supervisor provides a gold standard either opting for one of them or introducing an alternative one. besides, the system computes interannotator agreement metrics, such as cohen's kappa. gold standard annotations provided by supervisors are recorded separately and do not overwrite the original ones. the interannotator is designed to confront two or more annotated dialogue collections and resolve annotation conflicts between them. matilda automatically retrieves all annotated versions of one corpus partition present in the database; administrators are also allowed to upload a file to add to the confrontation. this can be seen in figure 2 3.1 system architecture matilda is designed as a web service: a browser hosts the user interface while the server supports data and control. our use case envisions all components running on user premises, but it is straightforward to distribute them on distinct hosts. on the server side, matilda is a bundle of two components: a web server and a database server. each of them is encapsulated in a docker, so that complex configurations are carried out by the designer and invisible to the non-technical enduser. in fact, matilda operation depends only on docker support, which is available for major operating systems. in order to have matilda operational, the end-user installs the docker support and launches a docker script that downloads and deploys on the user's pc the server-side dockers. matilda is then reachable from the browser at the url http://localhost/index.html . the tech-inclined user has the option to configure some features, like the use of an external database or the access through a network interface. the installation script and the operation manual are distributed on github https://github.com/wluper/matilda , while the dockers are available from https://hub. docker.com . as seen in figure 3 , the matilda engine is written in python using the flask framework, while the client-side javascript uses the vue framework. the mongodb database provides nosql access to the dialogs, the annotations and their metadata. this technology meets the required flexibility, allowing heterogeneous types of documents and an agile structure. the native support of json documents matches with the format used for the internal representation of the dialogs. finally, the availability of both an open-source server and a public service is useful when implementing either a service on-premises, according to the reference use-case, or, in a more advanced use-case, to implement a cloud database for sharing dialogs. the most stringent requirement on host hardware is that the processor must belong to the 64-bit family; this is inherited from docker. to analyse the footprint of matilda components, we installed it on a system based on the intel celeron j3355, a 2core microprocessor dated 2016, created for entry level desktop systems and with a 2gb ram. during a significant processing peak, induced with an upload, the footprint did not exceed a few percent of hardware capacity. the developer can find the engine source code in the github repository mentioned above; this allows them to customize or to add new features to matilda and to produce a new docker. localedependent information is recorded in an indepen-dent json document, and so introducing a different localization of the interface is non-intrusive (?). ## evaluation matilda was evaluated on two experiments: the first evaluated matilda's admin-related capabilities while and the second evaluated its annotation performance. both experiments were conducted across three different languages (english, italian and german) to assess matilda's cross-language adaptability. ## conclusion and future work we have presented matilda the first, to the best of our knowledge, multi-annotator, multi-language dialogue annotation tool that allows the user to annotate, distribute annotation work among crowdworkers or colleagues and to resolve annotation conflicts. we evaluate the tool based on the ease and rapidity of use and show that even untrained novices can quickly learn to use it. thanks to the open-source nature of the original lida project, we hope the community will pickup on this work both in terms of using it to create strongly needed corpora for different languages as well as extending it to allow even more use-cases and more advanced annotation styles. to this end we have conducted qualitative feedback sessions with study participants and provided a potential avenue of concrete improvements. we hope that this work will be a meaningful stepping stone for our community to create more useful resources in many languages.