Annobot: Platform for Annotating and Creating Datasets through Conversation with a Chatbot (2020)

Abstract: In this paper, we introduce Annobot: a platform for annotating and creating datasets through conversation with a chatbot. This natural form of interaction has allowed us to create a more accessible and flexible interface, especially for mobile devices. Our solution has a wide range of applications such as data labelling for binary, multi-class/label classification tasks, preparing data for regression problems, or creating sets for issues such as machine translation, question answering or text summarization. Additional features include pre-annotation, active sampling, online learning and real-time inter-annotator agreement. The system is integrated with the popular messaging platform: Facebook Messanger. Usability experiment showed the advantages of the proposed platform compared to other labelling tools. The source code of Annobot is available under the GNU LGPL license at https://github.com/rafalposwiata/annobot .

## introduction the basis of any machine learning model is data. in the case of supervised solutions, such data must be labelled, very often manually, especially if problems are closely related to human interpretation such as emotion classification @xcite or hate/offensive language detection @xcite . the preparation of such data sets is most often carried out using the crowdsourcing platforms such as figure eight (earlier called crowdflower) 1 or amazon mechanical turk 2 . these platforms undoubtedly have many advantages, but they are paid solutions that not everyone can afford. furthermore, due to their commercial nature, no changes or modifications can be made on your own. there are also open-source solutions on the market, but you have to organize a group of labellers yourself. another labelling issue is the lack of an intuitive, flexible, well-known, and responsive user interface. taking into account the above and the fact that chatbot technology has recently become more widespread, we decided to create the annobot platform. this is the first open-sourced platform for annotating and creating datasets through conversation with a chatbot. this natural form of interaction provides an interface that meets all criteria, while integration with the messaging application allows for more efficient activation of users. the rest of the paper is organized as follows. section 2 briefly describes open-source text annotation systems developed so far. section 3 shows an overview of our platform and its detail functions. experiment for showing the effectiveness of the created system is presented in section 4. next section presents possible applications and impact. finally, section 6 concludes this paper. ## related work if we trace the history of labelling tools foot_1 , most of them have been created for sequence tagging @xcite @xcite @xcite . recent years have brought tools for other purposes such as annotating dialogues @xcite , documents @xcite , and other types of data @xcite . when considering the user interface, most tools offer gui (all mentioned before), although there are also alternative approaches such as command line @xcite . what's worth mentioning is that the latest tools claim to support mobile versions @xcite @xcite and have more and more innovative functionalities, here we can mention alpacatag, which has, e.g. active, intelligent recommendations or automatic crowd consolidation module. our solution is in line with current trends related to support for mobile devices and advanced additional functionalities (e.g. active sampling). however, we have applied a different form of interaction than in other tools, namely cui (conversational user interface). our tool is integrated with a platform with billions of users (facebook) and has a very wide list of possible applications resulting from the cui. ## applications the annobot has many applications, such as data labelling for binary, multi-class/label classification tasks, preparing data for regression problems, or creating sets for issues such as machine translation, question answering or text summarization. if we are talking about the type of data: these can be short texts, sentences or tweets, as well as longer documents. potential recipients of the platform may be researchers (not only in the field of computer science, but, e.g. sociologists or psychologists who can create surveys with the help of annobot), companies that want to accelerate the development and improve their systems/models by making them accessible to the world, or ordinary people for whom teaching chatbot, e.g. to recognize what hate speech is, can be a form of participation in the development of ai, which has benefits for society. ## conclusion in this paper, we propose the annobot platform. we believe that our solution offers a lot of possibilities resulting from the form of interaction we have adopted. it is possible, for example, to introduce an element of "curiosity" of the bot, e.g., by asking about the reason for this and not another decision of the labeller. in the future, we would like to integrate our platform with slack foot_2 communication platform.