T ex P rax: A Messaging Application for Ethical, Real-time Data Collection and Annotation (2022)

Abstract: Collecting and annotating task-oriented dialog data is difficult, especially for highly specific domains that require expert knowledge. At the same time, informal communication channels such as instant messengers are increasingly being used at work. This has led to a lot of work-relevant information that is disseminated through those channels and needs to be post-processed manually by the employees. To alleviate this problem, we present TexPrax, a messaging system to collect and annotate _problems_, _causes_, and _solutions_ that occur in work-related chats. TexPrax uses a chatbot to directly engage the employees to provide lightweight annotations on their conversation and ease their documentation work. To comply with data privacy and security regulations, we use an end-to-end message encryption and give our users full control over their data which has various advantages over conventional annotation tools. We evaluate TexPrax in a user-study with German factory employees who ask their colleagues for solutions on problems that arise during their daily work. Overall, we collect 202 task-oriented German dialogues containing 1,027 sentences with sentence-level expert annotations. Our data analysis also reveals that real-world conversations frequently contain instances with code-switching, varying abbreviations for the same entity, and dialects which NLP systems should be able to handle.

## introduction the lack of annotated data-especially in languages other than english-is one of the key open challenges in task-oriented dialogue processing @xcite . this becomes even more challenging for very task-specific application domains with only a small number of experts that are sufficiently qualified to generate dialogue data or provide annotations @xcite . at the same time, using informal communication channels such as instant messengers at work has become increasingly popular @xcite . although this can accelerate troubleshooting, most of the knowledge that is communicated informally may be lost without an additional error tracking process; which in turn increases documenting work for employees @xcite . whereas this could be alleviated by nlp-based assistance systemsthat for instance automatically identify problems, their cause, and their solution-they cannot be built without any annotated data. our goal is to provide an application (texprax) to bridge the gap between the lack of annotated task-oriented dialogue data and the increasing need for nlp-based documenting assistance. figure 1 provides a high-level overview of tex-prax and all involved parties. our system communicates as a chatbot that acts as the user interface and recording service at the same time. for the server that hosts the messaging application, the bot appears as an additional user and hence, inherits all privileges and restrictions a user can have; including (1) reading any messages written in a chatroom, (2) being invited and removed by the chatroom moderator, and (3) being able to send messages in chatrooms it was invited to. we use privilege (3) to provide label suggestions from a pre-trained model and collect annotations via a reaction mechanism (figure 5b ); attaining a lightweight annotation process with minimal overhead. we also integrate texprax via the rest web api into an internal dashboard to automatically store recognized errors as a first step of the error documentation process. directly involving employees in data annotation and curation introduces four key advantages over previous approaches that involve crowdsourcing @xcite or use expert annotation tools such as inception @xcite . first, they are the very domain experts that hold qualified conversations which concern exactly the target-domain. this allows us to directly collect the dialog data instead of having to generate it semi-automatically or asking crowdworkers who can only provide limited expertise @xcite . second, the employees have an immediate benefit from annotating and improving the recommendation model as a dashboard integration saves time they would otherwise have to spend on documenting errors later on (hence, an intrinsic motivation). third, they have full control over their own data which saves time for nlp practitioners as it alleviates research data management. finally, the use of an end-to-end encryption protocol ensures that only parties selected by the employees will have access to the data even if the server is breached. foot_0 our contributions are: ## use case in this work, we focus on assisting employees on the shop floor (the production area in a factory). our goal is to improve shop floor management @xcite ; a systematic approach for solving processing problems. to efficiently solve such problems, shop floor management defines performance indicators which are used to detect deviations and identify problems which are also used to quantify their successful rectification. ## system description as shown in figure 1 , texprax involves three key parties: the users (employees), the chatbot and the server (e.g., hosted by a company). users communicate via chatrooms; each chatroom including at least two (for a private conversation) or more (for a group conversation) users. every message a user sends into a room can be read by any other user in the same room. the server is responsible for handling new incoming messages and the distribution of outgoing messages, as well as keeping track of currently active conversations and users. finally, the chatbot is responsible for monitoring conversations, suggesting labels, and storing the relevant data (locally or in an external database). ## data collection in contrast to our previous work that investigates expert-annotated named entity recognition @xcite , our goal is to provide a first solution for collecting annotated data and providing assistance with a minimal effort for users. we thus focus on sentence-level annotations that can be easily provided using message reactions and are suitable for existing shop floor management processes. ## experiments we conduct experiments to gain first insights on how well recent models can perform for providing label suggestions for our use case in future studies. ## conclusion we presented texprax, a system for collecting annotations and assisting employees by directly engaging them as domain-experts during their daily work. texprax allows users to exchange, modify, and delete end-to-end encrypted messages at any time, and an opt-in chatbot to ensure a high level of data privacy and security. we evaluate texprax in an assembly line at a learning factory (cip) where we find that daily work communication is noisy, but efficient and very problem-oriented. while existing models still have difficulties to provide the correct label suggestion, texprax still maintains a high usability. we conjecture that texprax could be especially beneficial to collect data and build assistance systems in domains with a high share of remote work, such as in software development. for future work, we plan to extend texprax to identify and suggest solutions for recognized problems and adapt it to new domains, such as our institute's reading group chat where researchers discuss papers relevant for their research.