Service registration chatbot: collecting and comparing dialogues from AMT workers and service’s users (2020)

Abstract: Crowdsourcing is the go-to solution for data collection and annotation in the context of NLP tasks. Nevertheless, crowdsourced data is noisy by nature; the source is often unknown and additional validation work is performed to guarantee the dataset’s quality. In this article, we compare two crowdsourcing sources on a dialogue paraphrasing task revolving around a chatbot service. We observe that workers hired on crowdsourcing platforms produce lexically poorer and less diverse rewrites than service users engaged voluntarily. Notably enough, on dialogue clarity and optimality, the two paraphrase sources’ human-perceived quality does not differ significantly. Furthermore, for the chatbot service, the combined crowdsourced data is enough to train a transformer-based Natural Language Generation (NLG) system. To enable similar services, we also release tools for collecting data and training the dialogue-act-based transformer-based NLG module.

## introduction task-specific neural dialogue models demand highquality annotated dialogue data. unfortunately, gathering human-generated and annotated dialogues is a costly and time-consuming task. easily accessible sources, like social-network feeds and online forums, are cursed by systematic problems such as extra-linguistic annotations, irregular turntaking, and the lack of a standard format leading to an intense pre-processing phase. even so, models trained with this type of data might not work well in a more natural domain @xcite . in recent times, thanks to online platforms like amazon mechanical turk (amt) foot_1 , crowdsourcing has become the most popular solution to tackle the problem of manually generating and annotating written dialogues. however, as a small business, minimizing such added costs while automating user-based workflows is essential. in this work, we consider leveraging voluntary submissions by business users for creating a chatbot. for the chatbot service, we consider a new class of broadly diffused tasks that we name service registration tasks (srts), which involves the domainagnostic act of registering to an online service. as a use case, we work with siirtosoitto foot_2 to provide users with a chatbot for service registration. si-irtosoitto is a free online service offered to the city of helsinki that notifies users about scheduled roadworks and imminent car towings. we employ a dialogue templating method called machine talking to machines (m2m) @xcite . it simulates the interaction between a user and system to automatically generate templates, which are then paraphrased by amt and service users. in this work, we make the following contributions. 1) we release the data collection tools to the public, including an integration with popular instant messaging platforms to engage with service's users (section 4). 2) we analyze and compare the data collected via amt workers and service's users in an empirical and human evaluation (section 5). ## service registration task (srt) here, we focus on a class of tasks named service registration tasks that consists of registering to a general online service. this human-machine interaction is characterized by the collection and val-idation of information and preferences from the user. as a specific instance of this class of tasks, we picked the use case of siirtosoitto, an online service that warns and notifies vehicle owners in the city of helsinki about road maintenance and imminent towings. ## machines talking to machines (m2m) for chatbot development, we employ the machine talking to machines (m2m) framework @xcite to setup the annotated data collection. conceived as being domain-independent, m2m generates dialogues centered on completing a specific task. the m2m consists of four major steps. 1), the developer provides the task-specific knowledge used by the system. it can be seen as a collection of all the units of information exchanged during the dialogue. 2) given a task specification, a simulated interaction of a user and the system generates sequences of dialogue acts exhaustively. the output sequences enclose the semantic content of the dialogue. the user is modeled as an agenda-based user simulator @xcite while the system is designed as a mealy machine. this process is also called self-play, where a simulated user interacts with the system. a generated example is shown in the first row of ## applying m2m to srt our srt is characterized by exchanging information such as telephone numbers, license plates, areas of interest, and the acceptance of terms and conditions. these characteristics form the taskspecification used to initialize the m2m's first step. a dialogue scenario is sampled by assigning a valid or invalid value to each entity. through self-play, we can generate sequences of dialogue acts until the goal of registering is reached or some invalid state is encountered (e.g., the user provides invalid values). next, we build a simple rule-based domain grammar that converts the anno- self-play annotations request(license plate), request (phone number) template utterance provide reference for: license plate and phone number paraphrase please list your license plates and your phone number in the next step, the same dialogues are used to set up a paraphrasing task on amt and on the rulebased chatbot that makes siirtosoitto available to the public. chatbot users are asked to participate voluntarily in an experimental task. they are presented with dialogue turns to rewrite sequentially on their preferred instant messaging application. a quick manual quality check removed roughly 25% of all amt feedback due to a lack of compliance with the instructions. in contrast only 10% of si-irtosoitto users failed to understand their task and produced unusable data. in the above process, instead of annotating natural utterances, we are building dialogues upon annotations. the automatic generation of the outlines guarantees greater diversity and explores all the relevant paths conceived by the task designer. finally, employing human writers ensures the naturalness of the utterances, and the variety is boosted by asking them to rephrase highly generic machinegenerated sentences. this reverse processing guarantees the quality of the semantic annotations. in in terms of effort, we set up the paraphrasing task on amt and the chatbot service in similar ## evaluation in this section, we compare the data collected via the two different crowdsourcing sources. we compare them quantitatively based on the lexical richness and language diversity. we also ask human evaluators to grade dialogues qualitatively. ## transformer-based language generator we train a neural model for natural language generation (nlg) to observe the effectiveness of the collected data. the neural model is a transformer network @xcite that converts the next dialogue acts into an output sentence. for the nlg use case, our transformer architecture includes two separate encoders. the first encoder inputs a sequence of dialogue acts capturing the semantic meaning of the sentence that needs to be generated. the second encoder inputs the user's turn. as a single person writes each paraphrase of an entire dialogue, the person's style is reflected in both user and system turns. intuitively, the second encoder employs the user's style to adapt the generated utterance to the user's persona. our trans- ## related work in our work, we applied m2m via two types of crowdsourcing methods. earlier work @xcite has shown that amt workers achieve significantly lower performances when the degree of experience and contextual knowledge is important. however, their performance improves with a more guided task structure. in our experiment, the service's users already had the background knowledge necessary for the task. moreover, considering the generated dialogue's lexical richness and diversity, their paraphrases were ranked higher than amt workers. however, at a qualitative level, both types of paraphrase ranked similarly. prior work @xcite has been concerned about the unnatural process of dialogue generation in the m2m approach. in our perspective, this issue affects scenarios where a simulated user cannot model the ambiguities of a real user, but for a simplistic srt use case, we disregard this issue. for creating the nlg module, we focus on the generation of surface expression based on sequences of dialogue acts. similarly, quite a few prior work @xcite @xcite @xcite have employed semantic structures to generate dialogue utterances. @xcite leveraged custom dialogue acts to implement a rule-based utterance generator as part of a bigger modular conversational system. recently, lstmbased machine translation models @xcite and transformers @xcite @xcite have also been successfully explored in nlg tasks for open-domain and task-specific dialogue systems. for both open-domain and task-specific modules, large corpora of annotations are required for training the modules. in contrast, our work considers a simple srt where even small amounts of crowdsourced data can help build good models. additionally, unlike most of the prior work, we release our nlg module code to the public. ## conclusions collecting annotated datasets for nlg is a challenging task which sees crowdsourcing as the preferred solution to balance costs and time. in this work, we considered voluntarily engaging siir-tosoitto's users to contribute towards a paraphrasing task for building a chatbot. our findings suggest that engaging siirtosoitto users might produce more diverse and lexically rich results than engaging amt workers empirically whereas, from a qualitative standpoint, both the datasets are similar for a simple service registration task. we can obtain similar amounts of data while running the data collection effort employing both sets of users for a comparable time. more importantly, through this process, we were able to reduce our costs of collecting data. additionally, in simple use cases like the srt, this data are enough to build a transformer-based nlg module conditioned on dialogue acts. to support other small businesses, we make our data collection pipeline and code to train the transformerbased nlg module public.