Information Extraction and Human-Robot Dialogue towards Real-life Tasks A Baseline Study with the M obile CS Dataset (2022)

Abstract: Recently, there have merged a class of taskoriented dialogue (TOD) datasets collected through Wizard-of-Oz simulated games. However, the Wizard-of-Oz data are in fact simulated data and thus are fundamentally different from real-life conversations, which are more noisy and casual. Recently, the SereTOD challenge is organized and releases the MobileCS dataset, which consists of real-world dialog transcripts between real users and customerservice staffs from China Mobile. Based on the MobileCS dataset, the SereTOD challenge has two tasks, not only evaluating the construction of the dialogue system itself, but also examining information extraction from dialog transcripts, which is crucial for building the knowledge base for TOD. This paper mainly presents a baseline study of the two tasks with the MobileCS dataset. We introduce how the two baselines are constructed, the problems encountered, and the results. We anticipate that the baselines can facilitate exciting future research to build human-robot dialogue systems for real-life tasks.

## introduction building human-robot dialogue systems is an important research question not only for artificial intelligence applications but also for artificial intelligence itself. in the turing test, if the human evaluator finds that human-robot dialogue and humanhuman dialogue are indistinguishable, the robot would be said to exhibit intelligent behaviour and pass the test @xcite . so presumably, the best strategy to build an intelligent dialogue system may be to train the system over a large amount of real human-to-human conversations to mimic human behaviors. this approach was once pursued and several human-human dialogue datasets have been released, such as the twitter dataset @xcite , the reddit conversations @xcite , and the ubuntu technical support corpus @xcite . it is argued in @xcite that the lack of grounding conversations onto an existing knowledge base (kb) limits the usability of the systems developed over these human-human dialogue datasets. so a class of wizard-of-oz simulated games have emerged to collect human-human conversations @xcite @xcite @xcite , particularly for task-oriented dialogue (tod) systems which help users accomplish specific goals such as finding restaurants or booking flights and usually require a task-related kb. in the wizard-of-oz set-up, through random sampling based on an ontology and a kb (both are pre-defined), a task template is created for each dialogue session between two crowd workers. one worker acts as the role of a user and the other performs the role of a clerk (i.e. the system side). in practice, multiple workers may contribute to one dialogue session. in this way, annotations of belief states and systems acts become easy, and grounding conversations onto the kb is realized. however, dialogue data collected in the wizardof-oz set-up are in fact simulated data and thus are fundamentally different from real-life conversations. during the wizard-of-oz collection, specific instructions (e.g., goal descriptions for the user side and task descriptions for the system side) are provided for crowd workers to follow. in contrast, real-life dialogues are more casual and freestyle, without instructions. even with some goals in mind, chit-chat or redundant turns are often exist in real-life conversations, e.g., asking for repeating or confirming key information. in some sense, we could say that real-life dialogues are more noisy. moreover, spoken conversations in real-world have a distinct style with those well-written conversations and are full of extra noise from grammatical errors, influences or barge-ins @xcite . for building dialogue systems that are more applicable to real-life tasks, real human-human dialogue datasets with grounding annotations to kbs are highly desirable. recently, the seretod challenge is organized @xcite and releases a new human-human dialogue dataset, called the mobilecs (mobile customer service) dataset. it consists of real-world dialog transcripts between real users and customerservice staffs from china mobile. based on the observation and analysis of those dialogue transcripts, a schema is summarized to our best foot_0 , according to which about 10,000 dialogues are annotated with entities, attribute triples and speaker's intents for every turn. the annotated part of the mobilecs dataset is randomly split into a train, development and test set, which consists of 8975, 1025 and 962 dialogues, respectively. based on the mobilecs dataset, the seretod challenge not only evaluates the construction of the dialogue system itself (task 2), but also examines information extraction from dialog transcripts (task 1), which is crucial for building the kb. the mobilecs data are more noisy and challenging, as compared to previous wizard-of-oz data. it is nontrivial to establish baseline systems on such dataset. this paper mainly presents a baseline study of the two tasks with the mobilecs dataset. two baseline systems are constructed for the two tasks respectively, which both are released as open source foot_1 and provided to the participating teams in the seretod challenge. we introduce how the two baselines are constructed, the problems encountered, and the results. the results clearly show the challenge for information extraction and human-robot dialogue, when trained and tested on real human-human data. we anticipate that the baselines can facilitate exciting future research to build human-robot dialogue systems for real-life tasks. ## mobilecs dialogue dataset the mobilecs dialogue dataset contains 10,000 dialogue labeled by crowd-sourcing and around 90,000 unlabeled dialogues. for each dialogue turn, the annotations consist of entities, attribute triples, and speaker's intents within the scope of the schema. another around 1000 dialogs are put aside as the test data. more detailed information about the mobilecs dataset can be found in the challenge description paper for the seretod challenge @xcite . the two tasks defined over the mobilecs dataset for the seretod challenge require different annotations. for information extraction (task 1), the annotations of entities and attribute triples are needed for training and evaluating the system. for tod system construction (task 2), user intents, system intents and a local knowledge base (local kb, which covers personal information and relevant public knowledge in a dialogue) are required. a global kb, which covers and fuses all public knowledge and all personal information in the mobilecs domain, is difficult to obtain during the research phase. thus, the seretod challenge introduces the concept of a local kb, which could be viewed as being composed of the relevant local snapshots from the global kb for each dialog. the local kb is obtained automatically by integrating all the annotations of entities and attributes into a sequence of entities foot_2 . besides, user goals are needed for evaluating the performance of tod systems in task 2. similarly, user goals are obtained automatically by integrating user intents and all the entities and attributes mentioned by the user. the examples of local kb and user goal can be found in listing 1 in the challenge description paper @xcite . ## discussion and conclusion the performance of task-oriented dialogue systems on wizard-of-oz datasets have been improved con-tinuously to a high level, for example, as shown in multiwoz 5 . however, wizard-of-oz dialogue data are in fact simulated data and thus are fundamentally different from real-life conversations, which are more noisy and casual. for further advancement of human-robot dialogue technology, real human-human dialogue data with grounding annotations to kbs are highly desirable. further, noting that the kb is an indispensable part for tod systems and usually is not readily available for reallife tasks, it is very important to investigate not only the dialogue system itself but also information extraction to construct the kb. with the mobilecs dataset released by the sere-tod challenge, this paper presents a baseline study of both information extraction (task 1) and humanrobot dialogue (task 2) over real human-human dialogue data. we introduce how the baselines for the two tasks are constructed, the problems encountered, and the results. it is found that the mobilecs dataset offers a challenging testbed for both tasks, with interesting open problems. our baselines provide an easy entry to investigate the new dataset, and we anticipate that the baselines can facilitate exciting future research to build human-robot dialogue systems for real-life tasks.