EVI : Multilingual Spoken Dialogue Tasks and Dataset for Knowledge-Based Enrolment, Verification, and Identification (2022)

Abstract: Knowledge-based authentication is crucial for task-oriented spoken dialogue systems that offer personalised and privacy-focused services. Such systems should be able to enrol (E), verify (V), and identify (I) new and recurring users based on their personal information, e.g. postcode, name, and date of birth. In this work, we formalise the three authentication tasks and their evaluation protocols, and we present EVI, a challenging spoken multilingual dataset with 5,506 dialogues in English, Polish, and French. Our proposed models set the first competitive benchmarks, explore the challenges of multilingual natural language processing of spoken dialogue, and set directions for future research.

## introduction computer systems need to be able to identify and verify their users before granting access to personalised services and confidential information @xcite . in particular, identification (i) is the process of specifying the identity of a person, i.e. answer the question: "who are you?". on the other hand, verification (v) (aka authentication) is the process of confirming the assertion about a claimed identity, i.e. answer "are you who you claim you are?" @xcite . in both processes, the system compares information given by the user with information held by the system; thus they presume enrolment (e), that is, the process of registering the identity information of a new user into the system @xcite . task-oriented dialogue systems that offer personalised and privacy-focused services (e.g. set up utilities, track a parcel, or access a bank account) should be able to enrol, identify, and verify new and recurring users, without interrupting their natural conversational interface. different types of authentication factors may be used @xcite : i) knowledge-based ("what you know"), rely on a secret password or personal information, e.g. full name, date of birth, mother's maiden name, etc.; ii) possession-based ("what you have"), rely on possession of a physical token, e.g. a smart card, a metal key, etc.; and iii) inherence-based ("who you are"), typically rely on biometric properties, e.g., a voiceprint, fingerprint, eye scan, or signature @xcite . most businesses use knowledge-based authentication in their call centres to identify customers over the phone @xcite @xcite . as conversational ai is increasingly being used to automate call centres, we seek to enable task-oriented spoken dialogue systems with evi functionalities. the core contributions of this paper are: 1. we motivate and formalise knowledge-based enrolment, verification, and identification as novel tasks for task-oriented spoken dialogue systems (section 2). verification task. the goal of verification is to decide whether a user who claims an identity is genuine or an impostor. for dialogue-based, knowledgebased verification, the system must be able to compare information stored in the kb about the claimed identity with information provided by the user in the dialogue to produce a verification score that quantifies the degree of the match (cf.fig. 1 ): the system designer can apply a threshold, θ, to obtain a crisp verification outcome and control the system's trade-off between security and usability (see later subsections 4.3 and 4.5). identification task. the goal of identification is to determine the identity of an unknown user from a kb of pre-enrolled user profiles. for dialoguebased, knowledge-base identification, the system must be able to query the kb with the information provided by the user in the dialogue to retrieve a ranked list of the best matching profiles (cf. fig. 1 ): the list might be empty if no qualifying profiles (i.e. above a score threshold) could be retrieved. ## a multilingual spoken dialogue dataset we set out to build a novel, first of its kind, humanto-machine conversational dataset that can be used to develop and evaluate task-oriented spoken dialogue systems that support the functionality of the knowledge-based evi tasks. the dataset is multilingual and covers 3 locales: british english (en-gb), french (fr-fr), and polish (pl-pl). foot_0 ## evi-oriented spoken dialogue systems this section presents the components of taskoriented spoken dialogue systems for evi tasks and provides benchmark implementations for the upcoming experiments (see sections 5.1, 5.2, and 5.3) 4.1 components of evi dialogue systems automatic speech recognition (asr). when collecting the evi dataset, we used google's locale-specific speech-to-text foot_6 in streaming mode to derive n-best transcriptions and to implement quality control (see subsection 3.2). consequently, this is the asr used in all experiments. the length of the n-best lists was on average 4.85, 2.60, and 8.65 for english, polish, and french, respectively (see appendix b) and was capped at a maximum of 20 items. ## experiments and results this section evaluates benchmarks and empirically explores the unique challenges of each evi task. ## conclusion we introduced novel spoken-dialogue tasks (knowledge-based enrolment, verification, and identification), the evi multi-lingual spoken-dialogue dataset with 5,506 dialogues, and benchmark models, evaluations, and upper-performance bounds that leave ample margins for future improvements. limitations. during data collection, our policy (fixed-length with reprompts for all items) might have caused artefacts in speaker behaviour (e.g. frustration, chuckling, simplification for later items). additionally, speaker behaviour of crowdsourced speakers who impersonate a fake profile will be qualitatively different to presenting one's own personal information (e.g. a young female speaker might be asked to impersonate an older male profile); however, ethical and privacy concerns preclude the publication of a dataset with real data. finally, our current evaluation considers each downstream task in isolation, although in practice they form a sequence (enrolment, identification, and then verification) that may propagate errors. future work. we invite the community to work on the novel evi tasks and challenge dataset, which pose a variety of unresolved technical challenges: speech recognition, multi-turn spoken language understanding, fuzzy matching and retrieval, etc.