Building K orean Linguistic Resource for NLU Data Generation of Banking App CS Dialog System (2022)

Abstract: Natural language understanding (NLU) is integral to task-oriented dialog systems, but demands a considerable amount of annotated training data to increase the coverage of diverse utterances. In this study, we report the construction of a linguistic resource named FIAD (Financial Annotated Dataset) and its use to generate a Korean annotated training data for NLU in the banking customer service (CS) domain. By an empirical examination of a corpus of banking app reviews, we identified three linguistic patterns occurring in Korean request utterances: TOPIC (ENTITY, FEATURE), EVENT, and DISCOURSE MARKER. We represented them in LGGs (Local Grammar Graphs) to generate annotated data covering diverse intents and entities. To assess the practicality of the resource, we evaluate the performances of DIET-only (Intent: 0.91 /Topic [entity+feature]: 0.83), DIET+ HANBERT (I:0.94/T:0.85), DIET+ KoBERT (I:0.94/T:0.86), and DIET+ KorBERT (I:0.95/T:0.84) models trained on FIAD-generated data to extract various types of semantic items.

## introduction task-oriented dialogue (tod) systems are rapidly growing in high demand among various industries seeking ways of improving service quality and coping with customers effectively. tod systems primarily focus on helping users achieve specific purposes such as hotel accommodation, food order, and product recommendation. natural language understanding (nlu) technology plays a critical role in tod systems: it helps understanding the information conveyed by user utterances, classifying the intents and filling their slots. for example, in the utterance 카카오 뱅크 계좌 개설해 줘 (khakhao payngkhu kyeycwa kayselhay cwe) 'create a kakao bank account,' an nlu model has to classify the user's intent as 'request for creating a bank account', but also recognize the argument with the named entity 카카오 뱅크 'kakao bank', fill it in the 'bank entity' slot, and identify 'account' as an entity in banking service. nlu tasks such as intent classification and slot filling are usually implemented by supervised learning, requiring a large amount of annotated training data. however, it is hard to find publicly available korean training datasets for tod oriented nlu, due to privacy issues, and constructing them costs considerable time and human labor. in addition, since users produce diverse utterances with different terminologies and linguistic forms depending on domains, annotated training data should reflect that diversity in each domain. the performance of nlu models is likely to be directly affected by this aspect of the training dataset. moreover, as most of the user utterances in tod are meant to request specific information or actions, covering the linguistic patterns that occur in directives and questions is essential to increase the nlu coverage of diverse utterances. in korean, discourse markers used in requests frequently contain certain patterns with word endings and performative predicates, which can be realized in different moods, e.g. -해 줘/봐(hay cwe/pwa). in order to classify korean request utterances, such discourse expressions should be included in the training data. this study proposes a linguistic resource named financial annotated dataset (fiad), which allows for generating annotated training data in the banking cs domain. fiad specifies linguistic forms with extensive linguistic variations, and has been constructed on the basis of a corpus of ## data analysis since collecting users' dialogue data raises privacy issues, we collected a corpus of banking app reviews as alternative data. 126,598 banking app reviews were collected from appstore and playstore. on a scale of 1 to 5, we focused on reviews with a score of 3 or lower because low score reviews tend to include more users' requests or complaints on banking services than high score reviews. we used the mecab-ko korean morphological analyze foot_1 to split the collected reviews into morphemes. then, we extracted key morphemes, nouns, predicates, and inflectional endings using their tf-idf (term frequency-inverse document frequency) weight. based on the extracted keywords and on the observation of utterance patterns, we set a language resource in three parts: topic(entity, feature), event, and discourse-marker. each part has submodules which are separated based on their semantic content. all the modules and their sub-modules are organized as shown in fig. 2 . the module/submodule hierarchy means that the expressions specified by the submodule are a subset of those specified by the module. more details on this resource are provided in section 5. 3 resource construction 3.1 topic we divide the topic part of fiad into entity and feature for modularity and scalability. entity includes named entities which refer to product names and names of specific banks, and feature includes common nouns frequently used in reviews and dialogues about banking and financial app services. the modules and submodlues of entity and feature are displayed in tables 1 and 2 . the main role of topic is to fill slots. modules and submodules of entity and feature are used to set slots in the utterances. this allows researchers to change or add training data of a specific domain by selecting entity modules or submodules, such as banks' product names and bank names. for example, kakao bank card can be replaced with toss card by replacing @xmath0. ## conclusion this study presents a linguistic resource named fiad and its use to efficiently construct nlu training data for korean banking cs tod systems. fiad consists of lggs and comprises three parts: topic(entity, feature), event, and discourse-marker. each part contains modules dedicated to semantic categories, and generates utterances where intents and entity slots are annotated as belonging to these semantic categories. topic(entity, feature) generates expressions used as 'slots' of intents. event provides utterances with an intent and its arguments realized as slots, taking into account semantic restrictions. discourse-marker includes diverse mwes which represent speech acts, sorted by moods and speech styles, in compliance with the korean system of honorific 5 https://github.com/monologg/hanberttransformers 6 https://huggingface.co/monologg/kobert/t ree/main markers. these three parts are combined in four ways into four types of utterance patterns for the training data generation. the models have a clear benefit from the pretrained embeddings. the performance of the diet+korbert models trained on fiadgenerated data shows 0.86 and 0.95 f1-score in entity and intent extraction, respectively, showing that the concept is applicable. in addition, the modular structure of fiad offers flexibility for building extensive training data adapted to specific aims, by changing the order of the components or by selecting the modules to focus on particular types of entities, intents or speech styles.