D ia B iz. K om - towards a P olish Dialogue Act Corpus Based on ISO 24617-2 Standard (2022)

Abstract: This article presents the specification and evaluation of DiaBiz.Kom – the corpus of dialogue texts in Polish. The corpus contains transcriptions of telephone conversations conducted according to a prepared scenario. The transcripts of conversations have been manually annotated with a layer of information concerning communicative functions. DiaBiz.Kom is the first corpus of this type prepared for the Polish language and will be used to develop a system of dialog analysis and modules for creating advanced chatbots.

## introduction the rationale of the current research was predominantly connected with the lack of corpora including dialogue texts in polish which could be used to train artificial intelligence for model creation. in order to bridge this gap, we decided to create a corpus that satisfies our expectations i.e. the one that contains dialogue samples from several different fields of business and is annotated for information concerning pragmatic functions. at the first stage of our work, we analysed the approaches adopted by other researchers, whose solutions are described in the "related works" section. further, we describe the process of data collection and its manual annotation. a separate section is dedicated to "key assumptions and limitations of the guidelines". the paper ends with "corpus overview" and "conclusions and future works". ## related work while creating the polish corpus of dialogue acts, we analyzed some pre-existing corpora. in theoretical perspective we refer to the iso/dis 24617-2, standardized annotations and the recommendations from the mentioned document. the iso standard @xcite is based on particular innovations such as distinguishing between annotations and representations (according to @xcite and sets of dialogue participants, dimensions, communicative functions, functional segments and qualifiers (inventory of diaml). both manual and automatic annotation of dialogue segments are possible according to the iso document and both have been tested in practice and described @xcite @xcite @xcite . dialog-bank is a language resource containing dialogues with gold standard annotations corresponding with the iso 24617-2 standard @xcite . the development of annotation standards for particular corpora can be vividly exemplified by the case of the switchboard dialogue act corpus (the collection of telephone conversations). telephone speech corpus (ldc97s62) was originally collected by texas instruments in 1990-1 and consists of approximately 260 hours of speech. the first release of the corpus was published in 1992-3 foot_0 . initially, the utterances in the corpus were annotated according to the damsl scheme for dialogue act analysis. subsequently, nxt-format switchboard corpus was created with additional annotations according to an international standard iso 64217-2:2012 @xcite . conversion of one annotation system to another required matching of tags between them: dams consists of 59 combine tags while iso -of 56 core tags. the re-annotation shows the significance of both standard scheme improvement and combining different standards on the same linguistic material. another work addressing the creation of corpora of dialogue acts concerns the dbox corpus, created within the dbox project and aimed at developing an interactive question-answering dialogue system @xcite . a more practical application of the project was to develop an interactive system used in computer games in three european languages. the authors collected 338 dialogues incorporating the continuous data collection method, i.e. they initially used the so-called wizard-of-oz paradigm with a human wizard mirroring the system's behavior, and later replaced the wizard with a complex dialogue system. a similar approach was adopted by the authors of the adele corpus of dyadic social text @xcite who created a corpus consisting of 193 dialogues resumed with the purpose of initiating interactions with other people. correspondingly to the dbox corpus, the adele corpus was predominantly constructed with the view of training a spoken dialogue system that could easily engage in a conversation during a role-playing computer game. both in the case of dbox and adele, the obtained dialogues were manually annotated with dialogue act information in accordance with the iso 24617-2 dialogue act annotation scheme, which was supplemented with additional dimension (for dbox) as well as several additional dimension-specific functions and general-purpose functions (for both corpora). other related works which are worth mentioning include the italian luna human-human corpus, which is a collection of 572 dialogues in the hardware/software helpdesk domain. the dialogues are conversations of the users engaged in problem solving tasks; a subset of 50 dialogues was annotated with the use of dialogues acts. furthermore, the diabiz.kom corpus correlates with the dialogbank corpus, which is mentioned as the current golden annotation standard. most dialogues from the dialogbank corpus were taken from other corpora and re-segmented and reannotated. all annotations were double-checked for inconsistencies, errors and omissions. the data include samples which may be considered illustrative examples for annotations @xcite . what is noteworthy here is the fact that suggestions and remarks with regard to limitations and extensions of the iso standard put forth by the authors of the dialogbank are often subsequently implemented in the updated versions of iso @xcite ). another point of reference was the corpus of vietnamese data using sources from iarpa babel vietnamese language pack @xcite . the corpus includes 28 selected conversations whose transcripts were manually segmented in turns and then annotated. the agreement scores is 0.84 fleiss'kappa measure. in comparison to the previously collected corpora diabiz.kom is much more extended in terms of the number of dialogues, and it covers different fields of communication. all the data were deliberately created to adhere to the research purposes and practical applications. as a consequence, diabiz.kom could be considered the only corpus which is to be used in all main business communication fields. also, especially in comparison to switchboard dialogue act corpus, the dia-biz.kom corpus uses much more up-to-date language materials. over the last 30 years the languages have been vastly influenced by overwhelming technological development especially by social networks that have severely modified communication strategies and behaviours. the innovation of our approach is based mainly on the detailed consideration of the mutual influence of dialogue dimensions and communicative functions, as well as on the designation of the new functions not included in the previously used standards. finally, diabiz.kom was not only fully manually annotated, but also verified in the agreement procedure, which enhances the credibility of the corpus. ## data diabiz.kom corpus development is an annotation effort performed simultaneously with diabiz corpus creation @xcite . diabiz is a large, multi-modal corpus of polish telephone conversations conducted in varied business settings, comprising 3,766 call center interactions from eight different domains, i.e. banking, energy services, telecommunication, insurance, medical care, debt collection, tourism and car rental. the phone-call interactions were based on 110 distinct customer service call scripts. they were then transcribed and enriched with punctuation. the selected dialogues from diabiz corpus are the basis for diabiz.kom annotation. ## annotation procedure in the first place, the annotations included communicative functions and dimensions. the annotation process was divided into two main stages: (1) initial phase and (2) the final annotation of dia-biz.kom corpus. both stages were performed by a team of qualified linguists with the use of the inforex system @xcite . initially, the first version of the annotation guidelines was developed with an aim of achieving an appropriate level of inter-annotator agreement. in order to ensure high data quality, we have performed several iterations of manual annotation prior to the annotations performed on the final corpus. three main sources were successively used as a dialogue base for manual annotation: luna corpus, samples of real-life data, and test sample from diabiz corpus. moreover, the team of linguists was systematically expanded, so that we received feedback from annotators not involved in the early stages of guideline development. this was done to avoid a situation in which many of the rules of conduct were not verbalized, but rather were based on the annotator's practical experience. all these efforts aimed at making the guidelines as complete as possible. we calculated inter-annotator agreement by applying positive specific agreement measure @xcite . the first stage was continued until achieving the satisfactory level of the inter-annotator agreement, which involved 8 iterations of manual annotation. the second stage (the final annotation of dia-biz.kom corpus) is currently underway. the interannotator agreement is constantly monitored and remains high. the figure presents the improvement of the average level of inter-annotator agreement. it is currently at the level of 0.78 (for annotation borders and categories) and 0.86 (for annotation borders). every dialogue included in diabiz.kom corpus is annotated by 3 specialists: 2 independently working annotators and a super-annotator who resolves all annotation inconsistencies (for current number of annotations see appendix a, during the two stages of annotation, we used essentially the same annotation categories (i.e., those specified in the iso 24617-2 standard). the main difference between the two stages was that during the first annotation stage we annotated a greater variety of texts, coming from diverse sources but greatly resembling the target texts which were later annotated at the second stage. dividing the processes into stages allowed us to test the model in a variety of domains. thanks to this solution, we did not adjust the guidelines to data acquired or produced in one specific way. furthermore, during the first annotation stage the agreement level between annotators was not particularly high. in order to improve the inter-annotator agreement, we decided to work on texts coming from other sources than the target corpus. as a result, the diabiz.kom annotation was quite consistent from the very beginning, and the need for corrections for the first iterations was significantly reduced (the second phase consisted of five iterations). once the annotation was established (in joint discussions of professional linguists), the super-annotators returned to the previously annotated documents. the correctness of the texts was additionally verified at the dimension marking stage, and in the future -it will also be double-checked at the relation marking stage. consequently, the material will be verified several times with a small chance of guidelines misinterpretations. ## key assumptions and limitations of the guidelines even though the annotation guidelines were constantly developed throughout the project, we decided to follow a set of certain unchanging assumptions. the increasing annotator agreement was the result of new specifications that were successively added to the guidelines. importantly, we were persistently mindful of the versatility of the guidelines, which was primarily aimed at facilitating various possible applications of the corpus in the future. this approach, however, also imposed certain limitations on our work. below we present the main assumptions as well as some selected issues which we encountered. one of the main assumptions involved the choice of the communication function for a given utterance as primarily influenced by its goal, effect and the context in which it is set. the form of the annotated statement is considered less importantit may lead to the proper function, but it cannot fully determine its choice. the above mentioned situations may be illustrated with the following examples. formally, the utterance in (a) points to be interpreted as questions, but due to its conventionalized form and structuring role in the dialogue, it is marked as interaction structuring. further, when there is a discrepancy between the intention and the effect (reaction), as illustrated in (b), we assign the specific function on the basis of the direct reaction. expressions that can naturally perform different functions depending on the context (e.g. lexemes, such as dobrze 'well', tak 'yes') have been approached more thoroughly in our guidelines, which presently include specific contexts alongside with the plethora of examples illustrating their use in a given function. the goal that the sender wants to achieve is a key criterion here. if the interlocutor's utterance is aimed at obtaining or transmitting some information, it is assigned an appropriate function from the information-transfer group, even if the form of this statement may initially indicate a function belonging to the action-discussion group (c) and (d). the agent wants to obtain some information from the client, and the usage of the word proszę ('please') is only meant to make the question more polite. what also needs to be emphasized is that due to the nature of the annotated dialogues, some of the functions described in the iso/dis 24617-2 standard were not used (e.g. functions from the turn management group), although they were included in the guidelines. as a result, it will be possible to apply them also to other types of dialogues in the future. the nature of the dialogues is related to the difficulty of the texts and this is also expressed by the degree of agreement between annotators (see appendix a, ## corpus overview the aim is to develop a well balanced corpus of annotated dialogues. thus, we decided to annotate 10 dialogues for each script. as a result di-abiz.kom corpus will consist of 1100 annotated dialogues: 260 for banking domain, 150 for energy services, 180 for telecommunication, 110 for insurance, 140 for medical care, 100 for debt collection, 100 for tourism and 60 for car rental. the annotation process continues. all the dialogues (for current statistics see the there are 138.968 annotated functional segments within diabiz.kom at this stage (see appendix a, agent: dzień dobry. ('good morning.') [initgreeting, dimension: social obligations management] client: tak, słucham.'yes, i'm listening.' [contactindication, dimension: contact management] all these decisions were preceded by a number of joint discussions of professional linguists over specifically extracted samples from the target corpus (i.e., the examples that had the potential to fall into the category of turn management functions). also, it is significant to mention that turn management functions are more natural to polylogues and the annotated corpus consisted solely of dialogues. ## conclusions and future work in this paper we have outlined diabiz.kom -the first corpus, which contains dialogues of various domains with gold standard dialogue act annotations in the polish language to satisfy the criteria set by machine learning applications. a crucial feature of this resource is the manual layer annotation of information about communication functions (based on iso standard). the achieved inter-annotator agreement provides a way to use the corpus for the purpose of machine learning. further development work on diabiz.kom will aim at adding annotation layers -especially those that specify the communicative intent of the speaker (using frame semantics) -and, subsequently, those that determine parameters congruent with the iso standard (communicative dimensions and relations between annotations). the next step consists in expanding the existing corpus with supplementary dialogues using active learning techniques.