Automatic Assignment of Semantic Frames in Disaster Response Team Communication Dialogues (2021)

Abstract: We investigate frame semantics as a meaning representation framework for team communication in a disaster response scenario. We focus on the automatic frame assignment and retrain PAFIBERT, which is one of the state-of-the-art frame classifiers, on English and German disaster response team communication data, obtaining accuracy around 90%. We examine the performance of both models and discuss their adjustments, such as sampling of additional training instances from an unrelated domain and adding extra lexical and discourse features to input token representations. We show that sampling has some positive effect on the German frame classifier, discuss an unexpected impact of extra features on the models’ behaviour and perform a careful error analysis.

## introduction in this paper we employ the theory of frame semantics as a meaning representation framework for dialogues from the domain of disaster response. our work is part of a larger research project developing methods to capture and interpret verbal team communication in disaster response scenarios and use the extracted run-time mission knowledge for mission process assistance, as described in @xcite . team communication interpretation encompasses several aspects, some of which have been addressed in earlier publications of our team: @xcite present dialogue act classification results; @xcite ) provide an analysis of contextual reference phenomena. the present paper complements this by results on semantic frame assignment. to our knowledge, our work is the first to use semantic frames in the domain of disaster response, and one of the few attempts implementing a frame classifier for dialogue. frame semantics is a paradigm defining the meaning of words through the context they are used in @xcite . this assumes that, depending on context, a word (or an expression) is able to evoke in our minds a certain event or situation together with a set of slots called frame elements associated with it, even if some of these slots were not explicitly filled in the sentence. using frame semantics as a meaning representation requires frame semantic parsing, namely identifying frame-evoking elements (targets) and the corresponding frames, as well as recognizing certain spans as frame elements and classifying them. in this paper we address the task of automatic semantic frame assignment, given a target. the novelty is that we work on english and german dialogues in robot-assisted disaster response teams. we use the tradr corpus @xcite , which contains transcribed communication in teams of firefighters using robots for incident site reconnaissance during a series of exercises that simulated situations after a disaster, such as a fire, explosion, etc. towards the aim of creating structured representations of the events and activities during a first response mission by means of semantic frames, we experiment with some existing models. we start with a simple sequence classification approach that assumes finetuning of a pretrained bert model @xcite on the tradr corpus. next, we use one of the existing state-of-the-art frame classifiers called pafibert @xcite . we re-implement and train it on the english framenet @xcite data, and evaluate the model on english tradr dialogues. we also experiment with retraining pafibert on the tradr data, despite the small corpus size. in addition, we investigate a possibility of training a frame classifier on mixed data -framenet or salsa @xcite plus tradr -and consider three sampling approaches. finally, we examine whether enriching the input with lexical and discourse features has an effect on the classifier performance. in contrast to many papers that report standard accuracy or f-score to measure the performance of a frame classifier, we use the index of balanced accuracy metric @xcite designed specifically for imbalanced data. in section 2 we give a brief overview of the theory of frame semantics. in section 3 we introduce the most notable frameworks designed to perform automatic frame assignment or frame-semantic parsing. in section 4 we examine the distribution of semantic frames and the role of ambiguous targets in the tradr corpus, compare our data with framenet and salsa, and explain how we prepared and split all the data into training, validation and test sets. section 5 describes the experiments and their results. in section 6 we make a conclusion and indicate possible further steps. ## frame semantics according to @xcite , frame semantics is a research program in empirical semantics which emphasizes the continuities between language and experience, and provides a framework for presenting the results of that research. the theory of frame semantics goes back to the 1970s. one of the pioneers in this area was charles j. fillmore. he suggested that a language description should include not only lexicon and grammar, but also a set of 'frames' that incorporate the semantics of the language elements @xcite . @xcite uses the word 'frame' as a general cover term for such concepts as 'schema', 'script', 'scenario', or 'cognitive model'. he defines a frame as a system of concepts which are related to each other, and states that one cannot understand a concept without understanding the whole structure it is a part of. frame semantics tries to describe and formalize such structures. the framenet project @xcite is considered one of the first practical realizations of the theory of frame semantics for english. one of its achievements was the creation of a lexical database that covers more than 13,000 word senses, is both human-and machine-readable and available online. besides, more than 200,000 sentences were annotated with about 1,200 semantic frames, and are now known as the framenet corpus. databases similar to framenet were also created for other languages. in section 4 we compare the framenet corpus and its german counterpart salsa with the tradr data. ## related work frame semantics is not one of the most common meaning representation frameworks. however, research in the area of frame-semantic parsing has increased since frame-semantic structure extraction was included as a task in semeval'07 @xcite . most of the existing works present models trained on text data. some of the projects deal only with automatic frame assignment, others have a bigger goal, namely, recognizing targets, frames and frame elements. in what follows we will focus on automatic frame assignment. most of the early frameworks are based on the idea of learning the frame labels from frameevoking targets represented as rather elaborated sets of features, which include the target's lemma, its part of speech, etc. many features rely on dependency syntax. for non-ambiguous targets a frame can be retrieved using a simple mapping. if the target is ambiguous, the correct label is learned using a naive bayes classifier, e.g., as shown by @xcite , or an svm classifier like in the framework called lth @xcite , or a discriminative probabilistic (log-linear) model like in semafor by @xcite . the success of neural networks for many nlp tasks resulted in a gradual switch from the featurebased approaches to embeddings and a broader usage of neural networks for the task of automatic frame assignment. one of the first semantic parsers to use embeddings was developed by @xcite . they represent targets as vectors, certain parts of which are reserved for certain argument representations. all frame labels are also vectors, and the classifier learns to minimize the distance between the targets and the correct labels. other frameworks based on embeddings and various types of neural networks include simple-frameid @xcite -a two-layer network which also allows to perform frame filtering using mappings of certain lexical units to certain frames from the framenet database; a framework by @xcite that performs frame identification using a simple multi-layer network; tsabcnn @xcite , which uses word2vec embeddings and convolutional neural networks. recently, there appeared frameworks that rely on bert embeddings and pretrained models. e.g., pafibert @xcite fine-tunes the pretrained bert model using an attention mechanism to give weights to words that make up the context of the target. an interesting alternative approach was presented by @xcite . they interpret frame-semantic parsing as a sequence-tosequence generation problem. their approach is based on the encoder-decoder architecture, namely on the t5 model, which is available via the hug-gingface library @xcite . @xcite treat automatic frame assignment as a clustering problem. they focus on verbal frame-evoking targets and represent them using contextualized elmo embeddings. the targets are treated as nodes in a graph, and clustered using the chinese whispers algorithm @xcite . a new instance is classified by determining the closest cluster. all the above frameworks were trained on text data. we found only two frame-semantic parsers designed specifically for dialogue. one of them was created in the course of the luna project @xcite and focuses mostly on frame element classification @xcite . the other was presented by @xcite . its main goal is actually to speed up the manual annotation process, not pure frame-semantic parsing. frames are detected with the help of a hand-crafted set of lexical triggers, which includes 200 most frequent words from 7 domains. a comparison of the frameworks mentioned above, as well as the results of their evaluation on the test data can be found in appendix d. we do not place them here for space reasons. for the experiments on the tradr data presented in this paper we have chosen the pafi-bert approach @xcite . pafibert is one of the state-of-the-art frame classifiers, it showed about 89% accuracy when evaluated on the framenet test set, and it is easy to re-implement. ## data for experiments the tradr corpus consists of 15 files with dialogues, six files contain dialogues in english, and nine -in german. six german dialogues were translated into english in order to get more english training data. tradr dialogues comprise the communication in first responder teams using robots for disaster site reconnaissance. each team consists of several operators (op) who control ground and airborne robots, a team leader (tl) and sometimes also a mission commander (mc). the english tradr data counts 434 different lus. their distribution is also not uniform: the top ten most common lus occur in about 40% of all the utterances and at the same time make only slightly more than 2% of the total of different lus. all lus are distributed between seven different pos tags. 75% of the utterances contain verbal targets. the second frequent pos tag is an interjection -almost 8% of all the targets. only about 15% of all lus in english tradr are ambiguous. however, they are realized in nearly 53% of utterances containing targets. simple calculations show that on average a single lu evokes 1.24 frames. so, while the ambiguous lus are not very frequent in comparison to non-ambiguous ones, the frames that they evoke are frequent, and this may become a problem for the frame classifier, as it is not always possible to perform frame disambiguation using the utterance context. besides tradr we also use the framenet and salsa datasets for our experiments, so it is necessary to compare them with our data. the differences between the corpora are summarized in the framenet and salsa data are very different from tradr, cf. ## experiments and discussion in this section we will present semantic frame classifiers for both english and german tradr dialogues. our main focus is on the english data. we introduce several models, split into basic and adjusted, and discuss their performance. as all our datasets have hundreds of classes and are highly imbalanced, many typical performance metrics, e.g., accuracy, precision, f-score, are not reliable @xcite . instead, we use the index of balanced accuracy (iba) metric as our main performance measure, calculated using the python imbalanced-learn package @xcite . the package also outputs the scores of the common metrics, such as recall, precision and f-score, and we show them for the sake of comparison, as most papers on automatic frame assignment report either accuracy, or these metrics. all the metrics are calculated using macro-averaging. ## conclusion and future work we investigated the potential of frame semantics as a meaning representation framework for english and german dialogues in the domain of robotassisted disaster response team communication. we found semantic frames convenient for capturing the meaning of an utterance depending on the target -the approach is span-based and does not require complex data annotation or pre-processing. we reused the pafibert model on the tradr data and achieved an iba score of 88%-90% on the test sets. our results are comparable with those reported by @xcite , who trained their models on the much larger framenet corpus. however, being a powerful model, pafibert memorized the small tradr training data, leading to overfitting and thus lack of generalization. we also studied the impact of sampling additional training instances from an unrelated domain on the classifier's performance, and found that it was useful only for the german frame classifier. error analysis indicates that sampling is beneficial for handling silly errors, but rather ineffective for cases that require disambiguation. we did not perform any experiments with over-and/or undersampling which imply sampling from the original dataset and are often used with imbalanced data. this can be a subject for further research. especially interesting is an approach that assumes generating synthetic training instances, e.g., embeddings incorporating the targets with their contexts. in contrast to our expectations, both lexical and discourse features failed to demonstrate a positive influence on the models' performance. error analysis showed that the largest group of errors is due to ambiguous targets, many of which evoke semantically close frames. the problem of disambiguation requires more research in order to improve the performance of the models.