Towards Efficient Dialogue Processing in the Emergency Response Domain (2023)

Abstract: In this paper we describe the task of adapting NLP models to dialogue processing in the emergency response domain. Our goal is to provide a recipe for building a system that performs dialogue act classification and domain-specific slot tagging while being efficient, flexible and robust. We show that adapter models Pfeiffer et al. (2020) perform well in the emergency response domain and benefit from additional dialogue context and speaker information. Comparing adapters to standard fine-tuned Transformer models we show that they achieve competitive results and can easily accommodate new tasks without significant memory increase since the base model can be shared between the adapters specializing on different tasks. We also address the problem of scarce annotations in the emergency response domain and evaluate different data augmentation techniques in a low-resource setting.

## introduction emergency response is a very challenging domain for nlp for a variety of reasons. first, this domain has strict requirements regarding memory and computational efficiency. often it is not feasible to load several large nlp models because of the limitations in the available infrastructure (e.g., memory of the machine where the models are running). second, the environment is often noisy and the speakers communicate using domain-specific lexicon and abbreviations. third, emergency situation environment is very changeable and the domain may vary from a rescue operation in a car accident to explosions or building collapse. hence, the ideal dialogue processing system for the emergency response domain should be memory efficient, robust and flexible at the same time. to address the efficiency aspect we use adapters foot_0 @xcite ) that were tested on a variety of nlp tasks and have shown a comparable performance with the full fine-tuning while using only 1% of the parameters of the fully fine-tuned models. adapters are small in size, can be easily shared and combined with different models. this is especially interesting in our use case since we deploy the same base model (bert-base-german-cased) for several tasks foot_1 . to tackle the problem of noisy, incomplete and domain-specific communication we investigate whether it is possible to boost the performance by integrating additional context and experiment with different ways of encoding it (e.g., by adding speaker, previous turn and dialogue summary information). we also experiment with various linguistic features and test how they affect the performance (e.g., by embedding the pos tags or including the iso-style dialogue act annotations). finally, to simulate the low-resource scenario which is very common for the emergency response domain we reduce the amount of the training and development data to 12% of the original dataset and apply different ways of data augmentation including backtranslation, lm-based word replacements and random edit operations. figure 1 provides an overview of different experimental settings addressed in this work. to our knowledge, this is the first work that explores dialogue processing in the emergency response domain with adapters and performs a comprehensive study of the context integration and data augmentation in this setting. ## related work adapters @xcite seem like a natural choice for lightweight and ef-figure 1 : overview of the experiments ficient nlp models. adapters implement a finetuning strategy that involves only a small amount of trainable parameters per task. each adapter adds a small set of newly initialized and trainable weights at each layer of the transformer architecture @xcite . hence, the original network has mostly fixed parameters and can be efficiently transferred between the tasks. adapters have shown good performance comparable to the fully fine-tuned models on a variety of tasks including, e.g., sentiment analysis, commonsense reasoning, paraphrase detection and entailment @xcite and further modifications and improvements to the original idea were proposed in the recent work by @xcite ; @xcite . adapters have been successfully used for low-resource speech recognition @xcite , cross-lingual transfer @xcite and tested on the named entity recognition and classification tasks @xcite . also, in the field of dialogue processing there is a growing body of work involving adapter models. for example @xcite inject knowledge into pre-trained language models using adapters and explore grounded dialogue response generation with adapters. another work by @xcite proposes a simple and efficient method based on residual adapters in the continual learning setting for task-oriented dialogue systems. @xcite design a gpt-adapter-copynet system that combines adapters and copynet modules into gpt-2 in order to perform transfer learning and dialogue entity generation. their system significantly outperforms the baselines models on both dstc8 and multiwoz data. efficiency and robustness are crucial in the lowresource setting when we have a limited amount of data. the main objective of data augmentation is to generate new data points by modifying the existing ones through a variety of transformations and while some of these transformations can be very simple such as random token deletion or insertion @xcite , others might require more computation and processing power, e.g., backtranslation @xcite or lmbased substitutions @xcite . @xcite and @xcite provide comprehensive surveys of the techniques and methods for data augmentation in nlp that served as a motivation for our work. ## data the dataset used in our experiments is based on the dialogues collected during several robotassisted disaster response training sessions @xcite . all dialogues are in german and they represent team communication between a team leader or mission commander and several operators who remotely operate robots in order to explore some area, find hazardous materials, locate fires, damage or victims. figure 2 shows a part of one dialogue translated into english. speaker original turn translation tl: ugv2 von teamleader. ugv2 for team leader. ugv: ugv2, kommen. ugv2, coming. tl: ja, ugv2, wir brauchen nochmal schärfere bilder von dem fass und der kennzeichnung. yes, ugv2, we need again sharper pictures of the barrel and the sign. ugv: ich habe sie nicht verstanden, können sie wiederholen? i didn't understand you, could you repeat? tl: ja, von dem fass brauchen wir nochmal bessere bilder, und auch von der kennzeichnung. yes, we need better pictures of the barrel, and also of the sign. the complete dataset contains 2,542 dialogue turns annotated with dialogue acts and domainspecific slots. for the dialogue act classification we reserve 2,261 turns for training, 281 turns for development and 283 for testing. in the low-resource setting we leave the test set unchanged but reduce the amount of the training samples to 310 (240 in training and 70 in development). figure 3 shows the overall distribution of different dialogue act labels in the data and figure 6 in the appendix provides an example for each label. there are seven main labels: call, callresponse, inforequest, infoprovide, confirm, disconfirm, order and the additional label other for the cases that do not fit in any of the main categories. the labels are derived based on the domain expertise and represent categories that are important for the emergency response domain. part of the dataset is also annotated according to the iso standard for dialogue act classification by @xcite figure 3 : dialogue act distribution and we use these fine-grained labels in some of the experiments described in section 4. in the emergency response domain it is very important to correctly recognize and annotate all deployment orders (einsatzbefehl in german). note that not every utterance classified as request according to the iso standard would qualify as order in our domain. e.g., the request "could you repeat, please?" is not a deployment order since it does not require performing a domain-specific action and should be classified as information request (inforequest). for each turn annotated as order we also perform the slot tagging. the slots are based on the regulation document of the emergency responders feuerwehr-dienstvorschrift (1999). we show an example containing all relevant order slots in figure 4. note that the distribution of slots is quite uneven (see figure 5 ). some slots are present in almost every dialogue turn classified as order (e.g., unit is present in 67% of the turns and task appears in 99% of them) while other slots are annotated only in 8% of the turns (way). also, the slots can be nested and the same token may belong to several slots. e.g., in "schickst du mir noch ein foto?" (will you send me also the photo?), "du" (you) is part of the slot task and also the slot unit. this is the reason why we train separate models for each slot and then combine the results to provide final annotations. for the slot tagging task we experiment with the full data as well as with the sampled data since the distribution of the negative versus positive instances per label varies a lot (see figure 5 for the details). for the sampled data we limit the amount of negative samples (turns without the slot annotafigure 5: slot distribution tion) to maximum 80% of the corresponding positive samples. our intuition is that having uneven distribution with too many negative samples may hinder the model's performance and it might be easier for the adapter model to learn the tagging task on more balanced data. we test this idea and describe our results in the next section. ## experiments our experiments aim to answer the following research questions: ## discussion our experiments show that adapter models can be successfully applied in a very specific and challenging domain such as emergency response. although fine-tuning bert gives a slightly better performance (0.80 vs. 0.79 f1 for the baseline), adapters are much more efficient in terms of memory and in the low-resource setting with 12% of the original data we find that adding backtranslated samples helps to improve the performance by up to 9 f1 points. however, multiple backtranslations are not necessarily useful and performance plateaus after one round of augmentation. lm-base word replacements and random edits can achieve similar performance but have a greater variance across the settings with different number of edits and augmentation rounds. the dialogue turn tokens have different relevance to the task in the emergency response domain and replacing words blindly may result in unrealistic or simply wrong instances. e.g., "kommen" (coming) has a specific meaning according to the communication protocol used by the responders and represents an instance of the callresponse class. replacing "kommen" with "gehen" (going) or another similar verb results in the wrong interpretation and should not be labeled as callresponse. in the future we would like to explore various constraints on the token substitutions and include more domain knowledge and ontology information to perform targeted replacements and edits. active learning for text classification @xcite is another approach that may work well in our domain. we have already shown that adapters benefit from balancing the data and it would be interesting to see whether they further improve by learning in stages when the model starts with he balanced dataset with easyto-classify labels and the difficulty level gradually increases with each epoch. also, in the future we would like to explore conditional text generation with the models like bart @xcite or t5 @xcite which can be trained to generate text given the corresponding label. ## limitations the main limitation of our work is the focus on the specific domain and the dataset that is not yet publicly available. however, we should note that the dataset can be requested for further research and replication studies and it will be released in the future.we believe that testing adapters with different settings in the emergency response domain is a valuable contribution but we are also aware of the fact that the dataset used in our experiments is not large or exhaustive enough to cover all the variety of topics relevant for the emergency response. for example, our data cover cases of explosions, leakages of hazardous materials and building collapse but do not include any dialogues for open field rescue operations or car accidents. another issue that is worth mentioning is the fact that all recordings were collected during the training sessions and not the actual missions. hence, the responders might be under less pressure than in a real life-threatening situation and their communication might be more of a textbook case. however, all simulations had a realistic setting that includes several operators, robots and points of interest (objects or locations) and we believe that the recorded communication is representative for the domain in question. ## conclusion in this work we evaluate the performance of several adapter models in the emergency response domain. we demonstrate that adapters show similar performance to the vanilla fine-tuned bert in the baseline setting (0.79 vs. 0.80 f1 score) while using only 1% of the parameters of the fully tuned model. our experiments show that including additional context such as previous turn and speaker can improve the performance by up to 6 points in f1 score. also adding linguistic annotations such as iso dialogue acts boosts the performance in dialogue act classification. the slot tagging task mostly benefits from the balanced data. as for the low-resource setting, it shows a substantial improvement over the baseline (9 f1 points) when a single round of backtranslated turns is added to the training set.