Low Resource Causal Event Detection from Biomedical Literature (2022)

Abstract: Recognizing causal precedence relations among the chemical interactions in biomedical literature is crucial to understanding the underlying biological mechanisms. However, detecting such causal relation can be hard because: (1) many times, such causal relations among events are not explicitly expressed by certain phrases but implicitly implied by very diverse expressions in the text, and (2) annotating such causal relation detection datasets requires considerable expert knowledge and effort. In this paper, we propose a strategy to address both challenges by training neural models with in-domain pre-training and knowledge distillation. We show that, by using very limited amount of labeled data, and sufficient amount of unlabeled data, the neural models outperform previous baselines on the causal precedence detection task, and are ten times faster at inference compared to the BERT base model.

## introduction since 2011, more than one million new articles are added to pubmed every year @xcite . the growth rate of newly published articles makes it hard to keep up with the important discoveries just by reading them. therefore, tremendous efforts have been made to automate knowledge discovery from biomedical papers by extracting the biochemical events described in the literature @xcite @xcite . in addition to the extraction of the biochemical events, there are existing efforts to detect the causal relationships among them @xcite , i.e., whether the occurrence of one event necessarily leads to the occurrence of another event. knowing the causal precedence order of the events helps to describe more accurately the underlying mechanisms of biological processes described on the scientific literature. however, annotating such causal event pairs requires significant domain expertise and effort @xcite . in this work, we investigate multiple strategies for improving the detection of causal precedence relations within biochemical events. the contributions of this paper are the following: ## related work the detection of causal precedence among chemical interactions from text is a long-standing problem. early methods include rule-based approaches @xcite and machine learning-based approaches @xcite @xcite . other work @xcite @xcite have also explored the combination of rulebased methods, machine learning-based or neuralbased methods. recently, large pre-trained language models (llm) have increased the state-of-the-art performance of many natural language processing tasks @xcite @xcite . however, such llm models require enormous computational resources, making it hard to deploy them in many applications. one popular approach to reduce the memory footprint of llms is distillation @xcite @xcite . distillation trains a relatively small model to imitate the behavior of a larger model, such as a llm, trading off performance for a significant reduction in the amount of parameters. tang et al. (2019b) shows that it is possible to distill bert-large to a task-specific compact lstm model with approximately 1 300 of the model's original parameters while maintaining a comparable performance. @xcite and @xcite investigated whether the performance of the distilled model depends on the nature of the task and the size of the student model. @xcite found that the general domain pre-training of the compact model is essential and helpful to the distillation on the downstream tasks. in addition, various data augmentation techniques are proposed to improve the distillation process with very limited labeled training data @xcite @xcite . finally, several works explored whether cross-task distillation helps the compact models to learn more robust representations @xcite . to the best of our knowledge, this work is the first to investigate model distillation specifically for the task of causal precedence detection in the biomedical domain. ## dataset we use of a dataset of causal precedence annotations of biochemical interactions @xcite . the dataset contains 858 interaction pairs. each pair is annotated with one of three classes: e1 precedes e2, e2 precedes e1, and no precedence relationship, with 109, 27 and 722 instances, respectively. working with this dataset presents multiple challenges. firstly, it's small, with only total of 858 annotated examples. the scarcity of training data is a challenge for a model with a relatively large number of parameters to pick up training signal from the data. second, prediction of some examples requires more than the shallow understanding of linguistic knowledge (i.e., understanding the phrases such as "leading to"), and also requires understanding the underlying mechanistic process described in the phrase. for example, in the last row of ## conclusion in this work we trained several neural models for causal precedence detection in the biomedical literature. to help with the deployment of neural models on systems without a gpu, we restricted the sizes of our architectures to approximately 1 20 of the size of a state-of-the-art language model such as bert. moreover, to overcome the challenge of scarcity of labeled training data, we used in-domain unlabeled data combined with pre-training and distillation and obtained robust neural models. finally, we compared our neural models with previous rulebased and feature-based classifiers and found the in-domain pre-trained models can mostly replace them.