SAFFRON : tran S fer le A rning For Food-disease R elati O n extractio N (2021)

Abstract: The accelerating growth of big data in the biomedical domain, with an endless amount of electronic health records and more than 30 million citations and abstracts in PubMed, introduces the need for automatic structuring of textual biomedical data. In this paper, we develop a method for detecting relations between food and disease entities from raw text. Due to the lack of annotated data on food with respect to health, we explore the feasibility of transfer learning by training BERT-based models on existing datasets annotated for the presence of cause and treat relations among different types of biomedical entities, and using them to recognize the same relations between food and disease entities in a dataset created for the purposes of this study. The best models achieve macro averaged F1 scores of 0.847 and 0.900 for the cause and treat relations, respectively.

## introduction the ongoing prevalence of malnutrition, the rising incidence of chronic diseases affected by diet, and the fact that even food that is generally considered to be healthy can be harmful to patients suffering from certain diseases or when ingested in combination with specific drugs, require a profound understanding of the role of nutrition in the complex environmental interactions that contribute to the development or treatment of different ailments. the effect of food on human health is the subject of numerous biomedical studies, however, the sheer volume and the predominantly unstructured form of newly published articles prevents medical professionals from keeping up with recent discoveries, and impedes the development of systems for knowledge-base construction, decision support, and question-answering (qa), which brings about the need for information extraction (ie) methods for structuring the newly published knowledge. knowledge graphs (kgs) are specialized data representation structures that store information as a collection of interlinked descriptions of entities. the development of relation extraction (re) methods is necessary for automatic linking of the nodes in kgs and reducing the amount of work required by the experts in order to create and extend these resources. a lot of research effort has been dedicated to extracting relations between different biomedical entities, however, the lack of annotated data impedes the development of food-disease re methods, which are necessary for linking food entities to concepts from the biomedical domain, and understanding the impact of nutrition on human health. transfer learning (tl) @xcite is a potential solution for this problem, which involves improving a learner from one domain by transferring information from a related domain. the use of tl in this paper is twofold. on the one hand, we use models that are pre-trained on large amounts of data, and fine-tune them for the re task. on the other hand, we investigate the feasibility of re-purposing existing annotated ie resources in the biomedical domain as a potential strategy for making up for the deficit of such resources in the food domain. we focus on the detection of cause and treat relations among food and disease entities, and represent the re task as a binary classification problem, meaning that we train separate classifiers that detect the presence of each relation type. we perform fine-tuning of bert @xcite , biobert @xcite and roberta @xcite ) models, which have achieved state of the art results in several natural language processing (nlp) tasks. to train the classifiers, we use the crowdtruth @xcite and adverse drug events (ade) @xcite datasets, which contain sentences annotated for the existence of relations between different types of biomedical entities. we then apply tl in order to use the classifiers trained on the source datasets to directly predict relations among food and disease entities. the reasoning behind the use of tl in this setting is that even though the sentences contain entities of different types, by masking the entity occurrences in the sentence, the models could use the context words around the entities and pick up on linguistic features such as keywords or sentence structure to detect the presence of a particular relation. even though our goal is focused on detecting the relations between food and disease entities, we believe the method to be general enough to be applicable for entities of any type, as long as the relation is the same as the one the model was trained to recognize. to evaluate the proposed models, we introduce a dataset of 608 sentences, which are extracted from abstracts of scientific articles from pubmed and are manually annotated for the presence of cause and treat relations between food and disease entities. to the best of our knowledge, this is the first english re dataset in the food domain, and it is publicly available on github foot_0 , as an open-source resource that can be reused in future studies. the rest of the paper is organized as follows. in the next section, we give an overview of the re work in the domains of biomedicine, and food and nutrition. the data sources used for the experiments are described in section 3. the text representation and classification models are presented in section 4, while their evaluation is discussed in section 5. ## related work in the past decade, numerous methods have been developed for extracting biomedical relations, such as drug-drug @xcite @xcite , proteinprotein @xcite @xcite , drug-disease @xcite , chemical-gene @xcite and chemical-protein @xcite interactions. in the domain of food and nutrition, the efforts directed at creating re systems have been quite more limited in comparison. a gold standard for food re has been generated for the german language @xcite , and different methods such as distant supervision (ds), pattern-matching, and the use of co-occurrence measures have been investigated for the detection of food relations for customer advice @xcite . a chinese food re system @xcite has also been developed, which treats re as a sequence labeling task and adopts conditional random fields (crfs) models to extract relations between food and disease entities from chinese biomedical data. however, resources in other languages are not easily re-purposed for the english language. a related resource in the english language which contains extracted relations of food and disease entities is the nutrichem database @xcite , which links plant-based foods with their small molecule components, drugs and disease phenotypes. a critical difference between nutrichem and the method we aim to develop in this work is the fact that nutrichem limits its scope to plant-based foods, while we do not pose a limitation on the type of foods or diseases between which the relations occur, and aim to extract relations from a broader range of food categories. the benefits of tl have previously been investigated for the purposes of biomedical ner @xcite and re @xcite @xcite . recent work has been aimed at solving the challenges of imbalanced relation distribution, linguistic variation and partial transfer using relationgated adversarial learning @xcite , and capturing dependency tree information using treelstm models @xcite . bert has achieved state-of-the-art results on natural language processing (nlp) tasks, including re between several types of biomedical entities, which is one of the tasks in the biomedical language understanding evaluation (blue) benchmark @xcite . a comparison of the performance of bert models for detecting relations between proteins and chemicals, and genomic factors and drugs or drug responses @xcite , finds that depending on the target corpus, different variants of bert may achieve the best results, and that fine-tuning the models is preferable over freezing the layers of the original model and only updating the weights of new layers added on top of the original ones. guided by these findings, we perform fine-tuning of several bert variants for the re task. the adverse drug events (ade) corpus @xcite , which is one of the source datasets in our experiments, has been extensively used for training re models, and more recently, for the exploitation of inter-task correlations for joint entity and relation extraction using different approaches, such as adversarial training @xcite , cross-modal attention networks @xcite and bert models @xcite . however, unlike the previous work done with this corpus, our goal is not to predict relations between the annotated entities, but to learn the context words used for expressing causal relations, so they can be recognized regardless of the entities between which they occur. ## data tl usually involves the use of two types of datasets: source datasets and target datasets, where models are trained on the source datasets, and adapted to make predictions on the target datasets. we are specifically interested in extracting relations between food and disease entities, and we use the help of two existing source datasets, the crowdtruth @xcite and the ade dataset @xcite , in order to extract relations in the target fooddisease dataset, which was created for the purposes of this study. ## methodology in this section, we describe the proposed re method, starting with the preprocessing applied to accomplish the generalization of the models trained on the source datasets to the target dataset. we then introduce the pre-trained transformer models used for text representation, and their fine-tuning for the re task. ## conclusion in this paper, we propose relation extraction (re) models for the detection of cause and treat relations between food and disease entities from raw text. to make up for the absence of annotated data for this task, we explore the feasibility of transfer learning (tl) by using the transformer models bert, roberta, and biobert, which are pretrained on large amounts of data, and fine-tuned for performing re between various types of biomedical entities. the models are trained to recognize relations based on the context words used to express each relation, rather than the entities themselves, so they can successfully generalize to the task of recognizing the relations between food and disease entities, and likely, other types of entities, though this is not evaluated in the scope of this paper. in order to evaluate the proposed approach, we introduce the fooddisease dataset, which consists of 608 sentences annotated for the existence of the cause and treat relations between food and disease entities in sentences of pubmed abstracts. the dataset is released as an open-source resource, and is, to the best of our knowledge, the first annotated english re dataset of such kind in the food domain. the best models that are fine-tuned on this dataset achieve macro averaged f1 scores of 0.847 and 0.900 for the cause and treat relations, respectively. the best models which are fine-tuned using the data where the entities are not food-disease pairs, but other biomedical entities of various types, achieve macro averaged f1 score of 0.805 for the cause relation and 0.900 for the treat relation. this indicates that in the event where no experts are available to annotate data, the proposed method enables the repurposing of existing re datasets for the training of models that can recognize the relation that the dataset is annotated for, between different types of entities. the developed models will be used as part of an information extraction pipeline which will structure the findings of experts in biomedical scientific literature, with the aim of alleviating the process of linking knowledge graphs from the domain of biomedicine to the domain of food and nutrition.