A Hybrid of Rule-based and Transformer-based Approaches for Relation Extraction in Biodiversity Literature (2023)

Abstract: Relation extraction (RE) is one of the tasks behind many relevant natural language processing (NLP) applications. Exploiting the information hidden in millions of scholarly articles by leveraging NLP, specifically RE, systems could benefit studies in specialized domains, e.g. biomedicine and biodiversity. Although deep learning (DL)-based methods have shown state-of-the-art performance in many NLP tasks including RE, DL for domain-specific RE systems has been hindered by the lack of expert-labeled datasets which are typically required to train such methods. In this paper, we take advantage of the zero-shot (i.e., not requiring any labeled data) capability of pattern-based methods for RE using a rule-based approach, combined with templates for natural language inference (NLI) transformer models. We present our hybrid method for RE that exploits the advantages of both methods, i.e., interpretability of rules and transferability of transformers. Evaluated on a corpus of biodiversity literature with annotated relations, our hybrid method demonstrated an improvement of up to 15 percentage points in recall and best performance over solely rule-based and transformer-based methods with F1-scores ranging from 89.61% to 96.75% for reproductive condition - temporal expression relations, and ranging from 85.39% to 89.90% for habitat - geographic location relations.

## introduction relation extraction (re) is a natural language processing (nlp) task that is concerned with the identification of binary semantic relationships between entities or concepts in text. re predicts whether a relationship holds between two entities (or concepts), based on the context of the sentence. many approaches to re take as input a sentence together with pre-extracted named entities (within the sentence), and identify relations between those entities using heuristics or machine learning-based approaches @xcite . for example, in the sentence "semangkok forest reserve is a designated hill dipterocarp forest conservation area located in selangor state about 60 km north of kuala lumpur, while pasoh forest reserve, is a designated lowland dipterocarp forest conservation area in negeri sembilan about 60 km east -southeast of kuala lumpur, peninsular malaysia", foot_0 an re system should be able to identify the relationship of the geographic location entity "semangkok forest reserve" with the habitat entity "hill dipterocarp forest", but no relation between "semangkok forest reserve" and "lowland dipterocarp forest". this type of information is helpful in associating information on habitats with geographic distribution of species. extracting information from text is better guided by domain knowledge of the targeted use case @xcite @xcite . for example, in the biodiversity domain, methods for extracting a plant species' geographic location with related habitat information, and reproductive condition (i.e., reproductive status) with related temporal expression that exist in biodiver-sity texts, are better crafted with the involvement of domain experts. rule-based methods lend themselves well to domain-specific re tasks @xcite @xcite . aside from being highly interpretable, such methods define a set of rules manually created by domain experts and capture syntactic patterns that are associated with different types of relations observed in a corpus @xcite @xcite . advancements in machine learning (ml) and, more recently, deep learning (dl), have led to state-of-the-art performance in re. ml and dl-based models learn features from data giving them strong generalization ability, adaptability, and scalability. however, the performance of ml-and dl-based methods relies on the availability of domain-specific annotated datasets; this assumption is not always viable for many specialised domains such as biodiversity, law or finance @xcite . in this paper, we integrate the advantages of both rule-based and dl-based re methods by developing a zero-shot hybrid re approach. our main contribution is a novel hybrid re method that is underpinned by a two-step approach. in the first step, we hand-crafted rules that capture syntactic patterns, which were implemented based on regular expressions (regexes). the second step leverages a state-of-the-art transformer model, textto-text transfer transformer or t5 @xcite , for natural language inference (nli). we created premise-hypothesis templates as input for the t5-based nli model, to determine if a relation holds between a given pair of entities. our method presents the following advantages over other re methods: (1) improved performance over solely rule-based or dl-based methods; (2) reduced computational bottleneck since a substantial proportion of the relations are extracted using the more computationally efficient rule-based method; and (3) reduced labeling cost associated with dataset or corpus annotation. we applied our method on documents drawn from the body of literature on biodiversity -a relatively under-resourced domain -focusing on two types of relations: (1) plant species' reproductive conditions and their related temporal expressions, and (2) habitats and their related geographic locations. harvesting these details from biodiversity literature will enable data-driven discovery of plant species' reproductive patterns and habitats. this, in turn, will aid in more informed plans for reforestation and restoration of land. in the remainder of this paper, we first provide a review of prior work related to our study (section 2). this is followed by our problem formulation (section 3) and a description of the dataset we developed and used in our experiments (section 4). then, we present details of the zero-shot hybrid re approach that we developed (section 5), and the results of evaluating the hybrid model (section 6). we then analyze our results (section 7) before providing a summary of our findings and directions for future work in section 8. ## related work existing re methods can be categorized into two broad types: traditional and dl-based methods. traditional methods use either rules or machine learning techniques (e.g., those based on statistical classifiers trained on hand-crafted features) to extract a set of predefined relations from a corpus @xcite . rule-based methods define rules, which are a set of hand-crafted extraction patterns typically created by domain experts @xcite @xcite . these rules are based on manually identified syntactic patterns that are associated with different types of relations, as observed in a corpus. rules have the advantage of being highly interpretable: they can be easily understood by humans, which makes them a good choice for tasks where it is important to explain the reasoning behind the system's output. however, rule-based methods have two main limitations: they can be time-consuming to create and they are domain-dependent (e.g., a rule-based system that is designed to identify relations in medical text may not necessarily be able to identify relations in financial text). meanwhile, ml techniques for re are based on the supervised training of a classification model on a dataset whereby relations have been manually annotated. there are feature-based methods @xcite that use selected syntactic and semantic features as the bases of similarity in training a classification model. there are also kernel-based methods @xcite that use kernel functions to determine similarity between two relation instance representations, together with a support vector machine (svm) model as a classifier. although ml-based re methods gained superiority in the past in terms of performance, their performance is greatly dependent on the set of selected features or the choice of kernel functions. as they are trained in a supervised manner, ml models also require labeled data which can be costly. dl methods that have emerged more recently have been shown to outperform traditional methods for re. these dl models learn higher-order, abstract feature representations from sentences that make them more generalizable, adaptable to new domains, and scalable. with the emergence of dl, models that employ neural architectures such as convolutional neural networks (cnns) @xcite , recurrent neural networks (rnn) @xcite , graph convolutional networks (gcn) @xcite , attention-based neural networks @xcite , and transformer-based language models @xcite have been utilized for re tasks. however, similar to traditional ml models, training or fine-tuning dl models for downstream applications such as re also requires labeled data @xcite . in recent years, zero-shot transformer-based approaches to information extraction requiring no labeled data have become popular @xcite @xcite . for instance, @xcite reduced re to the problem of answering simple reading comprehension questions. they mapped each relation type r(x, y) to at least one parameterized natural-language question q x whose answer is y. for example, the relation educated_at(x, y) can be mapped to "where did x study?" and "which university did x graduate from?". the success of these types of re methods is primarily due to the significant developments in and availability of transformer-based pre-trained language models (plms) @xcite @xcite . plms were pre-trained on large-scale corpora using unsupervised learning objectives such as masked language modeling, and were then fine-tuned for downstream tasks, such as question answering (qa) and natural language inference (nli), using relatively smaller amounts of task-or domain-specific labeled data @xcite . zero-shot transformer-based approaches to re are based on the careful, systematic construction of inputs for plms, which then elicit a model prediction (i.e., a label) that can be mapped to a decision on whether a certain type of relation holds. zero-shot methods significantly reduce the labeling cost associated with re because only a small amount of labeled data is required, i.e., test samples for evaluating the model. in this work, we developed a zero-shot approach to re that is the first of its kind to be applied in the biodiversity domain. our zero-shot methods for re are based on rules and transformer models that -when combined -demonstrate superior performance, in comparison to the use of rules or transformers alone. ## problem formulation given an input sentence i that is a sequence of tokens [t 0 , t 1 , ..., t n ], a source entity e @xmath0, ..., t j ] and a target entity e @xmath1, ...., t v ], we treat the re task as a binary classification task, whereby the input is the triple (i, e s , e t ), and the output is @xmath2, 1} where 1 indicates that a relationship from the source entity to the target entity (e @xmath3, otherwise 0. in this work, we focus on the two relation types described below. the has_time relation: this holds between a reproductive condition mention and a temporal expression, i.e., "reproductive condition has_time temporal expression", whereby the reproductive condition mention is considered to be the source and the temporal expression serves as the target. the has_location relation: this holds between a habitat mention and a geographic location, i.e., "habitat has_location geographic location", whereby the habitat mention is considered to be the source and the geographic location is the target. ## dataset to support the development of approaches to the above-mentioned problem, we utilized a corpus that is a subset foot_1 of the gold standard corpus for named entity recognition (ner) that was presented in the work of @xcite and was designed in accordance to the annotation scheme used in the copious project @xcite . it contains information relevant to the occurrence and reproductive patterns of a tropical tree family, dipterocarpaceae (dipterocarps). the corpus is comprised of 151 manually selected one-to twoparagraph documents from online environmental science and ecology journal repositories, e.g., journal of tropical ecology, journal of ecology, etc. for this re work, we are particularly interested in the annotations of the following named entity (ne) types: habitat, geographic location, temporal expression, and reproductive condition. the descriptions and examples of these ne types are shown in ## methods in this section, we present our methods for extracting (1) related reproductive condition and temporal expressions (has_time), and (2) related habitat and geographic location mentions (has_location). ## evaluation and results in this paper, we designed rules based on our training set. we tested and refined these rules on a held-out development (dev) set, and evaluated their performance using the test set. we applied the regular expression rules we created to extract related entities in a given sentence. this approach resulted in 100% precision for both relation types. this means that our regex-based rules can reliably identify positive samples, i.e., correct relations. however, this approach obtains poor recall, i.e., 33.91% and 36.96% for has_time and has_location relations, respectively, implying that our rules fail to identify many correct relations. this results in the lowest f1-scores (50.64% and 53.97%, respectively) among the methods we investigated. even using a simple co-occurrence approach obtains much higher f1-scores: 94.57% for has_time and 84.02% for has_location relations. to evaluate our transformer-based approach, we applied our chosen t5 model on the nli task, building upon the huggingface library foot_4 . our evaluation on the test set yielded f1-scores higher than our rule-based method. for the has_location relation type, our transformer method produced an f1-score of 84.75%, which is slightly higher than that of the co-occurrence-based method (84.02%). however, the transformer-based method was outperformed by the co-occurrence-based one by 7.59 percentage points (86.98% vs 94.57% in terms of f1-score). it is noticeable that combining our rule-based approach with the transformer model to form a hybrid approach improved the f1-score for the has_time relation type from 86.98% to 89.61%, and from 84.75% to 85.39% for the has_location relation type. apart from improved performance, our hybrid approach is also more efficient, in that it requires the application of the more computationally expensive transformer models only on instances that were not classified by the rule-based approach as pertaining to relations. we further improved our hybrid approach by using compound entities identified using regex rules in generating premise-hypothesis pairs for the transformer model, instead of separate single entities. we evaluated this method (referred to as 'hybrid + compound entities' in ## discussion our rule-based method is the most precise among all the methods we developed in this study. however, it is also the method that yielded the lowest recall, missing to identify more than half of true relations in the test set. the rule-based approach is suitable for applications that cannot compromise on precision, e.g., systems that support clinical decisions or automatic curation of databases. its main drawback, however, is its reliance on syntactic similarity only, i.e., solely on patterns found within sentences. thus, it is not robust to noisy data; any deviation from the expected sentence structure that is captured by the rules, would affect the performance of the method. among the methods presented in this paper, our transformer-based method is the most straightforward to implement. it is based on the population of natural-language hypothesis templates with named ## conclusions and future work in this paper, we present our unsupervised relation extraction methods to extract relationships pertaining to habitats and reproductive conditions of plant species as described in text. these methods include: (1) regular expression-based rules; (2) transformer-based models for nli; (3) a hybrid approach combining our rules and transformer model; and (4) an improved hybrid approach that captures compound entities. our rule-based method undepinned by regexes obtained the highest precision but lowest recall. meanwhile, our transformerbased method, which is based on the systematic generation of premise-hypothesis pairs as input for a t5-based nli model, resulted in f1-score values higher than those produced by the regex rules. the strengths of the rule-and transformer-based methods are combined in our hybrid approach. with the incorporation of compound entities in the generation of nli inputs, our hybrid approach produced the best performance, with f1-scores of 96.75% for the has_time relation type, and 89.90% for the has_location relation type. our work shows that even without a large labeled training dataset, it is viable to extract -with satisfactory performance -relations between entities from biodiversity literature. this also shows that the combination of rules or pattern-based methods with state-of-the-art transformer models can lead to an improvement in the performance of re, compared with a method that is solely based on transformers. for our future work, we plan to compare our hybrid approach with state-of-the-art zero-shot relation extraction methods, e.g., those proposed by @xcite and @xcite , evaluating it on other datasets such as fewrel @xcite and wikizsl @xcite which were drawn from the general domain. furthermore, we will explore using other transformerbased models and formulating re in terms of other downstream tasks, e.g., question answering. we also intend to integrate our hybrid approach into an application, i.e., an information extraction pipeline that can automate the curation of information from literature to populate a biodiversityfocused database.