Silver Syntax Pre-training for Cross-Domain Relation Extraction (2023)

Abstract: Relation Extraction (RE) remains a challenging task, especially when considering realistic out-of-domain evaluations. One of the main reasons for this is the limited training size of current RE datasets: obtaining high-quality (manually annotated) data is extremely expensive and cannot realistically be repeated for each new domain. An intermediate training step on data from related tasks has shown to be beneficial across many NLP tasks. However, this setup still requires supplementary annotated data, which is often not available. In this paper, we investigate intermediate pre-training specifically for RE. We exploit the affinity between syntactic structure and semantic RE, and identify the syntactic relations which are closely related to RE by being on the shortest dependency path between two entities. We then take advantage of the high accuracy of current syntactic parsers in order to automatically obtain large amounts of low-cost pre-training data. By pre-training our RE model on the relevant syntactic relations, we are able to outperform the baseline in five out of six cross-domain setups, without any additional annotated data.

## introduction relation extraction (re) is the task of extracting structured knowledge, often in the form of triplets, from unstructured text. despite the increasing attention this task received in recent years, the performance obtained so far are very low @xcite . this happens in particular when considering realistic scenarios which include outof-domain setups, and deal with the whole taskin contrast to the simplified relation classification which assumes that the correct entity pairs are given @xcite @xcite . one main challenge of re and other related information extraction tasks is the "domain-specificity": depending on the text domain, the type of information to extract changes. for example, while in the news domain we can find entities like person and city, and relations like city of birth @xcite , in scientific texts, we can find information about metrics, tasks and comparisons between computational models @xcite . while high-quality, domain-specific data for fine-tuning the re models would be ideal, as for many other nlp tasks, annotating data is expensive and time-consuming. 1 a recent approach that leads to improved performance on a variety of nlp tasks is intermediate task training. it consists of a step of training on one or more nlp tasks between the general language model pre-training and the specific end task fine-tuning (stilt, supplementary training on intermediate labeled-data tasks; @xcite . however, stilt assumes the availability of additional high quality training data, annotated for a related task. in this paper, we explore intermediate pretraining specifically for cross-domain re and look for alternatives which avoid the need of external manually annotated datasets to pre-train the model on. in particular, we analyze the affinity between syntactic structure and semantic relations, by considering the shortest dependency path between two entities @xcite @xcite . we replace the traditional intermediate pre-training step linear-fractional programming ( lfp ) is a generalization of linear programming ( lp ) . on additional annotated data, with a syntax pretraining step on silver data. we exploit the high accuracy of current syntax parsers, for obtaining large amount of low-cost pre-training data. the use of syntax has a long tradition in re @xcite @xcite . recently, work has started to infuse syntax during language model pre-training @xcite showing benefits for re as well. we instead investigate dependency information as silver data in intermediate training, which is more efficient. to the best of our knowledge, the use of syntax in intermediate pre-training for re is novel. we aim to answer the following research questions: 1 does syntax help re via intermediate pre-training (fast and cheap approach)? and 2 how does it compare with pre-training on additional labeled re data (expensive)? we release our model and experiments. foot_2 accordingly, we define the syntax pre-training instances as: in the next section we describe the detailed training process. ## pre-training data quantity analysis we inspect the optimal quantity of syntactic data to pre-train our re model on by fine-tuning this hyperparameter over the dev sets of crossre. the plot in figure 5 reports the average performance of the six models when pre-trained on increasing amounts of syntactic dependencies. 8 starting from 8.4k instances onward, the performance stabilizes above the baseline. we select the peak (20.4k, albeit results are similar between 18-20.4k) for reporting our test set results in ## conclusion we introduce syntax pre-training for re as an alternative to the traditional intermediate training which uses additional manually annotated data. we pretrain our re model over silver ud labels which most frequently connect the semantic entities via the shortest dependency path. we test the proposed method over crossre and outperform the baseline in five out of six cross-domain setups. pre-training over a manually annotated dataset, in comparison, only slightly increases our scores in five out of six evaluations, but at a much higher cost.