Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs (2023)

Abstract: Unsupervised relation extraction (URE) aims to extract relations between named entities from raw text without requiring manual annotations or pre-existing knowledge bases. In recent studies of URE, researchers put a notable emphasis on contrastive learning strategies for acquiring relation representations. However, these studies often overlook two important aspects: the inclusion of diverse positive pairs for contrastive learning and the exploration of appropriate loss functions. In this paper, we propose AugURE with both within-sentence pairs augmentation and augmentation through cross-sentence pairs extraction to increase the diversity of positive pairs and strengthen the discriminative power of contrastive learning. We also identify the limitation of noise-contrastive estimation (NCE) loss for relation representation learning and propose to apply margin loss for sentence pairs. Experiments on NYT-FB and TACRED datasets demonstrate that the proposed relation representation learning and a simple K-Means clustering achieves state-of-the-art performance.

## introduction unsupervised relation extraction (ure) aims to extract relations between named entities from the raw text in an unsupervised manner without relying on annotated relations from human experts. ure tasks are proposed to overcome the limitations of traditional relation extraction (re) tasks, which target to extract pre-defined relations and require a time-consuming and labor-intensive data annotation process. to discover relational facts without any predefined relation types, ure tasks are often formulated as semantics clustering tasks, where the entity pairs with the same relations should be clustered together. therefore, effectively learning the relation representations is an essential step toward 1 https://github.com/qingwang-isu/augure solving ure tasks. recent studies on relation representation learning @xcite @xcite adopt a contrastive learning process. contrastive learning aims to bring similar-relation sentences (positive pairs) closer while separating different-relation sentences (negative pairs). there are several ideas for constructing positive pairs, a key step in the contrastive learning process. some methods propose to augment the entity pairs and relation context of the original sentences as positive examples @xcite . to avoid similar sentences being pushed apart in the contrastive learning process, @xcite further propose to utilize cluster centroids to generate positive pairs within the clusters. however, these approaches often fail to provide diverse positive pairs for contrastive learning to obtain more effective relation representations. another challenge arises from the loss function perspective. the most recent ure work @xcite directly transfers the noise-contrastive estimation (nce) loss function from computer vision tasks @xcite @xcite and achieves good performance. however, the negative examples used by the nce loss function are randomly sampled sentences from the same minibatch, which may result in high false negatives, especially when the total number of relational clusters is not large. moreover, relation semantics should not be regarded as "same/different" in many cases but rather as a similarity spectrum. for example, the relation org:top_members/employees is semantically closer to org:members than org:founded_by. therefore, we argue that the binary nce loss function may suit poorly for relation representation learning in ure tasks, whereas a margin-based loss may fit better. in this paper, we propose several techniques to augment positive pairs to increase their diversity and strengthen the discriminative power of con-trastive learning. specifically, we propose withinsentence pairs augmentation and augmentation through cross-sentence pairs extraction. withinsentence pairs augmentation generates positive pairs from the same sentence using intermediate words sampling and entity pair replacement techniques. in the augmentation through cross-sentence pairs extraction, we aim to recognize sentences with the same relation semantics to form a more diverse positive sample pool. specifically, we employ a three-step approach. first, we utilize the open information extraction (openie) model @xcite to extract relation templates among all sentences. intuitively, sentences with the same relation template are likely to express the same relation. to further group the templates that share the same relation semantics, we employ natural language inference (nli) @xcite to discover mutually entailed templates. finally, we extract positive pairs from different sentences within the same template groups. to facilitate the extraction of more cross-sentence positive pairs, we further use chatgpt to rewrite the original sentences. to overcome the limitations of the binary nce loss function, we propose to use margin loss to capture the spectrum of semantic similarity between positive and negative pairs. in summary, our main contributions are: ## related work relation extraction is a crucial natural language processing (nlp) task and has been extensively studied in the past years @xcite @xcite . although existing re methods achieve decent performance with manual annotations and kbs, those annotations can be expensive to obtain in practice. in recent years, few-shot re and zeroshot re @xcite @xcite become popular, which significantly reduce the human annotation cost. however, fewshot re and zero-shot re still require a pre-defined relation label space which is often unavailable in new datasets or open-domain scenarios. an upsurge of interest in ure has been witnessed to extract relational facts without predefined relation types and human-annotated data. the task of openre has also attracted considerable attention from researchers. unlike ure, openre has additional access to existing annotated relations and aims to extract novel relational facts from opendomain corpora. relation representation learning is a key step for both tasks. amongst the prior studies, tran et al. (2020) find that the utilization of named entities alone to induce relation types can surpass the performance of previous methods. this observation underscores the significant inductive bias that entity types offer for ure. later, hu et al. (2020) assume the cluster assignments as pseudo-labels and propose a self-supervised learning framework that leverages pre-trained language models (lms) to learn and refine contextualized entity pair representations. however, the frequently re-initialized linear classification layers can hinder the process of representation learning. wang et al. (2021) are the first to utilize deep metric learning in the openre task and this scheme demonstrates good capabilities in representation learning. motivated by the achievements of contrastive learning in computer vision tasks @xcite @xcite , recent state-ofthe-art methods have embraced contrastive learning frameworks to acquire relation representations. @xcite integrate existing hierarchical information into relation representations by a contrastive objective function. @xcite intervene on the named entities and context to obtain the underlying causal effects of them. both methods adopt instance-wise contrastive learning objectives and exploit base-level sentence differences by increasing the diversity of positive examples. to further capture high-level semantics in the relation features, @xcite introduce a hierarchical exemplar contrastive learning schema to optimize relation representations using both instance-wise and exemplar-wise signals. however, this approach still lacks in providing diverse positive pairs for contrastive learning, limiting the effectiveness of relation representations. additionally, it employs the nce loss function without considering its alignment with the task requirements. ## methodology in this section, we provide a comprehensive exploration of the proposed model augure. figure 1 offers an overview of our framework. as mentioned in section 1, both positive and negative examples are important for relation representation learning in ure tasks. when handling positive examples, we proposed several augmentation techniques to generate more diverse positive pairs. when handling negative examples, we argue that relation semantics should not be simply viewed as 'same/different' but rather as a similarity spectrum. to this end, we propose to apply margin loss, which is also more robust to false negatives. ## conclusion in conclusion, this paper offers an exploration of ure from the perspective of relation representation learning. we introduce two novel augmentation techniques: within-sentence pairs augmentation and augmentation through cross-sentence pairs extraction, both of which aim to augment positive pairs, enhancing their diversity and bolstering the discriminative capabilities of contrastive learning. we also identify the limitations of nce loss in ure tasks and propose margin loss for sentence pairs. experiments on two datasets show the superiority of the proposed augure compared to competitive baselines, highlighting the efficacy of the augmentation techniques and the significance of employing margin loss. overall, our findings advance the understanding of relation representation learning and provide valuable insights for future research and practical applications.