Integrating Structural Semantic Knowledge for Enhanced Information Extraction Pre-training (2024)

Abstract: Information Extraction (IE), aiming to extract structured information from unstructured natural language texts, can significantly benefit from pre-trained language models. However, existing pre-training methods solely focus on exploiting the textual knowledge, relying extensively on annotated large-scale datasets, which is labor-intensive and thus limits the scalability and versatility of the resulting models. To address these issues, we propose SKIE, a novel pre-training framework tailored for IE that integrates structural semantic knowledge via contrastive learning, effectively alleviating the annotation burden. Specifically, SKIE utilizes Abstract Meaning Representation (AMR) as a low-cost supervision source to boost model performance without human intervention. By enhancing the topology of AMR graphs, SKIE derives high-quality cohesive subgraphs as additional training samples, providing diverse multi-level structural semantic knowledge. Furthermore, SKIE refines the graph encoder to better capture cohesive information and edge relation information, thereby improving the pre-training efficacy. Extensive experimental results demonstrate that SKIE outperforms state-of-the-art baselines across multiple IE tasks and showcases exceptional performance in few-shot and zero-shot settings.

## introduction information extraction (ie) aims to extract structured information from unstructured natural language texts @xcite , which encompasses several subtasks such as named entity recognition (ner) @xcite , relation extraction (re) @xcite , and event extraction (ee) @xcite . considering the inherent connections among these subtasks, recent methods propose to jointly resolve them within a unified framework, capitalizing on the generalization versatility of pre-trained language models (plms). for instance, uie @xcite embeds schema-based prompts into the corpora to pretrain a text-to-structure generative plm, enabling it to generate uniform representations. usm @xcite utilizes three kinds of supervised datasets and employs unified token linking to structure information during pre-training. mirror @xcite designs a unified data interface to reorganize datasets into multi-slot tuples for pretraining. metaretriever @xcite introduces a meta-pretraining algorithm to retrieve task-specific knowledge from plms for ie tasks. however, existing pre-training methods suffer from two major challenges. first, the high cost of annotation restricts existing datasets for ie tasks to a few predefined categories and small data volumes @xcite , limiting the amount of supervised datasets available for pre-training. second, these methods are constrained to solely utilizing annotated textual knowledge, neglecting the potential structural semantic knowledge inherent in texts, which hinders their ability to leverage complex structural knowledge. a feasible solution is to generate self-supervised signals from extensive unsupervised data by leveraging their structural semantic knowledge, instead of relying on limited supervised data. abstract meaning representation (amr) @xcite , which has demonstrated its ability to capture structural semantic knowledge within texts without additional human effort @xcite , stands out as a fitting choice. figure 1 illustrates an example of a text segment and its corresponding amr graph. the text is converted into an amr graph through amr parsing, where nodes represent basic semantic units such as entities and predicates, while edges denote their semantic relations @xcite . armed with this insight, we propose skie, a novel pre-training method that integrates structural semantic knowledge to enhance the model's versatility across multiple ie tasks. skie leverages amr parsing to generate self-supervised signals, offering a flexible and general approach to semantic representations. to capture more diverse semantic structures, skie introduces cohesive subgraphs, which are densely connected subsets of pivotal nodes within the graph. then, contrastive learning is employed to bridge the associations between texts and graphs. specifically, skie comprises three key modules: the topology enhancement module, the encoding cohesion module, and the contrastive learning module. capitalizing on the cohesion-guided topology enhancement, we extract cohesive subgraphs from amr graphs to acquire diverse multi-level structural semantic knowledge. to preserve edge relation information and cohesive information in the graphs, we propose a topology-aware encoder for cohesive encoding. by learning from correlations and distinctions between texts and graphs via contrastive learning, skie can comprehend semantic associations and intrinsic patterns within the texts, enabling better adaptation to ie tasks. our contributions are as follows: ## methodology the overall framework is shown in figure 2 , consisting of three key modules: topology enhancement, encoding cohesion, and contrastive learning, which are introduced in section 3.1, section 3.2, and section 3.3, respectively. section 3.4 describes how to fine-tune our plm to adapt to ie tasks.