FSUIE : A Novel Fuzzy Span Mechanism for Universal Information Extraction (2023)

Abstract: Universal Information Extraction (UIE) has been introduced as a unified framework for various Information Extraction (IE) tasks and has achieved widespread success. Despite this, UIE models have limitations. For example, they rely heavily on span boundaries in the data during training, which does not reflect the reality of span annotation challenges. Slight adjustments to positions can also meet requirements. Additionally, UIE models lack attention to the limited span length feature in IE. To address these deficiencies, we propose the Fuzzy Span Universal Information Extraction (FSUIE) framework. Specifically, our contribution consists of two concepts: fuzzy span loss and fuzzy span attention . Our experimental results on a series of main IE tasks show significant improvement compared to the baseline, especially in terms of fast convergence and strong performance with small amounts of data and training epochs. These results demonstrate the effectiveness and generalization of FSUIE in different tasks, settings, and scenarios.

## introduction information extraction (ie) is focused on extracting predefined types of information from unstructured text sources, such as named entity recognition (ner), relationship extraction (re), and sentiment extraction (se). to uniformly model the various ie tasks under a unified framework, a generative universal information extraction (uie) was proposed in @xcite and has achieved widespread success on various ie datasets and benchmarks. due to the necessity of a powerful generative pre-training model for the generative uie, the time overhead is extensive and the efficiency is not satisfactory. for this reason, this paper examines span-based uie to unify various ie tasks, conceptualizing ie tasks as predictions of spans. however, uie models still have some limitations. first, as it is the process of training machine learning models to extract specific information from unstructured text sources, ie relies heavily on human annotation which involves labeling the data by identifying the specific information to be extracted and marking the corresponding span boundaries in the text manually. however, due to the complexity of natural language, determining the correct span boundaries can be challenging, leading to the phenomenon of annotation ambiguity. as shown in figure 1 , different annotated spans can be considered reasonable. in the span learning of uie models, the method of teacher forcing is commonly used for loss calculation, making the model dependent on the precise span boundaries given in the training data. this can cause performance bottlenecks due to annotation ambiguity. when the model structure in uie places too much emphasis on the exact boundaries of ie tasks, it leads to insufficient utilization of supervision information. in order to predict span boundaries, positions closer to the ground-truth should be more accurate than those relatively farther away, as shown in figure 1 . for example, words close to the target "car" are more likely to be correct than the word "evening" which is farther away from the target. under the premise of positioning to the span where "car" is located, both the "yellow car" and the "yellow sports car" can be regarded as vehicle entities. this means that the span model learned should be fuzzy rather than precise. in addition, the use of pre-trained transformer @xcite in uie to extract the start and end position representations also poses a problem. the transformer model is designed to focus on the global representation of the input text, while uie requires focusing on specific parts of the text to determine the span boundaries. this mismatch between the transformer's focus on global representation and uie's focus on specific parts of the text can negatively impact the performance of the model. when there is a mismatch between the transformer architecture and the span representation learning, the model may not make good use of prior knowledge in ie. specifically, given the start boundary (end boundary) of the label span, the corresponding end boundary (start boundary) is more likely to be found within a certain range before and after, rather than throughout the entire sequence. this is a prior hypotheses that span has limited length, which is ignored in the vanilla uie model. to address this, a fuzzy span attention mechanism, rather than fixed attention, should be applied. in this paper, we propose the fuzzy span universal information extraction (fsuie) framework that addresses the limitations of uie models by applying the fuzzy span feature, reducing overreliance on label span boundaries and adaptively adjusting attention span length. specifically, to solve the issue of fuzzy boundaries, we design the fuzzy span loss that quantitatively represents the correctness information distributed on fuzzy span. at the same time, we introduce fuzzy span attention that sets the scope of attention to a fuzzy range and adaptively adjusts the length of span according to the encoding. we conduct experiments on various main ie tasks (ner, re, and aste). the results show that our fsuie has a significant improvement compared to the strong uie baseline in different settings. additionally, it achieves new state-of-the-art performance on some ner, re, and aste benchmarks with only bert-base architecture, outperforming models with stronger pre-trained language models and complex neural designs. furthermore, our model shows extremely fast convergence, and good generalization on lowresource settings. these experiments demonstrate the effectiveness and generalization of fsuie in different tasks, settings, and scenarios. ## fsuie in fsuie, incorporating fuzzy span into base uie model involves two aspects. firstly, for the spans carrying specific semantic types in the training data, the boundary targets should be learned as fuzzy boundaries to reduce over-reliance on span boundaries. to achieve this, we propose a novel fuzzy span loss. secondly, during the span representation learning, the attention applied in span should be dynamic and of limited length, rather than covering the entire sequence. to achieve this, we propose a novel fuzzy span attention. ## related work universal model building universal model structures for a wide range of nlp tasks has been a hot research area in recent years. the focus is on building model structures that can be adapted to different sources of data, different types of labels, different languages, and different tasks. several universal models have been proposed, such as models learning deep contextualized word representations @xcite , event extraction models that can predict different labels universally @xcite , models that can handle multiple languages @xcite @xcite , a universal fine-tuned approach to transfer learning @xcite , models that learning syntactic dependency structure over many typologically different languages @xcite and models that can universally model various ie tasks in a unified text-to-structure framework @xcite . this paper builds upon the uie by incorporating the fuzzy span mechanism to improve ie performance. information extraction ie is the task of extracting structured information from unstructured text data. this includes ner, re, aste, event extraction (ee), aspect-based sentiment analysis (absa), etc. research has proposed numerous approaches for ie, such as rule-based @xcite , machine learning @xcite , deep learning @xcite , active learning @xcite , and logic fusion @xcite . there are still many task-specific models being proposed, based on previous approaches and structures, e.g., ner @xcite @xcite ; re @xcite , absa @xcite ; and aste @xcite . more related work about sparse attention please refer to appendix b. ## conclusion in this paper, we proposed the fuzzy span universal information extraction (fsuie) framework, an improvement for universal information extraction. to make use of boundary information in the training data and learn a decision-closer span-aware representation, we proposed a fuzzy span loss and fuzzy span attention. extensive experiments on several main ie tasks show that our fsuie has a significant improvement compared to the uie baseline, and achieves state-of-the-art results on ade ner datasets, ace04 re, ace05 re and ade re datasets and four aste datasets. the experiments also reveal fsuie's fast convergence and good generality in low-resource settings. all the results demonstrate the effectiveness and generalizability of our fsuie in information extraction.