P eer DA : Data Augmentation via Modeling Peer Relation for Span Identification Tasks (2023)

Abstract: Span identification aims at identifying specific text spans from text input and classifying them into pre-defined categories. Different from previous works that merely leverage the Subordinate (SUB) relation (i.e. if a span is an instance of a certain category) to train models, this paper for the first time explores the Peer (PR) relation, which indicates that two spans are instances of the same category and share similar features. Specifically, a novel Peer Data Augmentation (PeerDA) approach is proposed which employs span pairs with the PR relation as the augmentation data for training. PeerDA has two unique advantages: (1) There are a large number of PR span pairs for augmenting the training data. (2) The augmented data can prevent the trained model from over-fitting the superficial span-category mapping by pushing the model to leverage the span semantics. Experimental results on ten datasets over four diverse tasks across seven domains demonstrate the effectiveness of PeerDA. Notably, PeerDA achieves state-of-the-art results on six of them.

## introduction span identification (spanid) is a family of natural language processing (nlp) tasks with the goal of detecting specific spans from the input text and further classifying them into pre-defined categories @xcite . it serves as the initial step for complex text analysis by narrowing down the search scopes of important spans, which holds a pivotal position in the field of nlp @xcite . recently, different domain-specific spanid tasks, such as social media named entity recognition (ner) @xcite , aspect-based sentiment analysis (absa) @xcite , contract clause extraction (cce) @xcite , span based propaganda detection (sbpd) @xcite and argument extraction @xcite , have emerged for various nlp applications. precisely, as shown in figure 1 (a), the process of spanid can be reinterpreted as extracting spancategory subordinate (sub) relation -if a span in the input text is an instance of a certain category. early works (chiu and nichols, 2016) typically tackle spanid tasks as a sequence tagging problem, where the sub relation is recognized via predicting the category for each input token under certain context. recently, to better utilize category semantics, many efforts have been made on reformulating spanid tasks as a machine reading comprehension (mrc) problem @xcite . as shown by the example in figure 1 (b) , such formulation first creates a sub query for each category and then recognizes the sub relation by detecting relevant spans in the input text (i.e., context) as answers to the category query. however, only leveraging the sub relation in the training data to build spanid models may suffer from two limitations: 1) over-fitting: with only sub relation, spanid models tend to capture the superficial span-category correlations. such correlations may misguide the models to ignore the semantics of the given span but make predictions based on the memorized span-category patterns, which hurts the generalization capability of the models. 2) data scarcity: for low-resource scenarios or long-tailed categories, the number of span-category pairs with sub relation (sub pairs) could be very limited and insufficient to learn a reliable spanid model. in this paper, we explore the span-span peer (pr) relation to alleviate the above limitations. specifically, the pr relation indicates that two spans are two different instances of the same category. the major difference between pr relation and sub relation is that the former one intends to correlate two spans without giving the categories they belong to. for example, in figure 1 (a), "hawaii" and "london" are connected with the pr relation because they are instances of the same category. by jointly recognizing sub relation and pr relation in the input text, the model is enforced to favor the usage of span semantics instead of span-category patterns for prediction, reducing the risk of over-fitting. in addition, the number of spanspan pairs with the pr relation (pr pairs) grows quadratically over the number of sub pairs. therefore, we can still construct a reasonable number of training data with pr pairs for categories having insufficient examples. in this paper, with the aim of leveraging the pr relation to enhance spanid models, we propose a peer data augmentation (peerda) approach that treats pr pairs as a kind of augmented training data. to achieve this, as depicted in figure 1 (b), we extend the usage of the original training data into two views. the first view is the sub-based training data. it is used to directly solve the spanid tasks by extracting the sub relation, which is the typical formulation of mrc-based approaches. the second view is the pr-based training data. it is our augmentation to enrich the semantics of spans by extracting the pr relation in the original training data, where one span is used to identify its peer from the input context. note that our pr-based training data can be easily formulated into the mrc paradigm. therefore, the knowledge learned from such augmentation data can be directly transferred to enhance the model's capability to capture sub relation (i.e., the spanid tasks). to better accommodate the mrc-style sub and pr data, we develop a stronger and more memoryefficient mrc model. compared to the designs in @xcite , our model introduces a bilinear component to calculate the span scores and consistently achieves better performance with a 4 times smaller memory consumption. besides, we propose a margin-based contrastive learning strategy to additionally model the negative spans to the query (e.g., when querying the context in figure 1 for "org" entities, "london" becomes a negative span) so that the spans from different categories are separated more apart in the semantic space. we evaluate the effectiveness of peerda on ten datasets across seven domains, from four different spanid tasks, namely, ner, absa, cce, and sbpd. experimental results show that extracting pr relation benefits the learning of semantics and encourages models to identify more possible spans. as a result, peerda is a new state-of-the-art (sota) method on six spanid datasets. our analyses further demonstrate the capability of peerda to alleviate scarcity and over-fitting issues. our contributions are summarized as follows: huang et al. (2021) leverage wikipedia to create distant labels for ner. chen et al. (2021) transfer data from high-resource to low-resource domains. jain et al. (2019); li et al. (2020c) tackle cross-lingual ner by projecting labels from highresource to low-resource languages, which is particularly common in real applications (kruengkrai et al., 2020). differently, the motivation of peerda is to leverage the augmented data to enhance models' capability on semantic understanding by minimizing(maximizing) the distances between semantically similar(distant) spans. mrc: mrc is to extract an answer span from a relevant context conditioned on a given query. it is initially designed to solve question answering tasks (hermann et al., 2015), while recent trends have shown great advantages in formulating nlp tasks as mrc problems. in the context of spanid, li et al. (2020b); xu et al. (2022b, 2023b) address the nested ner issues by decomposing nested entities under multiple queries. mao et al. (2021); zhang et al. (2021a) tackle absa in a unified mrc framework. hendrycks et al. (2021) tackle cce with mrc to deal with the extraction of long clauses. moreover, other tasks such as relation extraction @xcite , event detection @xcite , and summarization @xcite are also reported to benefit from the mrc paradigm. ## further discussions in this section, we make further discussions to bring valuable insights of our peerda approach. ## conclusions in this paper, we propose a novel peerda approach for spanid tasks to augment training data from the perspective of capturing the pr relation. peerda has two unique advantages: (1) it is capable to leverage abundant but previously unused pr relation as additional training data. (2) it alleviates the over-fitting issue of mrc models by pushing the models to weigh more on semantics. we conduct extensive experiments to verify the effectiveness of peerda. further in-depth analyses demonstrate that the improvement of peerda comes from a better semantic understanding capability.