Set Learning for Generative Information Extraction (2023)

Abstract: Recent efforts have endeavored to employ the sequence-to-sequence (Seq2Seq) model in Information Extraction (IE) due to its potential to tackle multiple IE tasks in a unified manner. Under this formalization, multiple structured objects are concatenated as the target sequence in a predefined order. However, structured objects, by their nature, constitute an unordered set. Consequently, this formalization introduces a potential order bias, which can impair model learning. Targeting this issue, this paper proposes a set learning approach that considers multiple permutations of structured objects to optimize set probability approximately. Notably, our approach does not require any modifications to model structures, making it easily integrated into existing generative IE frameworks. Experiments show that our method consistently improves existing frameworks on vast tasks and datasets.

## introduction information extraction (ie) aims to identify structured objects from unstructured text @xcite . recently, many research efforts have been devoted to using sequence-to-sequence (seq2seq) models to solve ie tasks @xcite @xcite . this generative approach enables the development of a universal ie architecture for different ie tasks @xcite @xcite . moreover, combining the generative approach with pre-trained language models (plms) has shown promising results in many ie tasks @xcite . to formulate an ie task as a seq2seq problem, two steps are involved: (1) transforming each structured object into a flat sub-sequence, and (2) sorting and concatenating multiple sub-sequences according to a predefined order. ## related work many studies have been conducted to explore how neural networks should be designed when dealing with sets as inputs or outputs. @xcite and @xcite proposed that neural networks should be permutations invariant to inputs when the inputs are sets. @xcite demonstrated that the permutation of outputs significantly influences the performance of rnn models when generating sets. tasks in nlp, such as multi-label classification and keyword generation, can be formulated as set generation tasks. @xcite proposed a permutation sampling strategy for multi-label classification tasks, which uses the dependencies between labels to generate informative permutations. @xcite uses beam search to generate all triplets corresponding to an input simultaneously for sentiment triplet extraction tasks. @xcite proposed the one2set model for keyword generation, which simultaneously generates all keywords corresponding to input through a parallel decoder. in contrast, we propose a more sample and universal approach to optimize various existing generative ie frameworks. our approach can easily combine with the off-the-shelf methods and achieve promising improvements. ## conclusion in this paper, we reveal the order bias issue in generative information extraction (ie) and pro-pose the set learning approach for generative ie to address this issue. the proposed set learning approach defines and optimizes set probability in seq2seq models and reduces the computational cost by permutation sampling. notably, the proposed approach can be easily integrated with existing generative ie frameworks as a plugin to enhance performance. experiments demonstrate that the proposed approach significantly improves existing frameworks in various tasks and datasets. we believe our work can inspire further research on the potential of set learning in other natural language processing tasks.