Span-based Named Entity Recognition by Generating and Compressing Information (2023)

Abstract: The information bottleneck (IB) principle has been proven effective in various NLP applications. The existing work, however, only used either generative or information compression models to improve the performance of the target task. In this paper, we propose to combine the two types of IB models into one system to enhance Named Entity Recognition (NER).For one type of IB model, we incorporate two unsupervised generative components, span reconstruction and synonym generation, into a span-based NER system. The span reconstruction ensures that the contextualised span representation keeps the span information, while the synonym generation makes synonyms have similar representations even in different contexts. For the other type of IB model, we add a supervised IB layer that performs information compression into the system to preserve useful features for NER in the resulting span representations. Experiments on five different corpora indicate that jointly training both generative and information compression models can enhance the performance of the baseline span-based NER system. Our source code is publicly available at https://github.com/nguyennth/joint-ib-models .

## methods the overall framework of our model is illustrated in figure 1 . we first encode span embeddings using a transformer-based network. the output from the encoder is then used in three components: (1) entity classification that classifies input spans into an entity type or non-entity using a supervised vib framework @xcite ; (2) span reconstruction that recovers input spans of gold entities; and (3) synonym generation that generates synonyms for a gold entity. similarly to a multitask setting, we simultaneously train the three components. we only run the first component at the inference step to predict named entities. ## analysis 5.1 ablation study to evaluate the effect of each component on the proposed model, we ran the following settings on the development sets: when we introduced synonym generation into the model (i.e., the all setting), we could obtain better performance than the baseline model on five over six corpora. this indicates that indirectly injecting semantic information into the span representation is mostly helpful. the exceptions were with medmention and share/clefe, where the all setting produced a lower f1 score than the supvib+spanreco. by jointly training two types of ib, i.e., generative model and supervised ib, we could enhance the ner performance over the baseline on four out of six testing corpora. especially with the two most complex, multi-category ner corpora, i.e., genia and medmention-21, the joint model could outperform the baseline by a larger margin than the other corpora. lastly, looking at the last column in ## conclusion we introduced a joint span-based ner model consisting of three components: vae-based span reconstruction, vae-based synonym generation, and vib-based ner. each component plays a differ-ent role in learning span representation. the vibbased ner tries to preserve the information about the target ner category in the estimated latent while filtering out irrelevant information from the input. the model is forced to keep contextualised span information when having the span reconstruction component. meanwhile, the synonym generation component indirectly injects semantic information about a span's synonyms. when testing on five different corpora, we found that the joint model could perform better than the baseline. the proposed model focuses on learning span representation, which is applicable not only to spanbased ner but also to other span-based tasks such as event coreference resolution @xcite and question answering @xcite . we plan to apply our model to such tasks in the future.