Boosting Textural NER with Synthetic Image and Instructive Alignment (2024)

Abstract: Named entity recognition (NER) is a pivotal task reliant on textual data, often impeding the disambiguation of entities due to the absence of context. To tackle this challenge, conventional methods often incorporate images crawled from the internet as auxiliary information. However, the images often lack sufficient entities or would introduce noise. Even with high-quality images, it is still challenging to efficiently use images as auxiliaries (i.e., fine-grained alignment with texts). We introduce a novel method named InstructNER to address these issues. Leveraging the rich real-world knowledge and image synthesis capabilities of a large pre-trained stable diffusion (SD) model, InstructNER transforms the text-only NER into a multimodal NER (MNER) task. A selection process automatically identifies the best synthetic image by comparing fine-grained similarities with internet-crawled images through a visual bag-of-words strategy. Note, during the image synthesis, a cross-attention matrix between synthetic images and raw text emerges, which inspires a soft attention guidance alignment (AGA) mechanism. AGA optimizes the MNER task and concurrently facilitates instructive alignment in MNER. Empirical experiments on prominent MNER datasets show that our method surpasses all text-only baselines, improving F1-score by 1.4% to 2.3%. Remarkably, even when compared to fully multimodal baselines, our approach maintains competitive. Furthermore, we open-source a comprehensive synthetic image dataset and the code to supplement existing raw dataset. The code and datasets are available in https://github.com/Heyest/InstructNER.

## introduction named entity recognition (ner) is a fundamental information extraction task that identifies named *corresponding author. despite the promising results of mner methods @xcite , they still struggle with acquiring large-scale and high-quality text-image paired corpus. common mner datasets, twitter-2015 @xcite and twitter-2017 @xcite , consist of 8,257 and 4,819 image-text pairs, respectively. this represents a mere 26% and 16% of the volume found in the frequently employed textual ner dataset conll03 @xcite . one potential solution involves crawling supplementary images by internet search engines. whereas, unlike raw texts, the internetcrawled images often fail to cover enough entities, which might further bring unexpected noise. as exemplified in figure 1 another critical challenge to the success of the mner task lies in the accurate alignment between text tokens and image regions. existing techniques for the text-image alignment can be summarized into two groups: explicit alignment and implicit alignment. explicit alignment extracts visual object regions, then maps them to corresponding textual tokens @xcite @xcite , but errors can propagate from initial inaccurate region extraction @xcite . consider the left image in figure 1 (c), only the object regions of messi and football are extracted, missing the fcb clothes logo and camp nou stadium in the background. in contrast, implicit alignment alleviates this issue by employing an attention mechanism to learn the alignment weights adaptively @xcite @xcite . despite the token-to-image alignment achieved by these approaches, the attention may not be sufficiently concentrated. regarding the right image of figure 1 (c ), the attention heatmap illustrates messi is located, yet it noticeably allocates irrelevant attention to surrounding areas. in response to the aforementioned challenges, we propose instructner, a novel approach that harnesses the rich real-world knowledge and the image synthesis capabilities of stable diffusion models @xcite to provide supplemental information, thereby transforming the textual ner task into mner task. specifically, we first feed the raw text into the pre-trained stable diffusion (sd) model to generate large-scale synthetic images. then, to alleviate the variability of image quality and select the best one, we employ an off-the-shelf visual bag-of-words (bow) method @xcite to measure the finegrained similarity between the internet-crawled images and the synthetic images, ultimately selecting the most similar one. particularly, the similarity metric of images considers both the coverage and accuracy of entities. as depicted in figure 1 (d ) (left), the selected image accurately encompasses all entities, including messi, the fcb logo, and the camp nou stadium. furthermore, the image synthesis process also produces a cross-attention matrix between the synthetic images and the raw text as a byproduct, inspiring our soft attention guidance alignment (aga) mechanism for mner model training. in addition to optimizing the original mner task, our objective also aims to minimize the kullback-leibler (kl) divergence between the mner model's cross-attention matrix and the aforementioned byproduct matrix. figure 1 (d) shows the attention of the aga mechanism paid to token messi, displaying a higher concentration. we conduct experiments on three representative mner datasets, i.e., twitter-2015, twitter-2017, and wikidiverse, while excluding images with only textual corpus. experimental results demonstrate the superiority of our method over all text-only baselines, with absolute f1-score improvements of 1.4%, 2.3% and 2.1%. moreover, our method still achieves competitive results even when compared to fully multimodal baselines. to sum up, the contributions of this paper are three-fold: figure 2 provides an overview of the comprehensive architecture of instructner. in stage #1, instructner leverages the pre-trained stable diffusion model to generate images, and selects the optimal synthetic image covering sufficient and accurate entities. in stage #2, instructner utilizes the soft attention guidance alignment (aga) mechanism to fuse the raw text and synthetic image. in ## conclusion in this paper, we propose instructner, a comprehensive method that utilizes the image synthesis capability of the stable diffusion model, in order to reframe the textual ner as the mner task and learn instructive alignment between synthetic images and raw texts. the experimental results demonstrate the effectiveness of synthetic images and the innovative soft attention guidance alignment (aga) mechanism in improving ner performance. moreover, we have made available a large-scale, high-quality dataset of synthetic images, which complements existing raw datasets and provides valuable insights for future research endeavors.