Prompting C hat GPT in MNER : Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge (2023)

Abstract: Multimodal Named Entity Recognition (MNER) on social media aims to enhance textual entity prediction by incorporating image-based clues. Existing studies mainly focus on maximizing the utilization of pertinent image information or incorporating external knowledge from explicit knowledge bases. However, these methods either neglect the necessity of providing the model with external knowledge, or encounter issues of high redundancy in the retrieved knowledge. In this paper, we present PGIM — a two-stage framework that aims to leverage ChatGPT as an implicit knowledge base and enable it to heuristically generate auxiliary knowledge for more efficient entity prediction. Specifically, PGIM contains a Multimodal Similar Example Awareness module that selects suitable examples from a small number of predefined artificial samples. These examples are then integrated into a formatted prompt template tailored to the MNER and guide ChatGPT to generate auxiliary refined knowledge. Finally, the acquired knowledge is integrated with the original text and fed into a downstream model for further processing. Extensive experiments show that PGIM outperforms state-of-the-art methods on two classic MNER datasets and exhibits a stronger robustness and generalization capability.

## introduction multimodal named entity recognition (mner) has recently garnered significant attention @xcite . users generate copious amounts of unstructured content primarily consisting of images and text on social media. the textual component in ## methodology pgim is mainly divided into two stages. in the stage of generating auxiliary refined knowledge, pgim leverages a limited set of predefined artificial samples and employs the multimodal similar example awareness (msea) module to carefully select relevant instances. these chosen examples are then incorporated into properly formatted prompts, thereby enhancing the heuristic guidance provided to chatgpt for acquiring refined knowledge. (detailed in §3.2). in the stage of entity prediction based on auxiliary knowledge, pgim combines the original text with the knowledge information generated by chatgpt. this concatenated input is then fed into a transformer-based encoder to generate token representations. finally, pgim feeds the representations into the linear-chain conditional random field (crf) @xcite layer to predict the probability distribution of the original text sequence (detailed in §3. 3 ). an overview of the pgim is depicted in figure 2 . ## conclusion in this paper, we propose a two-stage framework called pgim and bring the potential of llms to mner in a novel way. extensive experiments show that pgim outperforms state-of-the-art methods and considerably overcomes obvious problems in previous studies. additionally, pgim exhibits a strong robustness and generalization capability, and only necessitates a single gpu and a reasonable number of chatgpt invocations. in our opinion, this is an ingenious way of introducing llms into mner. we hope that pgim will serve as a solid baseline to inspire future research on mner and ultimately solve this task better.