C ode IE : Large Code Generation Models are Better Few-Shot Information Extractors (2023)

Abstract: Large language models (LLMs) pre-trained on massive corpora have demonstrated impressive few-shot learning ability on many NLP tasks. A common practice is to recast the task into a text-to-text format such that generative LLMs of natural language (NL-LLMs) like GPT-3 can be prompted to solve it. However, it is nontrivial to perform information extraction (IE) tasks with NL-LLMs since the output of the IE task is usually structured and therefore is hard to be converted into plain text. In this paper, we propose to recast the structured output in the form of code instead of natural language and utilize generative LLMs of code (Code-LLMs) such as Codex to perform IE tasks, in particular, named entity recognition and relation extraction. In contrast to NL-LLMs, we show that Code-LLMs can be well-aligned with these IE tasks by designing code-style prompts and formulating these IE tasks as code generation tasks. Experiment results on seven benchmarks show that our method consistently outperforms fine-tuning moderate-size pre-trained models specially designed for IE tasks (e.g., UIE) and prompting NL-LLMs under few-shot settings. We further conduct a series of in-depth analyses to demonstrate the merits of leveraging Code-LLMs for IE tasks.

## introduction information extraction (ie) aims to recognize structured information from plain text. it spans various tasks with diverse output structures such as named entity recognition (ner), relation extraction (re), etc. @xcite @xcite @xcite . to express and address these different tasks in a unified framework, recent works propose to linearize the output structures into unstructured ## codeie in this section, we first formulate the two ie tasks we focus on, named entity recognition (ner) and relation extraction (re) in section 2.1. then we describe how we recast these structured prediction tasks into code generation tasks (section 2.2) and prompt code-llms to perform them (section 2.3) under the few-shot scenario. we use python language for our code generation tasks since public python codebases are abundant and code-llms are sufficiently pre-trained on them. ## analysis to take a closer look at the difference between prompting nl-llms with textual format input and prompting code-llms with code format input, in this section, we define several informative metrics and conduct in-depth analyses to shed some light on the following question: what contributes to the final performance of codeie for ie tasks? ## conclusion we propose the first work to utilize the structured code-llms with code-style prompts to perform the few-shot ner and re tasks. experiments show our approach consistently surpasses the uie models and the nl-llms counterpart under the fewshot setting. we conducted extensive analysis and find the performances come from better format consistency and model fidelity, etc. we think these analyzes can facilitate future work. as the further works, we will employ codeie on more ie tasks in different domains, and inspect the robustness of it.