C ode4 S truct: Code Generation for Few-Shot Event Structure Prediction (2023)

Abstract: Large Language Model (LLM) trained on a mixture of text and code has demonstrated impressive capability in translating natural language (NL) into structured code. We observe that semantic structures can be conveniently translated into code and propose Code4Struct to leverage such text-to-structure translation capability to tackle structured prediction tasks. As a case study, we formulate Event Argument Extraction (EAE) as converting text into event-argument structures that can be represented as a class object using code. This alignment between structures and code enables us to take advantage of Programming Language (PL) features such as inheritance and type annotation to introduce external knowledge or add constraints. We show that, with sufficient in-context examples, formulating EAE as a code generation problem is advantageous over using variants of text-based prompts. Despite only using 20 training event instances for each event type, Code4Struct is comparable to supervised models trained on 4,202 instances and outperforms current state-of-the-art (SOTA) trained on 20-shot data by 29.5% absolute F1. Code4Struct can use 10-shot training data from a sibling event type to predict arguments for zero-resource event types and outperforms the zero-shot baseline by 12% absolute F1.

## introduction large language model (llm) trained on massive corpora of code mixed with natural language (nl) comments and docstrings 4 (e.g., chen et al. 2021, 1 inheritance is a way to create a hierarchy of classes in pl. a child class can base upon another class, retaining similar implementation. as a case study, we showcase our proposed code4struct on the event argument extraction (eae) task, which aims to extract event structures from unstructured text. eae is the ideal testbed for our method due to the close alignment between eae and pl as shown in by leveraging the alignment between pl and nlp problems, code4struct enjoys various advantages as shown in we outline our contributions as follows: ## code generation prompt construction in event argument extraction (eae) task, a model is provided with an event ontology and the tar- ## why represent event structure in pl? a wide range of nlp tasks have benefited from llm @xcite @xcite trained on web-scale language corpora. to effectively use llm trained on language for eae, one of the biggest challenges is to specify the desired output, namely event structures in our case, using natural language. there is a tradeoff between the effort put into defining the output or designing the prompt (e.g., text2event in essentially, this tradeoff is a result of the mismatch between the pretraining corpora and task output formats. instead of using llm trained on only unstructured text, we turn to llm trained with a mixture of text and code, where the text is often aligned in semantics with the accompanying code. such code-llms have the ability to convert text into corresponding code as demonstrated by @xcite . then we can map the desired output event structure into code in a straightforward manner and leverage the full pretraining power of these models. pls like python offer features (e.g., class, docstrings, type annotations, inheritance) that have a significant presence in the pre-training corpus of code-llm due to frequent usage. code4struct leverages these features to succinctly describe event structures, which makes it better aligned with code-llm. by leveraging llm's learned knowledge from diverse pre-training domains, code4struct can work well in open-domain, achieving non-trivial zeroshot performance given unseen event types ( §4.5). code4struct is also data-efficient as exemplified by reaching comparable performance to fullysupervised methods with much fewer annotated examples (20 per event type) ( §4.5). ## conclusions and future work we propose code4struct for structured prediction tasks in nlp by leveraging llms trained on language and code. as a case study, we use code4struct to extract event arguments from natural language sentences through code generation. we show that, with sufficient in-context examples, formulating eae as a code generation problem is advantageous over using text-based prompts. our proposed code4struct rivals fully-supervised models trained on 4,202 data instances only using 20-shot. it also outperforms a sota model by 29.5% absolute f1 when both are given the same 20-shot data. furthermore, benefitting from hierarchical event definitions, code4struct can predict arguments for zeroresource event types only using 10-shot training instances from its sibling event type and outperforms 0-shot baseline by 12% absolute f1 score. going forward, we plan to expand code4struct to a broader range of more complex structured prediction tasks (e.g., relation prediction, schema matching). we would further explore the executable nature of pl to improve llm's ability for structured prediction.