Annotating F rame N et via Structure-Conditioned Language Generation (2024)

Abstract: Despite the remarkable generative capabilities of language models in producing naturalistic language, their effectiveness on explicit manipulation and generation of linguistic structures remain understudied. In this paper, we investigate the task of generating new sentences preserving a given semantic structure, following the FrameNet formalism. We propose a framework to produce novel frame-semantically annotated sentences following an overgenerate-and-filter approach. Our results show that conditioning on rich, explicit semantic information tends to produce generations with high human acceptance, under both prompting and finetuning. Our generated frame-semantic structured annotations are effective at training data augmentation for frame-semantic role labeling in low-resource settings; however, we do not see benefits under higher resource settings. Our study concludes that while generating high-quality, semantically rich data might be within reach, the downstream utility of such generations remains to be seen, highlighting the outstanding challenges with automating linguistic annotation tasks.

## introduction large language models (llms) have demonstrated unprecedented capabilities in generating naturalistic language. these successes hint at lms' implicit capabilities to "understand" language; but are they capable of processing explicit symbolic structures in order to generate language consistent with the structures? not only would this help us understand the depth of llms' linguistic capabilities but would also serve to efficiently and cheaply expand existing sources of linguistic structure annotation. in this work, we investigate the abilities growing up, children are rewarded often. growing up, boys are rewarded for breaking the rules. ## framenet and extensions frame semantics theory @xcite posits that understanding a word requires access to a semantic frame-a conceptual structure that represents situations, objects, or actions, providing context to the meaning of words or phrases. frame elements (fes) are the roles involved in a frame, describing a certain aspect of the frame. a lexical unit (lu) is a pairing of tokens (specifically a word lemma and its part of speech) and their evoked frames. as illustrated in figure 1 , the token "disciplined" evokes the lu discipline.v, which is associated with the frame rewards_and_punishment, with fes including time, evaluee, and reason. grounded in frame semantics theory, framenet @xcite is a lexical database, featuring sentences that are annotated by linguistic experts according to frame semantics. within framenet, the majority of sentences are annotated with a focus on a specific lu within each sentence, which is referred to as lexicographic data; figure 1 shows such an instance. a subset of framenet's annotations consider all lus within a sentence; these are called full-text data; figure 1 does not consider other lus such as grow.v or break.v. framenet has defined 1,224 frames, covering 13,640 lexical units.the framenet hierarchy also links fes using 10,725 relations. however, of the 13,640 identified lus, only 62% have associated annotations. our approach seeks to automatically generate annotated examples for the remaining 38% of the lus, towards increasing coverage in framenet without laborious manual annotation. sister lu replacement @xcite propose a solution to framenet's coverage problem using an intuitive approach: since lus within the same frame tend to share similar annotation structures, they substitute one lu (the target lu) with another (a sister lu) to yield a new sentence. this replacement approach only considers lus with the same pos tag to preserve the semantics of the original sentence; for instance, in figure 1 , we replace the sister lu discipline.v with the target lu reward.v. however, due to the nuanced semantic differences between the two lus, the specific content of the fe spans in the original sentence may no longer be consistent with the target lu in the new sentence. indeed @xcite report such semantic mismatches as their primary weakness. to overcome this very weakness, our work proposes leveraging llms to generate fe spans that better align with the target lu, as described subsequently. for the rest of this work, we focus solely on verb lus, where initial experiments showed that the inconsistency problem was the most severe. details of framenet's lu distribution by pos tags, along with examples of non-verb lu replacements can be found in appendix a. ## generating framenet annotations via frame-semantic conditioning we propose an approach to automate the expansion of framenet annotations by generating new annotations with language models. given sister lureplaced annotations ( §2; @xcite , we select fe spans which are likely to be semantically inconsistent ( §3.1), generate new sentences with replacement spans by conditioning on framesemantic structure information ( §3.2) and finally filter inconsistent generations ( §3.3). ## augmenting data for frame-srl beyond improving framenet coverage, we investigate the extrinsic utility of our generations as training data to improve the frame-srl task, which involves identifying and classifying fe spans in sentences for a given frame-lu pair. here, we consider a modified frame-srl task, which considers gold-standard frames and lus, following @xcite . this remains a challenging task even for powerful models like gpt-4, which achieves a test f1 score of only 0.228 in contrast to @xcite 's state-of-the-art f1 score of 0.722. for experimental ease, we fine-tune a span-bert model on framenet's full-text data as our parser foot_3 and avoid using existing parsers due to their reliance on weaker, non-transformer architectures @xcite , complex problem formulation @xcite , or need for extra frame and fe information @xcite . as a pilot study, we prioritize augmenting the training data with verb lus with f1 scores below 0.75 on average. this serves as an oracle augmenter targeting the lowest-performing lus in the test set. for the generation of augmented data, we use our top-performing models within t5 and gpt-4 models according to human evaluation: t5 | fe and gpt-4 | frame+fe models. of 2,295 lus present in the test data, 370 were selected for augmentation, resulting in 5,631 generated instances. after filtering, we retain 4,596 instances from gpt-4 | frame+fe and 4,638 instances from t5 | fe. additional experiments using different augmentation strategies on subsets of framenet are in appendix g. ## conclusion our study provides insights into the successes and failures of llms in manipulating framenet's linguistic structures. when conditioned on semantic information, llms show improved capability in producing semantically annotated sentences, indicating the value of linguistic structure in language generation. under a low-resource setting, our generated annotations prove effective for augmenting training data for frame-srl. nevertheless, this success does not translate to a high-resource setting, echoing challenges reported in applying llms to other flavors of semantics @xcite @xcite . these outcomes underline the need for further exploration into how llms can be more effectively employed in automating linguistic structure annotation.