F rame EOL : Semantic Frame Induction using Causal Language Models (2025)

Abstract: Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke. In recent years, leveraging embeddings of frame-evoking words that are obtained using masked language models (MLMs) such as BERT has led to high-performance semantic frame induction. Although causal language models (CLMs) such as the GPT and Llama series succeed in a wide range of language comprehension tasks and can engage in dialogue as if they understood frames, they have not yet been applied to semantic frame induction. We propose a new method for semantic frame induction based on CLMs. Specifically, we introduce FrameEOL, a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label representing the given situation. To obtain embeddings more suitable for frame induction, we leverage in-context learning (ICL) and deep metric learning (DML). Frame induction is then performed by clustering the resulting embeddings. Experimental results on the English and Japanese FrameNet datasets demonstrate that the proposed methods outperform existing frame induction methods. In particular, for Japanese, which lacks extensive frame resources, the CLM-based method using only 5 ICL examples achieved comparable performance to the MLM-based method fine-tuned with DML.

## introduction frame semantics @xcite considers access to world knowledge that is related to a word to be essential for understanding the word. consider the following sentences: both sentences contain the same verb lost. however, sentence (1) conjures an image of "a competition comes to an end, with someone losing against an opponent according to their score," while sentence (2) evokes a situation where "someone may have unintentionally misplaced a possession." when reading such sentences, humans recognize the different meanings of lost from the context and understand each sentence while supplying the information necessary to understand the situation, such as competition or opponent for sentence (1). in frame semantics, such background knowledge is referred to as a semantic frame and words like lost are called frame-evoking words. the verb lost in sentence (1) corresponds to the finish_competition frame, which involves a competition coming to an end with an outcome determined by a score or ranking. in contrast, the verb lost in sentence (2) corresponds to the losing frame, which describes a situation where someone loses his or her possession. semantic frame knowledge is being developed for many languages. the representative resource is framenet @xcite , a largescale semantic frame resource that was manually developed for english. in addition, resource development following framenet is in progress for several languages, including brazilian portuguese, chinese, hebrew, japanese, korean, and swedish @xcite . however, manual resource development entails enormous costs, making it challenging to construct such large-scale frame knowledge. accordingly, automatic construction of frame knowledge from massive text corpora is an active research area. representative of such studies are those on frame induction: the task of clustering a set of words given together with their context according to the semantic frames they evoke. in recent years, several methods @xcite @xcite that leverage masked language models in the situa+on "he lost his gold medal at the restaurant," please briefly answer whether you think there was an opponent in the scene where he "lost" the medal. no, there is no indica+on of an opponent in the scene. the word "lost" suggests that he misplaced the medal rather than being defeated in a compe++on. in the situa+on "he lost the gold medal by just .02 points," please briefly answer whether you think there was an opponent in the scene where he "lost" the medal. yes, there was likely an opponent in the scene because losing "by just .02 points" suggests a compe++ve context where scores are compared, implying the presence of at least one compe+tor. (mlms) such as bert @xcite to obtain frame embeddings have been proposed. frame embeddings are vector representations that capture the semantic frame evoked by a given word and are used to induce semantic frames. in contrast, large-scale causal language models (clms) such as the gpt and llama series have not been used for semantic frame induction, despite their success in a wide range of language comprehension tasks @xcite . in fact, modern clms such as gpt-4o are potentially likely to understand semantic frames. for example, as shown in figure 1 ,when chatgpt receives the following prompt template with [sentence] replaced by sentence (1) and sentence (2), respectively, it tends to answer "yes" to the former and "no" to the latter. 2 ## proposed method we induce semantic frames by obtaining the frame embeddings of frame-evoking words with frameeol and then clustering the frame-evoking words in a corpus based on the obtained frame embeddings. following @xcite , we use two clustering methods: one-step and two-step clustering. in this section, we first explain frameeol, a method for obtaining frame embeddings via a clm; and then we describe how to optimize frame embeddings via icl and dml. ## experiments on framenet we first conducted experiments on semantic frame induction using clms for english. in these experiments, we used framenet, an english frame knowledge resource, as the dataset for evaluation. ## experiments on japanese framenet for languages besides english, the limited scale of frame knowledge resources has constrained the development of semantic frame induction methods. consequently, most research in this field has been conducted in english. for instance, japanese framenet @xcite , a frame knowledge resource for japanese, contains only 3,130 examples where verbs serve as frame-evoking words. this is a very small dataset, representing only about 3.2% of the number of similar examples in framenet. the scarcity of training data makes it challenging to effectively train deep learning models, which has hindered the advancement of semantic frame induction methods in non-english languages. in this study, to explore the performance of semantic frame induction for non-english languages, we also performed experiments using japanese framenet for training and evaluation. large-scale clms have demonstrated strong multilingual capabilities, even when they are primarily englishcentric @xcite . accordingly, we investigated whether leveraging these multilingual capabilities can be beneficial for semantic frame induction, even when the amount of training data is limited. ## conclusion in this study, we proposed a method using causal language models (clms) for semantic frame induction of verbs. by leveraging the characteristics of clms, our proposed method uses a sequence in which the next word is the frame name as input to the clms, and it uses the final layer embedding of the last token as the frame embedding representation. in experiments on framenet, deep metric learning (dml) exceeded the performance of existing methods using masked language models (mlms) with dml. in-context learning (icl) showed lower performance than existing methods using mlms trained on a large amount of supervised data with dml , but better performance than existing methods without dml. in experiments on japanese framenet, the icl method demonstrated performance comparable to methods based on mlms fine-tuned with dml. these results suggest that, when automatically constructing semantic frame knowledge for languages with no or very limited training data, a promising approach would be to create about 20 training examples and use a method combining clms with icl.