Semantic Frame Induction with Deep Metric Learning (2023)

Abstract: Recent studies have demonstrated the usefulness of contextualized word embeddings in unsupervised semantic frame induction. However, they have also revealed that generic contextualized embeddings are not always consistent with human intuitions about semantic frames, which causes unsatisfactory performance for frame induction based on contextualized embeddings. In this paper, we address supervised semantic frame induction, which assumes the existence of frame-annotated data for a subset of predicates in a corpus and aims to build a frame induction model that leverages the annotated data. We propose a model that uses deep metric learning to fine-tune a contextualized embedding model, and we apply the fine-tuned contextualized embeddings to perform semantic frame induction. Our experiments on FrameNet show that fine-tuning with deep metric learning considerably improves the clustering evaluation scores, namely, the B-cubed F-score and Purity F-score, by about 8 points or more. We also demonstrate that our approach is effective even when the number of training instances is small.

## introduction semantic frames are knowledge resources that reflect human intuitions about various concepts such as situations and events. one of the most representative semantic frame resources is framenet @xcite , which consists of semantic frames, lexical units (lus) that evoke these frames, and collections of frameannotated sentences. semantic frame induction is the task of grouping predicates, typically verbs, according to the semantic frames they evoke. 1 for example, given the verbs in the example sentences recent studies @xcite @xcite have demonstrated the usefulness of contextualized word embeddings such as elmo @xcite and bert @xcite in unsupervised semantic frame induction. figure 1 (a) shows a 2d t-sne @xcite projection of the vanilla bert 2 embeddings of verbs extracted from frameannotated sentences in framenet. we can confirm that the instances of the verb "cover" in exam-ples (1) and ( 7 ) are far apart in the space, which reflects the distance between their meanings. in contrast, "cover" in (7) and "treat" in (8), which are annotated with the same topic frame, are close together. however, instances are not always properly placed in the semantic space according to the frames they evoke. for example, "remove" in (5) and "drain" in (6) are annotated with the same re-moving frame but are not close together. this suggests that contextualized word embeddings are not always consistent with human intuition about semantic frames. hence, in this study, we tackle supervised semantic frame induction, which assumes the existence of annotated data for certain predicates, to induce semantic frames that adequately reflect human intuition about the frames. we propose methods that use deep metric learning to fine-tune the contextual word embedding model so that instances of verbs that evoke the same frame are placed close together and other instances are placed farther apart in the semantic space. figure 1(b) shows the 2d t-sne projection of bert embeddings after fine-tuning with adacos @xcite , which is a representative deep metric learning method. we can confirm that predicates that evoke the same frame are close together, such as those in (3) and (4) and those in (5) and ( 6 ). this suggests that deep metric learning enables fine-tuning of bert to obtain embedding spaces that better reflect human intuition about semantic frames. ## related work for automatic construction of semantic frame resources, studies on grouping predicates according to the semantic frames they evoke can be divided into two groups: those that work on semantic frame identification, in which predicates are classified into predefined frames; and those that work on semantic frame induction, in which predicates are grouped according to the frames that they evoke, which are typically not given in advance. semantic frame identification is often treated as a subtask of frame semantic parsing @xcite , and the methods using contextualized embedding have become mainstream. for example, @xcite used a bert-based model to generate representations for frames and lus by using their formal definitions. @xcite used a bert-based model with a context encoder, to encode the context surrounding the frame-evoking word, and a frame encoder, to encode the frames' definitions and semantic roles. yong and torrent (2020) treated semantic frame identification as a clustering task. 3 they first excluded predicates that evoke frames that are not included in framenet by applying an anomaly detection model; then, they grouped the remaining predicates according to their meanings by using contextualized embeddings of predicates and sentence embeddings of the frame definitions. semantic frame induction is the task of grouping predicates in texts according to the frames they evoke. instead of frames being given in advance, each grouping of given predicates is considered a frame. as with semantic frame identification, methods using contextualized embedding have become mainstream. @xcite first performed agglomerative clustering by using the bert embedding of a frame-evoking verb and then split each cluster into two on the basis of the verb's substitutes. @xcite used the embedding of the frame-evoking verb and the average word embedding of all the words in a sentence, as obtained by skip-gram @xcite or elmo and then performed agglomerative clustering. @xcite applied graph clustering by using the elmo embedding of the frame-evoking verb. yamada et al. (2021a) leveraged the embedding of the masked frame-evoking verb and performed twostep clustering, which comprised intra-verb and cross-verb clustering. yamada et al. (2021b) investigated how well contextualized word representations can recognize the difference of frames that the same verb evokes, and explored which types of representation are suitable for semantic frame induction. all of these studies focused on unsupervised semantic frame induction, with no training data. in contrast, in this study, we assume the existence of frame-annotated data for a subset of predicates appearing in a corpus, and work on supervised semantic frame induction. ## experiment to evaluate the usefulness of fine-tuning with deep metric learning, we experimented with supervised semantic frame induction by comparing previous non-fine-tuned models to various fine-tuned models ranging from typical to evolved ones. by varying the number of training instances, we also verified that our models were effective even for training a small number of instances. ## analysis of fine-tuned embedding it is not easy to analyze the properties of an embedding in clustering evaluation because the performance depends on the clustering method and the number of clusters. to better understand the fine-tuned embeddings, we performed a similarity ranking evaluation and visualized the embeddings. ## conclusion we worked on the supervised semantic frame induction, and we proposed a model that uses deep metric learning to fine-tune a contextualized embedding model and applied the fine-tuned contextualized embeddings to perform semantic frame induction. in our experiments, we showed that fine-tuned bert models with the triplet, arcface, and ada-cos losses are quite promising for semantic frame induction, as the human intuition in developing semantic frames such as those in framenet can be well captured by deep metric learning. in particular, the fine-tuned bert model with the triplet loss performed considerably better than vanilla bert even when the number of training instances was small; accordingly, the fine-tuned model is expected to have a wide range of applications. we also found that the one-step clustering can be a good choice in addition to two-step clustering when performing fine-tuning. the ultimate goal of this study is to automatically construct semantic frame knowledge from large text corpora. this goal requires not only grouping the verbs according to the frames that they evoke but also grouping their arguments according to the frame element roles that they fill. our proposed fine-tuned contextualized word embedding with deep metric learning could be effective for clustering arguments as it is for clustering verbs. we would like to explore how to achieve this goal.