Semantic Frame Induction using Masked Word Embeddings and Two-Step Clustering (2021)

Abstract: Recent studies on semantic frame induction show that relatively high performance has been achieved by using clustering-based methods with contextualized word embeddings. However, there are two potential drawbacks to these methods: one is that they focus too much on the superficial information of the frame-evoking verb and the other is that they tend to divide the instances of the same verb into too many different frame clusters. To overcome these drawbacks, we propose a semantic frame induction method using masked word embeddings and two-step clustering. Through experiments on the English FrameNet data, we demonstrate that using the masked word embeddings is effective for avoiding too much reliance on the surface information of frame-evoking verbs and that two-step clustering can improve the number of resulting frame clusters for the instances of the same verb.

## introduction semantic frame induction is a task of mapping frame-evoking words, typically verbs, into semantic frames they evoke (and the collection of instances of words to be mapped into the same semantic frame forms a cluster). for example, in the case of example sentences from framenet @xcite shown in (1) to ( 4 ) in recent studies have shown the usefulness of contextualized word embeddings such as elmo @xcite and bert @xcite for semantic frame induction. for example, the top three methods @xcite @xcite in subtask-a of 2020/09/17 研究 (1) we'll not get there before the rain comes. first, the contextualized word embeddings of the frame-evoking verbs strongly reflect the superficial information of the verbs. the left side of figure 1 shows a 2d projection of contextualized embeddings of instances of the verbs "get" and "acquire" extracted from example sentences in framenet. specifically, we extracted instances of "get" and "acquire" from framenet, obtained their embeddings by using a pre-trained bert, and projected them into two dimensions by using tdistributed stochastic neighbor embedding (t-sne) @xcite . as shown in the figure, among instances of "get", those that evoke the getting frame tend to be located close to instances of "acquire" that evokes the same getting frame. however, we can see that the difference be-tween verbs is larger than the difference between the frames that each verb evokes. to remedy this drawback, we propose a method that uses a masked word embedding, a contextualized embedding of a masked word. the right side of figure 1 shows a 2d projection of masked word embeddings for instances of the verbs "get" and "acquire". the use of masks can hide the superficial information of the verbs, and consequently we can confirm that instances of verbs that evoke the same frame are located close to each other. the second drawback is that these methods perform clustering instances across all verbs simultaneously. such clustering may divide instances of the same verb into too many different frame clusters. for example, if there are outlier vectors that are not typical for a particular verb, they tend to form individual clusters with instances of other frames in most cases. to solve this problem, we propose a two-step clustering, which first performs clustering instances of the same verb according to their meaning and then performs further clustering across all verbs. ## proposed method the proposed semantic frame induction method uses masked word embeddings and two-step clustering. we explain these details below. ## experiment we conducted an experiment of semantic frame induction to confirm the efficacy of our method. in this experiment, the objective is to group the given frame-evoking verbs with their context according to the frames they evoke. ## conclusion we proposed a method that uses masked word embeddings and two-step clustering for semantic frame induction. the results of experiments using framenet data showed that masked word embeddings and two-step clustering are quite effective for this frame induction task. we will conduct experiments in a setting where nouns and adjectives are also accounted for as frame-evoking words. the future goal of this research is to build a framesemantic resource, which requires not only the induction of semantic frames but also the determination of the arguments required by each frame and the induction of semantic roles of the arguments. a possible extension of our approach is to utilize contextualized word embeddings of arguments of verbs to see if it is possible to generalize our approach for achieving this goal.