Unsupervised Semantic Frame Induction Revisited (2023)

Abstract: This paper addresses the task of semantic frame induction based on pre-trained language models (LMs). The current state of the art is to directly use contextualized embeddings from models such as BERT and to cluster them in a two step clustering process (first lemma-internal, then over all verb tokens in the data set). We propose not to use the LM’s embeddings as such but rather to refine them via some transformer-based denoising autoencoder. The resulting embeddings allow to obtain competitive results while clustering them in a single pass. This shows clearly that the autoendocer allows to already concentrate on the information that is relevant for distinguishing event types.

## introduction in natural language processing, semantic frame induction refers to the task of clustering target word instances, specifically verbs, in a corpus according to their semantic frames in a given context. for example, in the sentences: we would like to cluster the verbs in (a) and (b) in one group and (c) in another. the problem of verb semantic frames induction has received its share of attention, particularly in the semeval 2019 shared task (subtask-a) @xcite , in which the gold labels are annotated according to the framenet @xcite frames inventory. frame-semantic resources are prohibitively expensive and time-consuming to construct due to difficulties in the frame definitions, as well as the complexity of the construction and annotation tasks, that require expert knowledge in lexical event semantics. to overcome these issues, researchers proposed to automate the process of framenet construction through unsupervised techniques @xcite @xcite . unsupervised semantic frame induction methods help to automatically build high-coverage frame-semantic resources. up until recently, state-of-the-art results for semantic frame induction were dominated by a series of models leveraging contextualised pretrained language model representations to cluster instances of verbs according to the frames they evoke @xcite . in recent work, @xcite achieve state-of-the-art results by applying a graph-clustering algorithm based on chinese whispers @xcite by using contextualized representations of frame-evoking verbs from bert @xcite ). another approach has been proposed by @xcite , who also use masked word embeddings and two-step clustering: each target instance is represented by three contextualized embeddings in a text, clustering is performed first over instances of the same verb and then across all verbs. however, these previous methods have one crucial shortcoming. as transformers based contextual words embeddings are originally designed to be fine-tuned on each downstream task to attain their optimal performance, it is unclear how best to extract representations of frame-evoking verbs from them, which are broadly applicable across diverse word-related tasks. in this paper, we further explore the use of lm representations by leveraging transformer-based sequential denoising auto-encoder @xcite (ts-dae) embeddings to tackle the aforementioned problem. the proposed method achieves state-ofthe-art performance on the frame induction task. the contributions of this paper are three-fold: ## method in this section, we provide a brief description of tsdae, and introduce the different components of our semantic frames induction model. the proposed model works in three stages: tsdae based word embeddings tsdae, as shown in figure 1 , is a popular unsupervised learning algorithm based on an encoder decoder architecture. the model is a modified encoder-decoder transformer where the key and value of the crossattention are both restricted to yield sentence embedding only @xcite . the encoder maps the original input vector to a hidden representation, and the decoder maps the hidden representation back to the original input space. during training, noise is added to each input text by deleting or swapping a fraction of all tokens (we delete 60% of words in our experiments foot_0 ), encoding the noisy text and reconstructing the embedding using the decoder module. the autoencoder minimizes the reconstruction error by approximating an identity function @xcite . a good reconstruction quality means that the semantics must be well captured in the word embeddings by the encoder. after training, the decoder module is discarded and the encoder is used to extract word representations. we re-implemented the tsdae algorithm based on huggingface's transformers. 2 the algorithm is a simplified version of methods described in @xcite . clustering after training tsdae on unlabeled sentences from the target task, contextualised vectors for the frame evoking verbs in the sentences are calculated, and then clustered using agglomerative clustering with average linkage and cosine distance. the number of clusters is defined based on clusters maximizing the average silhouette score of all frame evoking verbs. in this section, we introduce the datasets and experiment settings used for semantic frame induction. we also present the evaluation results of each model and compare them against existing semantic frame induction systems. ## analysis we extracted the cluster signatures and manually inspected all of the semantic frame clusters produced by tsdae+ga, our best system, along with their associated verbs in order to scrutinize the emerging semantic classes and gain insight into annotator decisions. we found that the most prominent reason for incorrect clustering was due to the hard partitioning output, while the evaluation dataset contained fuzzy clusters. we also observed that the semantic distinctions that are easier for humans to make often elude representation models, and that discriminating between similar and highly associated but dissimilar verbs remains a challenge for most systems. moreover, we noticed that the effectiveness of the models differ depending on the semantic frames, indicating discrepancies in the quality of representations for verbs from diverse domains. interestingly, we found that many clusters included an incoherent mix of multiple semantic frames along with an incoherent set of verbs. this suggests that frame induction should not be treated solely as a verb clustering task as it requires a distinct and separate approach. ## conclusion in this paper, we introduced the first implementation of tsdae for unsupervised frame induction and demonstrated that our method outperforms previous approaches in semeval'19 shared task 2, setting a new state-of-the-art. our error analysis revealed that a major source of incorrect clustering stemmed from the hard partitioning output, while the evaluation dataset consisted of fuzzy clusters. for future work, we aim to extend our work to the multi-lingual setup in future studies.