Robust Frame-Semantic Models with Lexical Unit Trees and Negative Samples (2024)

Abstract: We present novel advancements in frame-semantic parsing, specifically focusing on target identification and frame identification. Our target identification model employs a novel prefix tree modification to enable robust support for multi-word lexical units, resulting in a coverage of 99.4% of the targets in the FrameNet 1.7 fulltext annotations. It utilizes a RoBERTa-based filter to achieve an F1 score of 0.775, surpassing the previous state-of-the-art solution by +0.012. For frame identification, we introduce a modification to the standard multiple-choice classification paradigm by incorporating additional negative frames for targets with limited candidate frames, resulting in a +0.014 accuracy improvement over the frame-only model of FIDO, the previous state-of-the-art system, and +0.002 over its full system. Our approach significantly enhances performance on rare frames, exhibiting an improvement of +0.044 over FIDO’s accuracy on frames with 5 or fewer samples, and on under-utilized frames, with an improvement of +0.139 on targets with a single candidate frame. Overall, our contributions address critical challenges and advance the state-of-the-art in frame-semantic parsing.

## introduction frame-semantic parsing @xcite is the process of automatically identifying semantic frames @xcite and frame elements within text. semantic frames are structured events, concepts, or scenarios containing frame elements which describe different roles or entities associated with those events, concepts, or scenarios. semantic frames provide a structured framework for performing and explaining natural language processing tasks, such as knowledge extraction @xcite , question answering @xcite , fact checking @xcite , and event detection @xcite . an example figure 1 : an example of frame-semantic parsing annotations in framenet @xcite . targets are represented by bold words with their corresponding lexical unit italicized above and evoked frame boxed below. frame elements are labeled below the text and are colorcoded to match their corresponding frame. frame-semantic parsing typically consists of three primary tasks: target identification, frame identification, and argument identification @xcite . target identification is the identification of targets-words or phrases that evoke frames in a sentence, e.g., thought and discovered in figure 1 . each target is an instance of a particular lexical unit, a linguistic unit which pairs a word with a meaning, represented as lemma.part-ofspeech, e.g., think.v for thought and discover.v for discovered in figure 1 . frame identification is the classification of each target as a frame, e.g., opin-ion and achieving_first in figure 1 . argument identification is the identification and classification of each frame element belonging to a frame evoked by a target, e.g., "columbus", the cognizer frame element, and "that he [...] jamaica", the message frame element, in the opinion frame in figure 1 . in this paper, we present novel methods for target identification and frame identification which outperform previous state-of-the-art approaches. our target identification model generates a robust set of candidate targets for each lexical unit in the framenet 1.7 fulltext annotations @xcite , covering 99.4% of the targets in the test set, before filtering out false-positive candidates by employing a roberta @xcite classification model. previous works @xcite have also approached target identification using a candidate generation and filtering process due to poor performance of statistical models on directly predicting targets. however, neither of these were able to identify discontinuous lexical units-multi-word lexical units separated by auxiliary words. we address this problem by representing candidate lexical units in a modified prefix tree which supports indexing with part-of-speech (pos) tags. this allows our search algorithm to efficiently extract discontinuous multi-word targets without having to exhaustively enumerate all possible auxiliary words. candidate generation methods typically produce many false positives. we address this problem by using a transformer model based on roberta to filter out false-positive candidates, a deviation from past works which only used linguistic features for such filtering. combined, our target identification model reaches an f1 of 0.775, an increase of +0.012 over the previous best-performing target identification model, achieving state-of-theart performance. in summary, our work makes the following contributions. the codebase of this work is made publicly available at https://github.com/idirlab/ frame , including the source codes for the models and the scripts for preparing datasets. ## background and related work framenet @xcite is a collection of over 1,200 semantic frames, their respective inter-frame relationships, thousands of annotated sentences, and tens of thousands of exemplar sentences. each frame describes a distinct semantic idea and can be evoked by different words or phrases called targets. a target is an instance of a lexical unit (lu), a distinct pairing of a word with a meaning which is represented by its lemma and pos, e.g., think.v. frames are often evoked by several different lus. for example, discover.v, find.v, and invent.v are lus which can evoke the achiev-ing_first frame. frame elements describe roles within a frame and are defined separately for each frame, although commonly used frame elements with shared meaning may be found in multiple frames, e.g., time and agent. target identification is typically done by identifying potential lus using the lu index provided in framenet, which maps each distinct lu to the frame it evokes, followed by further filtering them out using linguistic features @xcite or neural models @xcite . semafor @xcite is the most prominent rule-based generate-then-filter approach for target identification. semafor first generates a master list of all morphological variants of lus within framenet, and then performs substring search on inputs using the master list to identify candidate targets. then, semafor filters out targets using manually defined rules in addition to rules proposed by @xcite . more recent approaches @xcite have instead proposed span-based predicate classification methods which overcome the primary limitation of previous generate-then-filter methods, i.e., not being able to identify discontinuous lus. while these approaches have strong performance, they rely heavily on the domain and quantity of their training data for the model to learn to distinguish predicate spans from non-predicate spans; however, framenet is known to have limited domain scope in the training data @xcite . fortunately, candidate generation approaches avoid this problem due to their reliance on framenet's defined lexical units. frame identification is a much more studied task, especially in recent years. approaches for frame identification can be generally categorized into graph-based classification methods @xcite @xcite @xcite which utilize inter-frame relationships, syntactic methods @xcite which extract syntactic features to enhance classification models, and semantic methods @xcite where semantic information about particular frames is encoded using pre-trained language models. in our work, we explored semantic methods for two primary reasons. first, semantic methods benefit from pre-trained language models which have been exposed to and trained on billions of tokens and very diverse corpora. this enables more ro-bust and generalized representations which support a wide variety of domains, an important feature when working with framenet. second, semantic methods are easily extendable and scalable via replacing the language model with a larger model, if a particular application needs more performance, or a smaller model, if the application needs more throughput. on the contrary, graph-based methods are not necessarily as easily scalable due to their frequent reliance on computing intermediate representations for inter-frame relationships, which requires a significant data quantity. prior to our work, for frame identification the graph-based kgfi @xcite was the state-ofthe-art model, and fido @xcite was the best-performing semantic-based model, having performance on-par with kgfi. fido represents each target in a sentence as a multiplechoice classification task, wherein the model is optimized according to the highest-scoring targetframe pair. this is done by encoding textual representations of frame name, frame definition, lu name, and lu definition, along with the input sentence, into a pre-trained language model such as bert @xcite . additionally, @xcite extended the use of semantic methods to argument identification, reaching state-of-the-art performance, further showing the value of semantic methods, particularly on zero-and few-shot tasks. ## target identification target identification is the task of determining which tokens in a given sentence evoke a frame. for example, thought and discovered in figure 1 evoke the frames opinion and achiev-ing_first, respectively. our target identification model, similar to semafor, consists of two steps: generating candidate targets, followed by filtering out false-positive targets. ## frame identification frame identification is the task of identifying which frame is evoked by a given target. a target is a single word in most cases but, as discussed in section 3, can also be a span of multiple consecutive or nonconsecutive words. furthermore, a sentence may have multiple frame-evoking targets or even multiple occurrences of a frame. ## experiments for evaluating the effectiveness of our system, we conducted experiments to answer the following questions: q1) what is the coverage of our candidate generation model for target identification? as discussed in section 3.1, maintaining a very high coverage is crucial due to error propagation on downstream tasks. q2) are contextualized embeddings beneficial for filtering candidate targets in target identification models? previous approaches have found success with manual rules for candidate filtering; however, no previous works have studied the impact of pretrained language models on candidate target filtering. q3) do additional negative frames improve frame identification model performance on under-utilized samples? as discussed in section 4, we hypothesize that fido does not effectively use all of the samples in the dataset due to the lack of negative training samples on many of the frames. to verify our hypothesis, we must evaluate the impact of additional negative training samples on the under-utilized samples. q4) does additional negative sampling improve frame identification performance on rare frames? similar to q3, because our negative sampling strategy pulls uniformly from all frames while framenet is not uniformly balanced, we must verify whether this strategy is beneficial for rare frames. q5) can our frame identification model substitute the target filtering model? because target filtering and frame identification models perform similar functions, we aim to evaluate whether we can use our frame identification model to directly filter candidate targets. ## conclusion in this work, we presented a novel approach for candidate target generation which utilizes a modified prefix tree to extend support to discontinuous lexical units, resulting in a 99.4% coverage on framenet annotations. following this, our model filters out false-positive candidates, resulting in performance exceeding the previous state-of-theart solution on target identification. additionally, we identified and solved an important issue with a previous multiple-choice classification approach, resulting in increased performance, particularly on frames with few samples and targets which only evoke a single frame. finally, we explored incorporating the frame identification model into the candidate target filtering task, and found promising results, indicating that future works may be able to merge candidate target filtering, frame identification, and argument identification into one step.