Comparing Distributional and Curated Approaches for Cross-lingual Frame Alignment (2022)

Abstract: Despite advances in statistical approaches to the modeling of meaning, many ques- tions about the ideal way of exploiting both knowledge-based (e.g., FrameNet, WordNet) and data-based methods (e.g., BERT) remain unresolved. This workshop focuses on these questions with three session papers that run the gamut from highly distributional methods (Lekkas et al., 2022), to highly curated methods (Gamonal, 2022), and techniques with statistical methods producing structured semantics (Lawley and Schubert, 2022). In addition, we begin the workshop with a small comparison of cross-lingual techniques for frame semantic alignment for one language pair (Spanish and English). None of the distributional techniques consistently aligns the 1-best frame match from English to Spanish, all failing in at least one case. Predicting which techniques will align which frames cross-linguistically is not possible from any known characteristic of the alignment technique or the frames. Although distributional techniques are a rich source of semantic information for many tasks, at present curated, knowledge-based semantics remains the only technique that can consistently align frames across languages.

## introduction to the workshop broadly speaking, research in computational linguistics encompasses two main streams: (1) work that relies primarily on operationalizing prior knowledge about language and its use, such as rulebased parsers @xcite , scripts @xcite , planning, scenarios, scripts for virtual assistants, and framenet (fn) frames @xcite , as well as lexical databases like wordnet @xcite , verbnet @xcite , and propbank @xcite , among others; and (2) work that seeks to derive knowledge directly from data (text, speech, and increasingly vision) with unsupervised (or distantly supervised) methods, which are distributional and frequency-based, in linguistics @xcite , cognitive science @xcite , and computational linguistics, notably vector embeddings like bert @xcite . they are often complementary; e.g. @xcite combine pos tagging and lemmatization to improve vector embeddings and @xcite combine syntactic knowledge with neural language models to improve accuracy. despite great advances in statistical approaches, many questions remain unresolved: in "multi-sense language modelling", andrea lekkas, peter schneider-kamp and isabelle augenstein use pretrained embeddings and also calculate new ones, combining them with many facets of the curated wordnet lexicon. they report on extensive testing of five different system architectures against a most-frequent-sense baseline on both next word prediction and wordnet sense prediction, on both the semcor and semeval datasets. ## overview of framenet framenet @xcite is a research and resource development project in corpus-based computational lexicography grounded in the theory of frame semantics @xcite . the semantic frame, a script-like knowledge structure that facilitates inferencing within and across events, situations, states-of-affairs, relations, etc., is at the core of the theory @xcite . fn defines a semantic frame in terms of its frame ele-ments (fes), or participants (and other concepts) in the scene that the frame captures; a lexical unit (lu) is a pairing of a lemma and a frame, characterizing that lu in terms of the frame that it evokes. the definition of a frame, represented both in prose and in structured relations between frames, is a bundle of inferences relating the frame elements whenever the frame is evoked. example 1 illustrates the frame semantics analysis for the verb buy, which fn defines in the commerce_buy frame, with the fes buyer, seller, goods, and money. 1 1. chuck buyer bought a car goods from jerry seller for $2,000 money ## cross-linguistic frame alignment as interest in frame semantics @xcite and the original framenet for english @xcite grew, research groups around the world started developing fn-like resources for their languages. such resources in many languages have made it possible to address the question of whether semantic frames are universal or merely languagespecific lexical phenomena. with these databases at hand, we may operationalize the question as: to what extent can these lexical databases be aligned to form a multilingual framenet lexical database connecting all of the languages, while also accounting for language-specific differences and domainspecific extensions to framenet? the goal of the multilingual framenet (mlfn) project @xcite ) was to answer this question by building a cross-linguistic database. though this database succeeded in partially aligning frames, the question remained of how to assess the validity and utility of the alignments. baker and lorenzi (2020) described a database of vectors that represent alignments between pairs of frames in different languages (e.g., english-spanish, english-japanese, etc.). 2 baker and lorenzi (2020) also described developing vitoxf, a freely available visualization tool for all of the alignments. 3 the tool allows interactive exploration of the alignments between english and one of seven other languages in the database. the alignments were created using 11 different methods, four (4) resource-based and seven (7) vector-based. the rest of this section characterizes these alignment methods; this is a revised version of the descriptions of alignment methods in @xcite . ## results to evaluate the alignments created by the various techniques described in section 3 (above), fn researchers defined a set of "gold-standard" frame alignments for a small set of frames from spanish framenet (sfn) aligned to english frames from bfn. 6 we determined gold-standard frame matches manually by comparing all of the information associated with the frames of each language, including frame definition, frame elements, lexical units with their translations, and frame relations (if any). since the time for a manual review precludes comparing all frames to all other frames, we only considered those frames with lexical translation overlap. 7 for each gold-standard alignment, we examined the full set of alignment techniques provided by vitoxf for the sfn frame. with vitoxf, each technique will align a sfn frame to different list of bfn frames, and each such pairing will have a score. the techniques have very different scores, even when normalized, so the best way to compare techniques is by how they order the proposed bfn frames to align. in what follows, we simplify the evaluation of a technique to the relative rank (1st, 2nd, etc.) of the gold-standard bfn frame. all the techniques show promise in accurately aligning certain frames and all perform less well on the inexact match, i.e., sfn motion_manner to bfn self_motion. unlike the english frame, the sfn frame does not allow complex path information, although many lus in the sfn frame have translation equivalents in bfn self_motion (e.g. spanish correr.v -> english run.v). also, no single technique ranks the gold standard as the strongest match for all of the listed frames. sfn desiring and similarity align correctly by all of the techniques listed; in contrast, sfn activity_finish only aligns unambiguously using just one technique, i.e., average core fe muse similarity. at least based on this limited data, it is not possible to predict which techniques will do a good job of aligning which frames cross-linguistically from any known characteristic of the alignment technique or the frames. this is just one example of the complex questions involved in comparing different approaches to alignment. ## concluding remarks and future work this paper has introduced the workshop exploring the strengths and weaknesses of different techniques for modeling meaning. specifically, the research of the berkeley framenet group has compared distributional approaches and curated approaches for cross-lingual frame alignment, illustrating the results from four different alignment techniques for five spanish framenet and bfn frames, finding that no distributional technique reliably predicts the gold-standard alignment. this initial study on a small set of frames in only two languages is suggestive, and points to the need for wider exploration of techniques for aligning lexical units and frames across languages in frame-based resources. we look forward to further development of hybrid semantic representations combining the advantages of distributional and curated semantic techniques, both for the alignment task and a wider range of applications.