Where’s the Learning in Representation Learning for Compositional Semantics and the Case of Thematic Fit (2022)

Abstract: Observing that for certain NLP tasks, such as semantic role prediction or thematic fit estimation, random embeddings perform as well as pre-trained embeddings, we explore what settings allow for this, and examine where most of the learning is encoded: the word embeddings, the semantic role embeddings, or “the network”. We find nuanced answers, depending on the task and its relation to the training objective. We examine these representation learning aspects in multi-task learning, where role prediction and role-filling are supervised tasks, while several thematic fit tasks are outside the models’ direct supervision. We observe a non-monotonous relation between some tasks’ quality scores and the training data size. In order to better understand this observation, we analyze these results using easier, per-verb versions of these tasks.

## introduction we examine to what extent models trained on a simplified semantic role labeling (srl) task can estimate thematic fit (aka semantic fit), as the training set size grows -and where most of the learning is stored: in the word embeddings, the thematic role embeddings, or elsewhere in the neural net. a major goal of natural language processing (nlp) is to understand the semantics of language. one traditional nlp task around this is srl, which labels word spans in a sentence with thematic roles. consider the sentence "i cut the cake with a knife". we can interpret 'cut' as the action, 'i' as the agent (the performer of the action), 'cake' as the theme of the action (the thing that underwent the action), and 'knife' as the instrument of * these authors contributed equally to this work the action. these words, labeled with roles such as agent, theme, and instrument, would be our representation of the event that the sentence conveys. other sentences with similar meanings, e.g., "the cake was cut with the knife by me", should have the same (or very similar) event representations. in this work, we focus on model training with a simplified version of srl: each event is represented only by the lemmatized syntactic head of each event argument (including the predicate), and the semantic roles are the simplified propbank roles (arg0, arg1, etc.). the reason for this is the current limitations of available evaluation sets for thematic fit: they are all comprised of lemmatized syntactic argument heads as well. thematic fit is related to srl. this task aims to identify how well a given word or concept fits into a role of an event. in our example sentence, consider these potential replacements for 'knife': scissors, fork, and brick. as humans, we understand that while 'knife' is the most typical object for this situation, both 'scissors' and 'fork' could also fit, even if not as naturally. this is because we have the general intuition that all three objects are plausible instruments for cutting. more so, we know that 'brick' is unlikely to fit given the context of cutting a cake. since thematic fit datasets are scarce, one challenge in computational linguistics (and computational psycholinguistics) revolves around how machine learning models can learn thematic fit indirectly -perhaps from srl training. to the best of our knowledge, the state-of-art in this line of work is the residual role-filler averaging model (resrofa-mt) proposed by @xcite , with an adjusted embeddings representation and training data annotation in @xcite . it has been repeatedly observed that in some settings, random word embeddings perform as well as pretrained ones, or very nearly, including in our baselines @xcite @xcite . in this paper, we design experiments to answer the following questions: q1. why is this so in our compositional semantics and psycholinguistic tasks? q2. for such semantic tasks and architecture, where is the learning encoded? is it in the word embeddings, role embeddings, or elsewhere in the neural network? q3. training set size effect: is more data better for this indirect setting and tasks? in this work, 1. we compare updating the word embeddings during training to freezing them. 2. we modify the resrofa-mt model architecture in various ways to understand what contributes the most to the learning: the pretrained (or random) word embeddings, the thematic role embeddings, or the rest of the network. 3. in order to be able to train on larger data, we optimized the code of @xcite and @xcite . we release our optimized codebase * , which trains 6 times faster and includes ablation architectures and a correction to the training data preparation step. ## related work in event representation models, the main goal is to predict the appropriate word in a sentence given both the role of that word and the surrounding context in the form of word-role pairs. one of the best early neural models was the non-incremental rolefiller model (nnrf), by @xcite . this model was based on selectional preferences, or a probability distribution over the candidate words. however, one drawback of this model is that representations of two similarly-worded sentences differing hugely in meaning would closely resemble each other, e.g., "kid watches tv" and "tv watches kid". another drawback is that the embeddings of the word-role pairs are summed together to represent the sentence, and so the resulting event representation vector does not weight the input vectors differently based on their importance and is not normalized for varying numbers of roles in a sample. hong et al. ( 2018 ) extend this model in three ways: first, in addition to the word prediction task of nnrf, the task of role prediction given the corresponding word is added, and the two tasks are trained simultaneously (multi-task learning). this model is known as the non-incremental role-filler multitask model (nnrf-mt). second, they apply the parametric rectified linear unit (prelu) non-linear function to each word-role embedding, which acts as weights on the composition of embeddings, and subsequently average the embeddings, which normalizes for variable length inputs. this model is called the role-filler averaging model (rofa-mt). third, in an effort to tackle the vanishing gradient problem, residual connections between the prelu output and the averaging input were added together. this third iteration is known as the resrofa-mt model. they showed that it performs the best on our thematic fit tasks, and so we use it as our baseline. our work differs from @xcite and @xcite in that while they focused more on state-of-the-art performance through new modeling and annotation methods, we aim to understand what controls the learning in such networks. also, although @xcite in private communication that they found pre-trained and random embeddings performance similar in preliminary studies, none of the surveyed previous work published experiments with pre-trained embeddings. we are the first to do so (using glove) and compare that to using random embeddings. previous work suggests a difference between "count" and "predict" models, where "count" models represent lexical semantics in terms of raw or adjusted unsupervised frequencies of correlations between words (such as local mutual information; @xcite and syntactic or semantic phenomena; "predict" models involve supervised training to achieve their representations, e.g., neural models. @xcite do a systematic exploration of tasks vs. state-of-the-art count and predict models and find that predict models are overall superior; for thematic fit, predict models are the same or better than count models on the best unsupervised setup for the task, although they are easily beaten by third-party baselines based on supervised learning over count models. more recently, @xcite demonstrate that predictmodels are not reliably superior to count-models, but depend on the task and the way the models are trained. they also show that even recent contextual models (e.g., bert) are not necessarily better for out-of-context tasks than well-tuned static representations, predict or otherwise. see appendix a for details on why we do not use bert in this work. ## datasets we use the rollenwechsel-english, version 2 (rw-eng v2) corpus @xcite as the training set for all our experiments. this corpus is sentence-segmented, annotated with morphological analyses, syntactic parses, and syntax-independent propbank-based semantic role labeling (srl). the syntactic head word of each semantic argument is determined by using several heuristics to match the parses to the semantic argument spans. note that a sentence may have multiple predicates (typically verbs) and therefore multiple semantic frames (sometimes called "events"), each with its own semantic arguments, whose span may overlap the argument span of other frames in the sentence. the first version of this corpus contained nltk lemmas, maltparser parses, parts-of-speech (pos) tags, and senna srl tags @xcite @xcite . the second version added layers from more modern taggers: morfette lemmas, spacy syntactic parses and pos tags, and lsgn srl tags @xcite @xcite . in our experiments here we use the lemmas of the semantic arguments' head words in v2. the sentences themselves are taken from both the ukwac @xcite and the british national corpus (bnc). this corpus contains 78m sentences across 2.3m documents. this includes 210m verbal predicates with 700m associated role-fillers. we use the same training, validation, and test split as @xcite . that is, we have 99.2% ( 201.5m samples) in the full training set, 0.4% in validation, and 0.4% in testing. we run our training experiments on different subsets of the training data, ranging from 0.1% up to the full dataset. we cap our vocabulary size at the 50,000 most common words in that specific subset. we used the following psycholinguistic test sets: padó @xcite 414 verb-argument pairs and the associated judgement scores. these were constructed from 18 verbs that are present in both framenet and propbank. for each verb, the three most frequent subjects and objects from each of the underlying corpora were selected. this process yielded six arguments per verb per corpus, with some overlap between corpora. for each verbargument pair, a judgement was collected online with an average of 21 ratings per item for the ar-gument in subject and object role. the rating was collected on a likert scale of 1-7 with the question "how common is it for [subject] to @xcite ?" or "how common is it for [object] to be [verbed]?" @xcite 1444 pairs of verbargument pairs in a similar format to padó. these were created using a similar rating question as the padó dataset, but is a compilation of ratings collected over several studies with considerable overlap and heterogeneous selection criteria. ferretti-instruments and ferretti-locations (ferretti et al., 2001) 274 predicate-location pairs and 248 predicate-instrument pairs. based on the mcrae dataset (psychological norms). gds (greenberg et al., 2015) 720 predicateobject pairs and their ratings. only objects (no subjects), matched for high and low polysemy and frequency, well fitting vs. poorly fitting. greenberg and mcrae overlap by about a third, but the human scores are obtained from new surveys. bicknell (bicknell et al., 2010) 64 cases. congruent vs incongruent patient in an agent-verb-patient paradigm. hand crafted, not corpusbased, designed for event-related potentials-based neurolinguistic experiments. ## modeling and methodology in this setup, an input event is represented as role-word pairs, where the role is one of the following propbank @xcite roles: arg0, arg1, argm-mnr, argm-loc, argm-tmp, and the predicate. the word is the argument's syntactic head's lemma. both the role and the head word are taken from rw-eng v2. * we train a feed-forward network in a multi-task learning setting to optimize word and role prediction accuracy. for target word prediction we give the prediction layer the target role and a context vector formed as a multiplication of the input wordrole pairs. similarly, for target role prediction we feed the same context vector along with the target word, following the resrofa-mt architecture @xcite (figure 1a ). since the network initialization is random, we perform 5 runs of each experiment and report the mean with a 95% confidence interval. following @xcite ; @xcite , we test each model on the psycholinguistic datasets (section 3), for which the models were not directly optimized. the idea behind using the latter test battery is that the model, even though trained on (simplified) srl and word prediction (aka role-filling) tasks, is expected to be able to make indirect generalizations about predicate-argument fit level from the training data and the related objectives. these psycholinguistic tasks are evaluated with spearman's rank correlation between the sorted human scores and the sorted model scores, except for bicknell, for which we take accuracy of predicting which argument in each patient role-filler pair is (more) congruent @xcite . all prior work with the resrofa-mt model uses two random word embedding sets (one for input words and one for the target word) and similarly two role embedding sets. see figure 1a . our implementation differs in these key aspects: ## conclusions and future work in this work, we explored why random word embeddings counter-intuitively perform as well as pretrained word embeddings on certain compositional semantic tasks (some being outside the models' explicit objective), where the learning is actually stored (teasing apart the word embeddings, role embeddings, and the rest of the network), and how training set size affects performance on these tasks. we found out that tuning (or further tuning) the word embeddings helps and can bridge the gap between random and pretrained embeddings. moreover, our tuned embedding space is different from pretrained embeddings like glove. we saw that the target role is more important than the input roles on our tasks. furthermore, our experiments suggested that much of the learning happens also in the rest of the network outside word and role embedding layers. no single factor (word and role embeddings or the network) is most important for all tasks. training set size had a surprising negative effect on padó and mcrae beyond 20% of the training data. we attempted explaining this with an alternative evaluation method, but this remains to be explained further. we release our code, including our preferred network architecture -a modified version of resrofa-mt with shared embedding layers. one avenue in which we want to invest is to better understand the complex relationship between word/role accuracy and our psycholinguistic tasks. while our initial hypothesis was that training the network to minimize loss on word/role prediction would also optimize performance on all our tasks, this did not always hold. we suspect that the groundedness is the missing link for (artificially and naturally) learning psycholinguistic tasks, and therefore adding grounding seems promising to us. another future avenue is to investigate the high variability in psycholinguistic task performance compared to the fairly stable results on the directly optimized-for word and role prediction tasks.