How Relevant Are Selectional Preferences for Transformer-based Language Models? (2020)

Abstract: Selectional preference is defined as the tendency of a predicate to favor particular arguments within a certain linguistic context, and likewise, reject others that result in conflicting or implausible meanings. The stellar success of contextual word embedding models such as BERT in NLP tasks has led many to question whether these models have learned linguistic information, but up till now, most research has focused on syntactic information. We investigate whether Bert contains information on the selectional preferences of words, by examining the probability it assigns to the dependent word given the presence of a head word in a sentence. We are using word pairs of head-dependent words in five different syntactic relations from the SP-10K corpus of selectional preference (Zhang et al., 2019b), in sentences from the ukWaC corpus, and we are calculating the correlation of the plausibility score (from SP-10K) and the model probabilities. Our results show that overall, there is no strong positive or negative correlation in any syntactic relation, but we do find that certain head words have a strong correlation and that masking all words but the head word yields the most positive correlations in most scenarios –which indicates that the semantics of the predicate is indeed an integral and influential factor for the selection of the argument.

## introduction motivated by their semantics, the vast majority of predicates (such as verbs) have a tendency to favour certain arguments to others. consider the following examples: the athlete runs a marathon. most native speakers would readily accept example (1) as a well-formed english sentence, while example (2), even though it is syntactically correct, is more likely to be judged as awkward and ill-formed. the first example is semantically felicitous (since it is perfectly normal for athletes to run, and a marathon is something that can be run), while the second example is semantically infelicitous (both a bassoon and a banana are inanimate entities without a literal capability of motion). this preference of predicates for particular arguments is known as selectional preference @xcite . a proper understanding of this phenomenon is important within various natural language processing (nlp) applications, and selectional preferences have indeed been used as an additional knowledge source for various nlp tasks, such as word sense disambiguation @xcite and semantic role labeling @xcite . while language processing architectures prior to the neural network paradigm primarily made use of a sequential nlp pipeline, where designated modules sequentially provide increasingly complex linguistic annotations (such as part of speech tagging and syntactic parsing), more recent approaches tend to tackle nlp problems with a single, overarching neural network architecture: words are modeled as multidimensional embeddings that are fed to the neural network architecture, without any additional linguistic annotations; all the linguistic knowledge-that the neural network might exploit for its final decisions-is modeled implicitly throughout the neural network's hidden layers. this line of research has culminated in the recent contextual embedding language architectures, such as elmo @xcite , bert @xcite and roberta @xcite , which adapt the individual word embeddings in order to yield a fine-grained representation of the meaning within the context of a specific text fragment, and these embeddings can subsequently be exploited for language prediction tasks. the implicit nature of the linguistic knowledge contained within these recent approaches opens up a number of questions with regard to the relevance of selectional preferences in the neural network paradigm. whereas previous nlp approaches generally considered selectional preferences as a standalone module that may be easily evaluated in vitro, this is not the case for current neural approaches: the neural network functions as a black box, and as such it is impossible to find out exactly what kind of information has led the model to make a final decision, and whether selectional preference information played a significant role in it. it is therefore necessary to design specific experimental settings in order to inspect what information the network actually made use of. lately, there has been a great deal of interest in comprehending the way contextual language models encode linguistic knowledge. however, most work has focused on syntactic phenomena, such as the identification of constituents and dependencies, and syntactic agreement; investigating how these models encode semantics proves to be a more difficult issue, and research into the question is much less common @xcite . in this paper, our goal is to investigate whether transformer models include selectional preference information in their embeddings and whether they rely on it in order to make their predictions-and if so, to what extent that information is relevant or vital. we focus on the bert architecture, and consider the influence of selectional preferences for the model's standard masked language model pretraining objective. our sentences consist of pairs of head and dependent words from the sp-10k dataset, annotated with a plausibility score by human judges, and spanning five different types of syntactic relations @xcite . we create prompt sentences based on these word pairs and a syntactically annotated version of the ukwac corpus @xcite in order to extract full sentences with said pairs. we then feed the prompt sentences masking the dependent word of the word pair to bert. then, we are able to observe whether there is a strong correlation between the plausibility score of a word pair (how likely it is for the head and the dependent word to exist in an utterance) and the probability of the dependent word in the context of a sentence, as assigned by bert. subsequently, we can make assumptions on how much the head word contributed to the prediction of the dependent word, by applying attention masks to the input sequences of the model. our findings show that, over all sentences and word pairs, there is no strong positive or negative correlation between the plausibility score (as retrieved from sp-10k) and the probability of the dependent word; however, we make note of some head words and phenomena that showed significant positive or negative correlation (>0.4) in some syntactic relations. moreover, our results show that masking all words but the head word yields the most positive correlations in most scenarios, which indicates that the semantics of the predicate is indeed an integral and influential factor for the selection of the argument. this paper is structured as follows. in section 2, we provide an overview of related work, both with regard to selectional preference modeling and with regard to interpretability of neural architectures. in section 3, we outline the methodology of our research. section 4 provides an overview of the results, including a quantitative comparison of the various experimental settings as well as a qualitative discussion. section 5 describes some of the challenges that we faced in our research, as well as a general discussion of our results. finally, section 6 summarizes our findings and our key points, and provides a number of avenues for future work. ## results as seen in note that for the amod syntactic relations (noun and its modifier) the correlation scores are among the highest, reasonably so, as the noun should place stronger constraints on its modifier. the nsubj relations show slightly more positive correlations than the dobj ones, probably due to the constraint of animacy for a subject (word pairs included a variety of animate and inanimate subjects). the two-hop relations, nsubj amod and dobj amod show lower correlations, still relatively high for the context attention mask setting, which is in accordance with the hypothesis of @xcite that selectional preferences span further than one-hop relations. taking a closer look at the head words of the word pairs, we searched for strong positive or negative correlations for each head word that exists in at least two different word pairs, per syntactic relation. we examining the dobj relations, verbs (head words) showed inconsistent correlations among the different attention mask scenarios; out of the few verbs that showed consistently positive or negative correlation, we were not able to identify semantic clusters of verbs, neither differences based on verb transitivity (monotransitive/ditransitive). the presence of a strong correlation relied more on bert's semantic knowledge rather than world knowledge or utterance plausibility; for example, the word pair "blame customer" has (correctly) a moderate plausibility score (6.75), is found twice in the ukwac corpus, but the assigned probability by bert of the word "customer" is very low in the standard and context attention mask scenarios. the word pair "blame management", on the other hand, with slightly lower plausibility (6.25) is assigned a proportionally good probability. this leads us to the conclusion that, even though both syntactic pairs are grammatically correct and have commonly used words, the pretrained model has learned that "management" (someone in control and responsible of a service) is a more probable direct object for the verb "blame" than the word "customer (the receiver of a service), especially when the only given context is the verb. when attention to the head word was removed, there was no strong negative correlation between "blame" and the given plausibility score. concerning the amod word pairs, again no semantic class of nouns appear consistently in the positive or relative correlation groups. an interesting observation is that high-frequency adjectives of size and age, such as "small", "big", "old" and "new" were almost always assigned a high probability by bert, but the variations in plausibility score (from 8.25 to 4.25) led to strong positive or negative correlations, especially since word pairs with these adjectives are quite frequent in our corpus, for example "new house", "small bird" and "new face" had many occurrences in the corpus and a strong positive correlation (high plausibility/high probability), "new material" (6.5) and "old daughter" (4.25) had lower plausibility scores and subsequently lower probabilities, in all attention mask scenarios. in the nsubj amod word pairs, again we see that high-frequency descriptive adjectives (dependent words) are still assigned higher probabilities, even though the plausibility scores are more mediocre for the word pairs of this relation, therefore high-frequency adjectives can be found in both the strong positive and negative correlation groups. we also do not notice distinctive semantic classes among the verbs (head words), and neither can we make assumptions based on the animacy of the subject, since the adjective modifiers do not follow such a constraint ("new", "local", "national", "exact", "different") and the given verbs do not have the animacy constraint either ("bring", "attract" had a strong positive correlation, "increase", "reflect" a strong negative). some verbs that do prefer animate objects were found to have a strong positive correlation (e.g. "compare", "operate"), others to have a strong negative (e.g. "lift"). concerning the different attention scenarios, there is a noticeable positive shift in correlations (+.30, +.20) with the context attention mask compared to no mask or masking the head word, which hints at the influence that the verb had in the predictions, and how the context (including the one-hop dependency to the verb subject) produced less polarizing probabilities. finally, for the dobj amod word pairs, as in the direct object word pairs, we do not notice verb grouping based on semantics or transitivity. many of the verbs with strong positive ("teach", "promise") or negative correlation ("claim", "confirm") are verbs with varied subcategorization frames. in this syntactic category, we observe the smallest positive shifts with the use of the context attention mask, and even a decrease in correlation (-.03) in the micro-averaged results. however, the results still show a weak positive correlation similar to the ones of the other syntactic relations, for the most part; this observation supports the fact that the role of the verb is quite important for predictions. ## conclusion in this paper, we explored whether selectional preference information makes part of the linguistic information that is learned by a transformer model, by examining the correlation of the plausibility of a head-dependent word pair and the assigned probability of the dependent word in a sequence including the head word. our overall results did not show a strong correlation that would definitively prove or disprove the presence of selectional preferences, but there are indications that bert's embeddings have captured enough syntactic-semantic information to be able to assign probability based on "the right fit" for a head word. in addition, some specific cases between syntactic relations and metaphorical uses have given us the incentive to further investigate these phenomena, and to delve deeper in the architecture of the model for answers. the code we used for our experiments will be made available at a github repository foot_4 . as for future work, a further exploration of the attention output would be quite beneficial in understanding the attention weights during prediction, in each scenario, and how they differ per layer and attention head. researchers who have attempted to thoroughly analyze bert's attention behavior in syntactic probing tasks have noted that the attention maps have "a fairly thorough representation of english syntax" [sic] @xcite , and have noticed that specific attention heads and layers specialize on learning specific linguistic knowledge, such as syntactic dependencies @xcite . visualizing attention for 12 layers and 12 attention heads, over 145k sentences, is a complicated task which we intend to tackle soon, with the help of visualization libraries @xcite .