Syntagmatic Word Embeddings for Unsupervised Learning of Selectional Preferences (2021)

Abstract: Selectional Preference (SP) captures the tendency of a word to semantically select other words to be in direct syntactic relation with it, and thus informs us about syntactic word configurations that are meaningful. Therefore SP is a valuable resource for Natural Language Processing (NLP) systems and for semanticists. Learning SP has generally been seen as a supervised task, because it requires a parsed corpus as a source of syntactically related word pairs. In this paper we show that simple distributional analysis can learn a good amount of SP without the need for an annotated corpus. We extend the general word embedding technique with directional word context windows giving word representations that better capture syntagmatic relations. We test on the SP-10K dataset and demonstrate that syntagmatic embeddings outperform the paradigmatic embeddings. We also evaluate supervised version of these embeddings and show that unsupervised syntagmatic embeddings can be as good as supervised embeddings. We also make available the source code of our implementation.

## introduction selectional preference (sp) @xcite encodes the syntagmatic relatedness between two words. relations between words are either syntagmatic or paradigmatic @xcite . two words are said to be paradigmatically related if one word can replace the other in a sentence. words belonging to a narrow semantic class, such as 'cat', 'dog' can often be substituted with each other in a sentence. syntagmatic relations are between syntactically related co-occurring words in a sentence. such word relations encode both syntactic and semantic aspects of words. a noun may be modified by an adjective, but any particular instance of a noun tends to go more with some adjectives than others. for example black dog is more likely than green dog. sp deals with such semantic preferences between syntactically related word pairs. common sp relations include 'adjective-noun', 'subjectverb', 'verb-object'. sp finds use in important nlp tasks like sense disambiguation @xcite , semantic role classification @xcite , co-reference resolution @xcite , etc. a computational method to induce sp from instances of syntactically related word pairs in a parsed corpus was introduced by @xcite . in order to generalize to unseen data, this method made use of ontological classes obtained from wordnet @xcite . @xcite showed that the dependence on external knowledge resources could be removed by learning the classes from the corpus itself using the em algorithm. @xcite showed that generalization is also possible via co-occurrence similarity between seen and unseen words. sp models are usually evaluated using the pseudo-word disambiguation task (van de cruys, 2014) which requires the identification of the more probable dependent word, from a less probable (random) word, given the head word and a syntactic relation. the dataset is generally created from the unseen part of a parsed corpus used for learning the model. therefore this task measures only how well the model fits the corpus, which may be biased, and not how well it learns sp as perceived by humans. recently, @xcite introduced sp-10k, a dataset for sp evaluation across 5 syntactic relations with a total of 10,000 items each with a human-annotated plausibility score. sp-10k measures the correlation between a model's sp score for a given word pair and the average human score. therefore it is a better test for sp learning. the current state-of-the-art on sp-10k is reported by multiplex word embeddings (mwe) @xcite . it is a negative sampling based word embedding model, trained on relationspecific word pairs from a parsed corpus. compared to unsupervised embedding models such as word2vec @xcite and glove @xcite , mwe provides a substantial boost in sp learning as it has access to syntactic relations. it also improves over d-embeddings @xcite which is a supervised embedding model. however, a dependency-parsed corpus is not readily available in many languages. therefore the need for an effective unsupervised sp induction technique is palpable in the wider nlp community. in this work we show that unsupervised word embeddings can easily be extended to get better at learning sp. we do this by taking directional (left/right) word context windows unlike symmetric windows of word2vec, glove, etc. having directional context windows gives two embeddings per word, one of its left context and other of its right context. this allows us to approximate syntactic relations with directions; all relations that happen to the left of a word are captured by the left embedding and those that happen to the right of a word are captured by the right embedding. then the cosine similarity between the right embedding of a word and left embedding of another word indicates how likely the two are to be syntagmatically related. in summary, our contributions are: 1) we provide a simple and effective method to capture selectional preference, called syntagmatic embeddings 2) demonstrate that syntagmatic embeddings are superior to paradigmatic embeddings 3) we also show that our unsupervised syntagmatic representations can be as good as their supervised counterparts, therefore showing that a good range of sp information can be learned even without a dependency-parsed corpus. ## syntagmatic representation symmetric and non-directional context windows in embedding techniques, such as glove, relate words that have similar (paradigmatic) contexts. context words are other words that are in the immediate vicinity of a target word. a symmetric window considers equal number of words on the left and right as context words. though syntagmatically re-lated words may have similar contexts, a symmetric window tends to encode more of paradigmatic relations. but these paradigmatic embedding spaces do encode syntagmatic properties to a certain degree. for example, we may find that the cosine similarity between 'coffee' and 'cup' is generally greater than 'coffee' and 'car'. these embeddings are considered unsupervised as they are learned from a plain un-annotated corpus. since their contexts are not dictated by syntactic relations they are generally inferior, at learning sp, compared to an embedding technique that has access to such information @xcite . also, there is no direct way to extract syntagmatically related words. the nearest neighbours of a given word will largely be all paradigmatically related. though it may include, given a larger context window, associated words ('coffee', 'cup') which have a syntagmatic nature. ## sp evaluation examples of word association in the previous section gives a qualitative feel about the degree to which syntagmatic embeddings can capture selectional preference. in the next section we follow this up with detailed analysis using quantitative studies. ## experiments in the following experiments, we compare our syntagmatic embeddings with its paradigmatic counterpart, identify its best parameters, distinguish weighted from unweighted factorisation, evaluate against baseline embeddings and test how our unsupervised sp learning method compares with a supervised model. the parameters involved in the factorisation of the word co-occurrence matrix are: 1) size of the co-occurrence window (ws), 2) termweight or the co-occurrence weighting function (tw), 3) dimensionality of the embedding space or the number of principle components (dim), and 4) the exponential weight on singular values (p). we experiment with the following parameter values: @xmath0, 2, 3, 4], @xmath1, 50, 100, 300], @xmath2.5, 0, 0.5, 1], @xmath3, log, pmi, ppmi]. in termweights raw denotes the co-occurrence frequency of the word as it is , log is the log 2 of the raw cooccurrence frequency, pmi is the point-wise mutual information given by equation 4 where subscript '*' stands for a summation across a particular axis, and ppmi is the positive-only variant of pmi as given by equation 5. ## related work there have been previous studies that explored syntagmatic representations. @xcite ; @xcite viewed syntagmatic representations as firstorder word co-occurrence statistics, and paradigmatic representations as second-order statistics. first-order models represent words using text units in which they appear. text units are generally documents or large regions of text, like paragraphs. thus, first order statistics come from a worddocument co-occurrence matrix, whereas paradigmatic representations come from word-word cooccurrence matrix and hence called second order. while their evaluation of paradigmatic representation as second-order statistics was appropriate, their claim of syntagmatic representation as first-order statistics is not well justified. this is because the evaluation datasets they used for first-order models were a mix of (mostly) paradigmatic and syntagmatic relations, and not purely syntagmatic. a large-scale study by @xcite showed that fine-tuned second-order statistics can capture both syntagmatic and paradigmatic relations. different parametrisations, mainly window size and dimensionality reduction, were shown to adapt the second-order statistics to either relations accordingly. the notion of syntagmatic representation explored in our work is adapted from @xcite , in which the syntagmatic representation is introduced qualitatively without resorting to any quantitative studies. our study on the other hand applies syntagmatic representation to the task of selectional preference, exploring various model parametrisations. ## discussion our experiments have shown that a weakly structured model can be as good as a strongly structured model. the spvec model, though unsupervised, incorporates a simple linguistically motivated bias/structure -directionality or word order. such a weakly biased model, when coupled with low-rank embedding process, seems to pickup appropriate linguistic structure by effectively getting rid of noise. but why did the supervised mod-els not have a bigger advantage when compared to the unsupervised model? we can hypothesize that words that are not directly related by a dependency relation but are in the vicinity of a target word make substantial contribution to the semantics of the word which may not be captured by a dependency-parsed model. it can also be because the low-rank embedding process is as good at removing noise as a dependency parse. a closer look at the results reveal that amod and dobj relations do benefit from supervision, although it is minor. the effect of window-size on each of the dependency relation, may help us to better understand this (figure 4 ). in the unsupervised model, amod relation is maximized with a window-size of 1, but the results reported in weighted factorisation of word co-occurrences is generally found to produce high quality word embeddings. previously such embeddings showed improvements in tasks such as word similarity and solving word analogies. but we have shown that, when it comes to selectional preference and syntagmatic embeddings, weighted factorisation may be detrimental. we also observe that appropriate co-occurrence term-weights are crucial for the performance. ppmi has been shown to work well for tasks that test paradigmatic nature such as word similarity @xcite . @xcite remarked that log is better for solving word analogies than ppmi. our experiments show that log is also valuable for learning selectional preference. here we have tested our syntagmatic embeddings only on english, but it should be directly applicable to other word-ordered languages also. ## conclusion in this paper, we have introduced syntagmatic word embeddings, a simple and effective method, for learning selectional preference (sp). our model is simple because it captures sp by direct factorisation of a word co-occurrence matrix. we have showed that by incorporating a weak linguistic bias of directionality as a proxy for syntactic relations, our model can be made as effective as a model with access to syntactic relations. this is important because sp has always been seen as a task that requires a dependency-parsed corpus, our work shows that it need not be the case. we hope that syntagmatic embeddings will be a valuable source of selectional preference information for resource-poor as well as resource-rich languages. we also hope that the structural bias of directionality will be further explored in simple models for other nlp tasks, instead of relying on models that are complex and opaque to interpretation.