Multidirectional Associative Optimization of Function-Specific Word Representations (2020)

Abstract: We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures. Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO compositions lie close together. The model retains information about word group membership even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure. We show the robustness and versatility of the proposed framework by reporting state-of-the-art results on the tasks of estimating selectional preference and event similarity. The results indicate that the combinations of representations learned with our task-independent model outperform task-specific architectures from prior work, while reducing the number of parameters by up to 95%.

## introduction word representations are in ubiquitous usage across all areas of natural language processing (nlp) @xcite @xcite . standard approaches rely on the distributional hypothesis @xcite and learn a single word vector space based on word co-occurrences in large text corpora @xcite @xcite . this purely context-based training produces general word representations that capture the broad notion of semantic relatedness and conflate a variety of possible semantic relations into a single space @xcite . however, this mono-faceted view of meaning is a well-known deficiency in nlp applications @xcite as it fails to distinguish between fine-grained word associations. in this work we propose to learn a joint functionspecific word vector space that accounts for the study eat need food help support assistance subject art science researcher chicken scientist implementation cat chicken figure 1 : illustration of three neighbourhoods in a function-specific space trained for the svo structure (marked (s), (v), $(o)). the space is optimised such that vectors for plausible svo compositions will be close. note that one word can have several vectors, for example chicken can occur both as s and o. to validate the effectiveness of our representation framework in language applications, we focus on modeling a prominent linguistic phenomenon: a general model of who does what to whom (gell-word nearest neighbours subject memory dream, feeling, shadow, sense, moment, consciousness country state, nation, britain, china, uk, europe, government student pupil, participant, learner, candidate, trainee, child verb see saw, view, expect, watch, notice, witness eat drink, consume, smoke, lick, swallow, cook, ingest avoid eliminate, minimise, anticipate, overcome, escape object virus bacteria, infection, disease, worm, mutation, antibody beer ale, drink, pint, coffee, tea, wine, soup, champagne joint svo study (v) researcher (s), scientist (s), subject (o), art (o) eat (v) food (o), cat (s), dog (s) need (v) help (o), implementation (s), support (o) the quantitative results are reported on two established test sets for compositional event similarity @xcite . this task requires reasoning over svo structures and quantifies the plausibility of the svo combinations by scoring them against human judgments. we report consistent gains over established word representation methods, as well as over two recent tensor-based architectures @xcite which are designed specifically for solving the event similarity task. furthermore, we investigate the generality of our approach by also applying it to other types of structures. we conduct additional experiments in a 4-role setting, where indirect objects are also modeled, along with a selectional preference evaluation of 2-role sv and vo relationships @xcite , yielding the highest scores on several established benchmarks. ## background and motivation representation learning. standard word representation models such as skip-gram negative sam-pling (sgns) @xcite , glove @xcite , or fasttext @xcite induce a single word embedding space capturing broad semantic relatedness @xcite . for instance, sgns makes use of two vector spaces for this purpose, which are referred to as a w and a c . sgns has been shown to approximately correspond to factorising a matrix @xmath0, where elements in m represent the co-occurrence strengths between words and their context words @xcite . both matrices represent the same vocabulary: therefore, only one of them is needed in practice to represent each word. typically only a w is used while a c is discarded, or the two vector spaces are averaged to produce the final space. levy and goldberg (2014a) used dependencybased contexts, resulting in two separate vector spaces; however, the relation types were embedded into the vocabulary and the model was trained only in one direction. @xcite proposed to learn separate sets of relation vectors in addition to standard word vectors and showed that such relation vectors encode knowledge that is often complementary to what is coded in word vectors. @xcite and @xcite described related task-dependent neural nets for mapping word embeddings into relation-specific spaces for scoring lexical entailment. in this work, we propose a task-independent approach and extend it to work with a variable number of relations. neuroscience. theories from cognitive linguistics and neuroscience reveal that single-space representation models fail to adequately reflect the organisation of semantic concepts in the human brain (i.e., semantic memory): there seems to be no single semantic system indifferent to modalities or categories in the brain @xcite . recent fmri studies strongly support this proposition and suggest that semantic memory is in fact a widely distributed neural network @xcite @xcite @xcite , where sub-networks might activate selectively or more strongly for a particular function such as modalityspecific or category-specific semantics (such as objects/actions, abstract/concrete, animate/inanimate, animals, fruits/vegetables, colours, body parts, countries, flowers, etc.) @xcite @xcite . this indicates a function-specific division of lower-level semantic processing. singlespace distributional word models have been found to partially correlate to these distributed brain activity patterns @xcite @xcite , but fail to explain the full spectrum of fine-grained word associations humans are able to make. our work has been partly inspired by this literature. ## function-specific representation space our goal is to model the mutual associations (cooccurrences) between n groups of words, where each group represents a particular role, such as subject or object in an svo structure. we induce an embedding matrix r |v i |×d for every group @xmath1, . . . , n , where |v i | corresponds to the vocabulary size of the i-th group and the group vocabularies can partially overlap. for consistency, the vector dimensionality d is kept equal across all variables. multiple groups. without loss of generality we present a model which creates a function-specific vector space for @xmath2, referring to those groups as a, b, and c. note that the model is not limited to this setup, as we show later in section 6. a, b and c might be interrelated phenomena, and we aim for a model which can reliably score the plausibility of combining three vectors ( a, b, c) taken from this space. in addition to the full joint prediction, we aim for any two vector combinations ( a b, b c, c a) to have plausible scores of their own. observing relations between words inside single-group subspaces (a, b, or c) is another desirable feature. ## multidirectional synchronous representation learning the multidirectional neural representation learning model takes a list of n groups of words (g 1 , g 2 , . . . , g n ), factorises it into all possible "group-to-group" sub-models, and trains them jointly by combining objectives based on skipgram negative sampling @xcite . we learn a joint function-specific word vector space by using sub-networks that each consume one group g i on the input side and predict words from a second group g j on the output side, i, @xmath3, 2 . . . , n ; @xmath4. all sub-network losses are tied into a single joint loss and all groups g 1 , . . . , g n are shared between the sub-networks. sub-network architecture. we first factorise groups into sub-networks, representing all possible directions of prediction. two groups would lead to two sub-networks @xmath5; three groups lead to six sub-networks. similar to @xcite , we calculate the dot-product between two word vectors to quantify their association. for instance, the sub-network @xmath6: l @xmath7. we leave experiments with more sophisticated sub-networks for future work. ## evaluation preliminary task: pseudo-disambiguation. in the first evaluation, we adopt a standard pseudodisambiguation task from the selectional preference literature @xcite @xcite @xcite . for the three-group (s-v-o) case, the task is to score a true triplet (i.e., the (s-v-o) structure attested in the corpus) above all corrupted triplets (s-v'-o), (s'-v-o), (s-v-o'), where s', v' and o' denote subjects and objects randomly drawn from their respective vocabularies. similarly, for the two-group setting, the task is to express a higher preference towards the attested pairs (v-o) or (s-v) over corrupted pairs (v-o') or (s'-v). we report accuracy scores, i.e., we count all items where score(true) > score(corrupted). this simple pseudo-disambiguation task serves as a preliminary sanity check: it can be easily applied to a variety of training conditions with different variables. however, as pointed out by chambers and jurafsky (2010), the performance on this task is strongly influenced by a number of factors such as vocabulary size and the procedure for constructing corrupted examples. therefore, we additionally evaluate our models on a number of other established datasets @xcite . ## results and analysis pseudo-disambiguation. accuracy scores on the pseudo-disambiguation task in the 2/3/4-group setups are summarised in event similarity. we now test correlations of svo-based event representations composed from a 4 we also provide baseline scores taken from prior work, but the reader should be aware that the scores may not be directly comparable due to the dependence of this evaluation on factors such as vocabulary size and sampling of corrupted examples @xcite . function-specific vector space (see thematic-fit evaluation. correlation scores on two thematic-fit evaluation data sets are summarised in ## conclusion and future work we presented a novel multidirectional neural framework for learning function-specific word representations, which can be easily composed into multiword representations to reason over event similarity and thematic fit. we induced a joint vector space in which several groups of words (e.g., s, v, and o words forming the svo structures) are represented while taking into account the mutual associations between the groups. we found that resulting function-specific vectors yield state-of-the-art results on established benchmarks for the tasks of estimating event similarity and evaluating thematic fit, previously held by task-specific methods. in future work we will investigate more sophisticated neural (sub-)networks within the proposed framework. we will also apply the idea of functionspecific training to other interrelated linguistic phenomena and other languages, probe the usefulness of function-specific vectors in other language tasks, and explore how to integrate the methodology with sequential models. the pre-trained word vectors used in this work are available online at: https://github.com/cambridgeltl/fs-wrep .