Representation Learning for Type-Driven Composition (2020)

Abstract: This paper is about learning word representations using grammatical type information. We use the syntactic types of Combinatory Categorial Grammar to develop multilinear representations, i.e. maps with n arguments, for words with different functional types. The multilinear maps of words compose with each other to form sentence representations. We extend the skipgram algorithm from vectors to multi- linear maps to learn these representations and instantiate it on unary and binary maps for transitive verbs. These are evaluated on verb and sentence similarity and disambiguation tasks and a subset of the SICK relatedness dataset. Our model performs better than previous type- driven models and is competitive with state of the art representation learning methods such as BERT and neural sentence encoders.

## introduction we develop a novel technique for learning word representations by using syntactic type information of the words to learn representations for them and the constituency-based structure of the sentence to compose the representations. the word representations are multilinear maps, i.e. maps with variable number of arguments, where the number of arguments and the type of each map come from the syntactic type of each word. the word representation are composed via the application and further composition of the results of these maps, based on constituency structure. for instance, a noun such as children or ball is represented by a vector, i.e. ## multilinear skipgram embeddings the skipgram model with negative sampling generates word embeddings by optimising a logistic regression objective in which target vectors have high inner product with context vectors for positive contexts, and low inner product with negative ones. given a target word n and a set of positive contexts c, a set of negative contexts c is sampled from a unigram distribution raised to some power (here: 3/4, after @xcite ). initially, both target vectors n and context vectors c are randomly intialised, and during training the model updates both target and context vectors to maximise the following objective function: we generalise the skipgram model following the typing of combinatorial categorial grammar @xcite ). ccg has a transparent interface between syntax and semantics and robust wide-coverage parsers @xcite . syntactic types of ccg are either atomic, e.g. nouns/noun phrases: np and sentences: s , or functional. functional types are either of the form y /x or y \x ; they take an argument of type x and return an argument of type y , where for \ the argument occurs to the left and for / it occurs to the right. examples of functional types are adjectives: np /np , intransitive verbs: s \np and transitive verbs: (s \np )/np . types are composed with each other through the combinatorial rules of ccg, which include forward and backward application and composition, type-raising, and backward-cross and forwardcross composition. an example of forward application is when an adjective composes with a noun, producing a noun. an example of backward application is when a verb phrase composes with a noun phrase producing a sentence. forward and backward composition are used in composing auxiliary phrases; cross composition and the type-raising combinators are used in cases of coordination and gapping. following the tensor semantics of ccg, developed in @xcite , in our model, we represent a word w with a functional type of n arguments by a n-ary map w from the argument spaces to the result space: given a functional word w of n arguments and representations d 1 , ..., d n of its arguments, we denote by w (n) d 1 ...d n the application of the representation of w to its arguments' representations. the model that learns the maps has the following objective function: when w is a noun, w (0) is a 0-ary map, equivalent to w i 1 : a 1st-order tensor, i.e., a vector. in this case, the objective function reduces to the original skipgram model of equation 1. for w an adjective, w (1) is a unary map, equivalent to w i 1 i 2 : a 2-nd order tensor, i.e., a matrix. the objective function that learns it was developed in @xcite , and is as follows: since adjective-noun combinations are themselves nouns, taking the original skipgram linear context window will produce sensible adjective representations. this is however not the case for all words with functional types: for verbs, for example, subjects and objects may not be directly adjacent to the verb in a sentence and so one needs to commit to a full sentential context, leading to uninformative training data. aside to that, training of a cube leads to over parameterisation. to overcome these issues simultaneously, we define lower order approximations of higher order tensors, where one argument of the functional type is left out of the composition and used as context. more formally, a word w with a functional type of n arguments, is approximated by n maps of n -1 arguments. equivalently, in tensor form, we are approximating a full n+1th-order tensor, by decomposing it into n separate partial tensors of one lower order each. we denote the map equivalent of these partial tensors by w i (n-1) . the objective function of the model thus become as follows: here, d i is a word with representation d i : an observed argument of w , and d i is the set of all such arguments. whereas, d i is a word with representation d i , which can in principle serve as an argument of w , but it is randomly sampled, so it is an unobserved argument of w . similarly, d i is the set of all such arguments. we can decrease the order of the tensors even more by parameterising over subsets of contexts. for a word w of n arguments, when including 1 to @xmath0, we obtain an (n -i)th-order tensor, with the equivalent map w 1...i|i+1...n (n-i+1) . the application of this map to the remaining i + 1 to n arguments is w 1...i|i+1...n (n-i+1) d i+1 ...d n . so we predict as context the 1...ith arguments by composing with the vectors for the i + 1...nth arguments. we write w 1...i (n-i+1) when @xmath1, i.e. we use all arguments as context. ## conclusion we generalised the skipgram model @xcite to learn multilinear map representations for words with functional types using the setting of combinatory categorial grammar. our model reduces to the original skipgram for atomic types such as nouns, and to the adjective skipgram model of @xcite , for functional types of one argument. to overcome potential sparsity issues we approximated higher arity maps with a set of lower arity ones and showed that such approximations provide better results. the model was implemented on transitive verbs, learning binary and a set of approximated unary representations. these were evaluated on verb similarity and disambiguation and sentence similarity tasks. the unary map approximations significantly outperformed previous type-driven verb representations. they also outperformed sentence encoders and pre-trained bert embeddings. when moving to datasets of longer sentences, e.g. sentences with elliptical phrases and the sick relatedness, some sentence encoders and fine-tuned bert representations were superior. our multilinear skipgram model paves the way for a new generation of type-driven representations, in line with recent research highlighting benefits of syntactic biases injected into representation learning @xcite . furthermore, our model is fast to train, guided by a linguistic calculus (ccg), and produces syntax-aware sentence embeddings. performance could potentially be improved by adding non-linearities to the model, as in @xcite and by modelling complex syntactic phenomena such as auxiliaries and negation.