High-dimensional vector spaces can accommodate constructional features quite conveniently (2023)

Abstract: Current language processing tools presuppose input in the form of a sequence of high-dimensional vectors with continuous values. Lexical items can be converted to such vectors with standard methodology and subsequent processing is assumed to handle structural features of the string. Constructional features do typically not fit in that processing pipeline: they are not as clearly sequential, they overlap with other items, and the fact that they are combinations of lexical items obscures their ontological status as observable linguistic items in their own right. Constructional grammar frameworks allow for a more general view on how to understand lexical items and their configurations in a common framework. This paper introduces an approach to accommodate that understanding in a vector symbolic architecture, a processing framework which allows for combinations of continuous vectors and discrete items, convenient for various downstream processing using e.g. neural processing or other tools which expect input in vector form.

## one can have both for a linguist, the entire approach causes some frustration. one view of a linguist's job description is to provide the appropriate features for a learning system to pay attention to, and to strive to optimise the convergence of features, categories, and dimensions of variation for a natural language processing system to deliver the best results, the steepest learning curves, and the smallest set of parameters for some set of tasks. the current generation of natural language processing tools do not invite such intervention or such hypothesis testing: indeed, the practice of feature engineering is somewhat frowned upon. in more general terms, a continuous geometric model has intuitively appealing qualities (even to the extent that the metaphors such a model invites coupled with our understanding about geometry in a physical world can lead our intuitions astray @xcite ): where a symbolic model allows for greater transparency, greater explanatory power, convenient inspectability and editability, and a more direct path to hypothesis testing, a continuous geometric model offers robustness, coverage, and generalisability. and today, by representing linguistic observations as vectors we will have a convenient interface to downstream computation using various past, current, and most likely many future flavours of machine learning. there is no inherent contradiction between continuous models and discrete elements of study. words are discrete observations and are routinely represented as vectors in language processing tasks, typically encoded by a shallow neural net which takes local context into account in narrow windows to model syntactic dependencies or slightly larger windows to model topical association. configurational features such as elements from construction grammars can be represented similarly. ## the general idea of high-dimensional computation high-dimensional computation allows for the incorporation of linguistic items, single lexical items, configurational elements, or constituent structures jointly, using simple operations. this framework was first introduced by plate under the name holographic reduced representation (hrr; @xcite and further developed as multiply-add-permute (map: gayler, 1998), vector symbolic architecture (vsa @xcite , and hyperdimensional computing @xcite . the idea is to encode information in a vector with three simple linear algeabric operations that keep vector dimensionality constant: vector addition, vector multiplication, and permutation. addition of two vectors yields a new vector similar to its operand vectors; addition can be used to represent a set. multiplication, coordinate by coordinate, yields a product vector which is dissimilar to its operand vectors. permutation takes a single vector, rearranges its coordinates, and produces a vector that is dissimilar to the operand. the operations are invertible: the operations can be undone and the component vectors retrieved from the result. these operations are based on a well-understood computational algebra, similarly to how most vector models rely on geometry. a vector space together with linear algebraic manipulation operations and geometric access and analysis operations can be used to combine observations into a common vector represtentation systematically, transparently, and explicitly, and for our purposes allows us a convenient way to evaluate the information value of features we expect to be important to understand the linguistic signal. random indexing is a high-dimensional computation framework, which assigns randomly generated fixed-dimensional index vectors to observations of interest, to be combined using the above operations. random indexing traces its roots to kanerva's sparse distributed memory framework @xcite . a randomly generated index vector is defined to be sparse, i.e. to mostly contain 0s with a small number of 1s and -1s-say 10 non-zero elements in a 1000-dimensional vectorand the characteristics of high-dimensional spaces are such that two such randomly generated vectors will be very close to orthogonal. we assign random index vectors to each linguistic item of interest: single words and constructions alike. if a new previously unencountered item shows up during processing, it can be assigned a new unique index vector without retooling the previously known space of items. in random indexing of linguistic material, addition is used to combine observations that are collocated into a joint vector: an utterance can be represented as the sum of index vectors for every word in it. permutation can be used to distinguish item occurrences in different roles or different surface forms, to distinguish cases, semantic roles, or head-attribute relations, e.g. this framework will allow us to represent sequences and configurations together with their constituent elements conveniently. ## some examples this section is based on some previously published example implementations and experiments @xcite . previous work using this approach was used to build general associative lexical resources @xcite using permutations to differentiate between left hand and right hand context @xcite . this model is a parsimonious method to aggregate cooccurrence statistics and has proven quite useful in practical application, but has today been superseded by large language models that are able to capture longer range dependencies. this purely lexical model can be enhanced by adding constructional elements, inflectional information, semantic roles, and even contextual extralinguistic information. sentences such as the ones given in example (1) both involve fish. this is of course for some purposes a notable observation, but we can add the observation that the fish in question participate in different roles in the sentences. this can be encoded by adding a semantic role label to the fish vector, adding a tense and aspect annotation to the main verb or even to all referential expressions to indicate that fish in sentence (1-b) are agents doing their thing in present progressive in contrast with the fish in sentence (1-a). what features to represent are up to the hypotheses being considered, and adding spurious features will not confuse the system except adding a slight noise to the resulting vector. the features of interest can be retrieved separately, since the operations used are invertible. the representation of f ish agent can be derived from the representation of f ish by invoking a permutation specific for agent roles. in a series of experiments on a commercial data set of customer reviews we tested the effect of adding amplifiers, negations, and constructional markers for attitudinal expression in addition to lexical features. we represented each sentence in the collection as a vector into which we added the index vector for each term present in it, weighted by inverse frequency. we also added a separate amplif ier vector if an amplifier was present; a negation vector if the main verb of the sentence was negated; a number of verbal class vectors cogitation, expression, privatesensation, and some others; separate vectors for each observed tense form in the sentence; vectors for presence of a personal pronoun; vectors for a number of attitudinal classes; and a series of constructional vectors for presence of subclause, presence of auxiliaries, and presence of adverbial constructions. these vectors form a lexical-semantic-constructional space with all features represented jointly. to demonstrate how this space can be queried for features, we represented the sentence (3-a) separately with only lexical features and with only semantic and constructional features. we then retrieved the most similar sentence from the joint vector space: the lexical vector retrieves sentence (3-b); the semantic and constructional vector retrieves the more personally expressive utterance with negative sentiment in sentence (3-c). the original objective of this experimentation was to find attitudinal expressions for certain types of product: here it is useful to show how a large number of features can be used to build a space and then that space can be queried with attention paid to subsets of features. in a continued set of experiments on attitude analysis we experimented with constructional features and their distribution in a dataset of some one million microblog posts that mention among other things corporate entities. we represented each microblog post as a sum vector of index vectors for individual lexical items; for unique triples of part-ofspeech tags-each triple such as dt -jj -n n or v bd -rb -rb having been assigned its own index vector; for observed tense and aspect for the main verb; for observations of the presence of modal auxiliaries and various adverbs; for semantic role labels for the agent of each clause; for the presence of several categories of amplifiers; and for some extracted configurations for verbs of utterance and cogitation, e.g. utteranceverbthat and other frequent constructions. these features were extracted using the nltk toolkit @xcite and on lexical resources coded using a comprehensive lexically oriented grammatical description of english @xcite . similarly to the example (3) above, we found, as shown in example (4) that the resulting representa-tion if only tested for lexical content indicated that the most similar utterance to sentence (4-a) would be sentence (4-b) while incorporating the various constructional features sentence (4-c) was found to be the best match. this we found to be useful for a project on tracking corporate sentiment online @xcite . in this case, the agency and animateness of the corporation in question is the significant feature linking the first and the third utterance. such resulting vectors are similar to pre-trained off-the-shelf word embeddings such as are routinely used as input to downstream machine learning models and indeed constructional features can be combined with such word embeddings through the application of addition operations. in general, the current generation of natural language processing tools are agnostic to the content of their input and are able to accommodate even weak signals found in the input. this suggests that a useful validation path to test hypotheses of the effectiveness and usefulness of constructional features is not only to study the end results on benchmark tasks, but the path to get to those results, and to investigate how the model takes the potentially enriched information into account in its training. ## take home this short paper attempts to convince its readers that it is possible to combine lexical and configurational features in a joint vector space model; that combining configurational or constructional features and lexical features in a joint vector space model is a useful and desirable path to investigate the validity of constructional hypotheses; that radical construction grammars provide a theoretical back end to such representations; and that hyperdimensional computing or vector symbolic architectures provide a well-established computational framework for processing such discrete information into a form which can be ingested by today's most popular processing tools. the details of our implementation are not important for this argument but random indexing is a convenient and lightweight approach to combining heterogenous information into a geometric representation.