QAVSA : Question Answering using Vector Symbolic Algebras (2024)

Abstract: With the advancement of large pretrained language models (PLMs), many question answering (QA) benchmarks have been developed in order to evaluate the reasoning capabilities of these models. Augmenting PLMs with external knowledge in the form of Knowledge Graphs (KGs) has been a popular method to improve their reasoning capabilities, and a common method to reason over KGs is to use Graph Neural Networks (GNNs). As an alternative to GNNs to augment PLMs, we propose a novel graph reasoning module using Vector Symbolic Algebra (VSA) graph representations and a k-layer MLP. We demonstrate that our VSA-based model performs as well as QA-GNN, a model combining a PLM and a GNN-module, on 3 multiple-choice question answering (MCQA) datasets. Our model has a simpler architecture than QA-GNN and also converges 39% faster during training.

## introduction models that perform question answering tasks require some amount of knowledge, whether it is structured or implicit, about the concepts to be reasoned over. modern large pretrained language models (plms), linguistic knowledge is implicit in the token embeddings that have been learned using a self-attention mechanism on large text corpuses to perform next-token prediction @xcite . with enough model parameters and training data, these plms have been successful in a wide range of question-answering and reasoning benchmarks. however, when analyzing deductive reasoning performance, i.e. the ability to learn and generalize from logic rules, smaller plms like bert and roberta demonstrate inconsistent performance @xcite . as a result, there have been many studies that work to integrate external, structured knowledge and reasoning modules with plms to perform more reliable logical reasoning. one type of knowledge structure that is commonly used are knowledge graphs (kg), due to their suitability for symbolic reasoning @xcite . graph neural networks (gnns) are deep learning networks that have gained popularity for reasoning over graphstructured data. consequently, methods that integrate kgs with plms often also use gnns @xcite . one recent approach to combining these techniques is captured by the qa-gnn @xcite , which is able to answer multiple-choice questions by scoring the plausibility of each question answer. rather than using gnns to model structured data, our model, qavsa, uses a lesser known method, vector symoblic algebras (vsas; also known as vector symbolic architectures), to represent and combine high-dimensional concept vectors in a structured way. the integration of plms and vsas has not been previously proposed, to our knowledge. as a result, the focus of this paper is to perform a comparison between our vsa-based method and the gnn-based method of qa-gnn for improving reasoning capabilities of plms in the context of multiple-choice question answering. specifcally, we compare the performance of qavsa and qa-gnn on commonsenseqa @xcite , openbookqa @xcite , and medqa-usmle @xcite , with the first two datasets being focused on commonsense reasoning, and the latter focusing on domainspecific medical questions. the main contributions of this paper are: 1) a novel combination of plm text encoders and vsas that performs as well as an analgous gnn-based method on three mcqa datasets while training faster; 2) a comparative evaluation of different vsas and plms used in the model; 3) a study on ablation and variations of the model architecture and graph embedding representations; and 4) a new vsa-specific method of analyzing model explainability. ## related work many of the top performing models on mcqa benchmarks involving reasoning are larger 10b or 100b+ parameter plms. high performance can come from fine-tuning on mcqa datasets as has been shown with unifiedqa @xcite and @xcite . high performance with these models has also been shown to be achievable with prompting techniques including few-shot prompting @xcite , chain-of-thought prompting with self-consistency @xcite , and ensemble refinement @xcite . more relevant to our research, several approaches have been proposed to integrate external knowledge graphs with plms in order to make use of more domain-specific structured knowledge. these include kear @xcite and dek-cor @xcite , which use a plm and knowledge retrieved from an external kg and wiktionary to train an attention mechanism on these external knowledge bases and plm representations to improve commonsense reasoning. similarly, kagnet (lin et al., 2019) performs commonsense reasoning by grounding the concepts within each question-answer (qa) pair of multiple-choice datasets to extract subgraphs from an external knowledge graph. kagnet then uses a combination of graph convolutional networks, lstm-based relational path encodings, and a pathbased attention mechanism to identify important reasoning paths to generate graph vector encodings for each qa pair to subsequently score them. adopting similar graph preprocessing to kagnet, mhgrn @xcite performs multi-hop, multi-relation reasoning by using multi-hop message passing from relational graph convolutional networks, structured relational attention, and node attention pooling to generate its graph representations and score qa pairs. another family of models that also use the same graph processing above stem from the qa-gnn @xcite model. qa-gnn fuses a plm representation of the qa context as a node into the qa subgraphs. using a graph attention network (gat), qa-gnn updates the subgraph concept embeddings, including the qa context node and edge weights. the initial plm qa context representation, along with the final graph rep-resentation of the qa context and graph concept attention pooling is used to score the qa pairs. this method is refined in greaselm @xcite in which the plm token and graph node modalities of the qa context are mixed over several layers to simultaneously update the plm and gnn concept embeddings. further refinements are made in dragon @xcite , where the cross-modal encoder from greaselm is pretrained in a self-supervised fashion to perform both masked language modeling and kg link prediction. our qavsa model uses similar pre-processing to qa-gnn but with a different representation of the graph. consequently, most of our direct comparisons are to qa-gnn. there exist other models that use external kgs but are not gnn-based, such as gsc @xcite and mvp-tuning @xcite . gsc uses a simple graph neural counter to reduce the node and embeddings to one dimension and performs gnn-like embedding updates on these single values. mvp-tuning makes use of semantically similar qa pairs in the training set to improve knowledge retrieval and tunes prompt tokens of the plm by using the qa context and retrieved kg triplets as input to the plm. ## vector symbolic algebras vector symbolic algebras (vsas) are defined by a set of three vector operations that are useful for building up structured vector representations. these include the similarity, bundling (or collecting), and binding operations. there are a wide variety of possible choices for these operations, but we consider only two sets of operators, those for holographic reduced representations @xcite and for vector-derived transformation bindings @xcite . a similarity operation is necessary to compare different vectors within the vsa space, and it is often computed with a normalized vector dot product, i.e., cosine similarity. for two vsa-encoded vectors x, @xmath0, this is defined as s(x, y) = <x,@xmath1. both hrrs and vtbs use this similarity operator. a bundling operation is used to represent a set of objects and is usually defined by element-wise addition. thus, the bundling of x, @xmath2, y) = x + y. in a vsa, this operation should result in a new vector that is similar to both x and y, as is the case with element-wise addition. both hrrs and vtbs use this bundling operator. a binding operation, is used to combine two symbols together in a single representation, which is often used to represent slot-filler pairs. in hrrs, circular convolution is used as a binding operator, defined by (x⃝ * y) i := d @xmath3, @xmath4, 2, ..., d}. a desired property of a binding operator is that the resulting vector from x ⃝ * y should be dissimilar to both x and y. also, an unbinding operation, or a pseudo-inverse to binding should exist. with circular convolution, this is done by binding the pseudo-inverse of the given operand: (x ⃝ * y) ⃝ * -1 y ≈ x ⃝ * y ⃝ * y + ≈ x. for hrrs the approximate inverse to y is y + := (y 1 , y d , y d-1 , ..., y 2 ) ⊤ @xcite . since circular convolution is commutative, there is no directional relation to two bound vectors in the hrr vsa. in contrast, vtb has a non-associative and non-commutative binding operation defined on vectors x, @xmath5. specifically, given the approximate inverse to y is y + , where the elements of y are permuted such that v y + = v ⊤ y . the vtb binding operation has only right inverses and identities, so there exists an alternative transposed vtb (tvtb) algebra, with two-sided inverses and identities with the following binding operation: as a simple example, a scene of a dog holding a stick could be represented with vsas in a slotfiller fashion as: @xmath6, given the slot vectors subject, verb, @xmath7, holds, @xmath8. one could then query the subject of the scene with unbinding: scene ⃝ * subject + ≈ dog + noise, which can be cleaned up to the exact dog vector by finding the vsa vector in the vocabulary with the highest similarity to the result. we use these techniques for representing structure to capture concept relations in a knowledge graph to propose a novel question answering neural network model. ## methods given a multiple-choice question q and an answer option a, as in qa-gnn @xcite , the purpose of the model is to score the plausibility of each (q, a) pair from the set of all answer options by performing joint reasoning using a (q, a) context, z, generated from a plm text encoder, and a working graph g w that contains relations and concepts pertaining to each specific (q, a) pair. in qa-gnn, the graph reasoning portion of the model consists of a specific type of gnn called a graph attention network (gat; @xcite ). as a replacement for the gat in qavsa, a single vsa vector representation of the (q, a) graph is generated. this representation is used as input to a simple k-layer mlp to realize the graph reasoning portion of qavsa. using this learned mlp along with the plm (q, a) context embeddings, the (q, a) pair is scored, as shown in figure 1 . the nodes of the working subgraph g w are selected by first linking the concepts from the question, v q , and from the answer, v a , to nodes in g, where v q ∪ v @xmath9,a ⊂ v . all of the nodes on a 2-hop path between the nodes in v q,a , i.e., all nodes in v related to two of the nodes in v q,a are also included in the working graph, producing v w . finally, v w is pruned down to 200 nodes by scoring the relevance of each node to the (q, a) pair using a plm as described in @xcite . all of the edges connecting each pair of nodes in v w , defined as e w ⊂ e, are included in the final working graph, g @xmath10, e w ). following yasunaga et al. ( 2021 ), the initial 1024-dimensional embeddings of these nodes are defined by feeding each triple composed of a head, relation and tail entity, (h, r, t) ∈ v w × r × v w , as a sentence into a plm text encoder. the representations for each concept are pooled using the corresponding portion of each triple that they appear in. computing relation embeddings r in yasunaga et al. (2021) in this way was unnecessary, as they represent the relation type as a one-hot vector for their gnn module. however, in our approach, each relation is a distributed representation, r, so we use an initial vector embedding computed in the same way as each graph concept embedding is computed. feng et al. ( 2020 ) provide embeddings computed for each concept in the graphs used. however, these embeddings do not include relations. as a result, we computed all embeddings following the process defined in @xcite . ## experiments in order to perform a comparison to qa-gnn, we evaluate qavsa on the same three datasets that were used to evaluate qa-gnn in @xcite . however, we hyperparameter tune our model rather than keeping parameters the same as those used in qa-gnn to maximize accuracy on the benchmark development splits. openbookqa (obqa) @xcite ) is a 4-way multiple choice dataset aiming to assess human understanding of a subject in an openbook setting. the dataset consists of a list of 1326 science facts along with 5957 elementary school science questions with a train/dev/test split of 4957/500/500. medqa-usmle (medqa) @xcite ) is a 4-way multiple choice dataset based on questions from the united states medical license exams (usmle). the english version of the dataset has 12,723 questions, with a train/dev/test split of 10178/1272/1273. for csqa and obqa, the external kg used is conceptnet @xcite . conceptnet consists of 799,273 common words or phrases connected by edges of 17 different merged relation types, after preprocessing. the method to initialize the concept and relation embeddings for concept-net are also described in section 4.1, and uses plms bert-large or roberta-large. following @xcite , roberta-large is he plm used to encode the qa contexts in qavsa for csqa, and aristoroberta @xcite is used for the qa contexts for obqa. the external kg used for medqa is the unified medical language system ( umls; bodenreider (2004 ## discussion on the csqa and obqa datasets, there is no definitive top performer between qa-gnn and qavsa due to the fact that both models have the best performance on one of the dataset splits. the accuracy of qa-gnn is slightly lower than plm baseline for the csqa inhouse dev. split and the obqa test split, and this is most likely due to a combination of larger variance of model accuracy between seeds along with the fact that the model learning rate was not tuned for our recomputed concept embeddings. for medqa however, qavsa slightly outperforms qa-gnn on both the dev. and test splits. medqa is a significantly harder than csqa and obqa due to the nature of the in-depth medical questions asked. this is evidenced by the difference in accuracy of more than 30% compared to csqa and obqa. this more consistent performance increase may suggest that the increased semantic complexity of the concepts and relations in umls benefit more from the structured vsa representations generated and reasoned over in qavsa rather than using multi-relational gnns to update concept embeddings. although qavsa does not consistently outperform qa-gnn, the architecture is much simpler than the graph attention network in qa-gnn as it feeds vsa vectors into standard mlp layers. there is no requirement to use node embeddings matrices during computation along with linear and nonlinear transformations on node and relation type embeddings to perform message passing between concepts. also, there is no requirement to use graph attention layers to create attention weights on the relations between concepts. in qavsa, the attention on the relational edges within the graph arises naturally and can be analyzed as shown in section 6.3. additionally, the qavsa memory requirements for its graph representation is constant at the dimensionality of the vsa vector, d. this compares favorable to gnns that require an n × d node embedding matrix and an n × n adjacency matrix. for these benchmarks, qa-gnn requires graphs with exactly 200 nodes for each qa pair. if one wanted to scale up the number of nodes in the graph significantly, the memory resources required would grow quadratically. in contrast, with an increase in the number of graph nodes and edges, the memory requirements for qavsa are constant. ## conclusion we presented qavsa, a new type of model that leverages vsa-represented knowledge graphs along with general linguistic knowledge from plms to perform reasoning on mcqa benchmarks. through a direct comparison to the gnn-based model qa-gnn, we exhibit the ability of qavsa to perform similarly to qa-gnn on three datasets, while using a simpler k-layer mlp reasoning module. we also demonstrate faster convergence during training than qa-gnn and highlight the explainability of our model outputs through our vsa graph representations. for future study, our method of representing knowledge graphs with vsas could be useful in a wide variety of knowledge graph qa tasks involving information retrieval, like multi-hop reasoning @xcite . there are many other ways that kgs are integrated into llms, such as using them to augment llm input or using them as training objectives during llm pretraining, so having an efficient vsa representation of these kgs may be beneficial to these methods @xcite . gnns are also widespread for tasks outside of natural language processing, such object detection, chemical reaction prediction, and disease classification, and it is worthwhile to determine if our vsa-based approach is useful for representing structures that are not linguistic @xcite .