GNN IS A C OUNTER ? R EVISITING GNN FOR Q UESTION A NSWERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Question Answering (QA) has been a long-standing research topic in AI and NLP ﬁelds, and a wealth of studies have been conducted to attempt to equip QA systems with human-level reasoning capability.
To approximate the complicated human reasoning process, state-of-the-art QA systems commonly use pre-trained language models (LMs) to access knowledge encoded in LMs together with elaborately designed modules based on Graph Neural Networks (GNNs) to perform reasoning over knowledge graphs (KGs).
However, many problems remain open regarding the reasoning functionality of these GNN-based modules.
Can these GNN-based modules really perform a complex reasoning process?
Are they under- or overcomplicated for QA?
To open the black box of GNN and investigate these problems, we dissect state-of-the-art GNN modules for QA and analyze their reasoning capability.
We discover that even a very simple graph neural counter can outperform all the existing GNN modules on CommonsenseQA and OpenBookQA , two popular
QA benchmark datasets which heavily rely on knowledge-aware reasoning.
Our work reveals that existing knowledge-aware GNN modules may only carry out some simple reasoning such as counting.
It remains a challenging open problem to build comprehensive reasoning modules for knowledge-powered QA.
I NTRODUCTION
Accessing and reasoning over relevant knowledge is the key to Question Answering (QA).
Such knowledge can be implicitly encoded or explicitly stored in structured knowledge graphs (KGs).
Large pre-trained language models (Devlin et al., 2018; Radford et al., 2018; 2019; Brown et al., 2020) are found to be effective in learning broad and rich implicit knowledge (Petroni et al., 2019; Bosselut et al., 2019; Talmor et al., 2020) and thus demonstrate much success for QA tasks.
Nevertheless, pretrained LMs struggle a lot with structured reasoning such as handling negation (Ribeiro et al., 2020;
Yasunaga et al., 2021).
In contrast, the explicit knowledge such as knowledge graphs (KGs) (Speer et al., 2017; Bollacker et al., 2008) works better for structured reasoning as it explicitly maintains speciﬁc information and relations and often produces interpretable results such as reasoning chains (Jhamtani & Clark, 2020; Khot et al., 2020; Clark et al., 2020).
To utilize both implicit and explicit knowledge for QA, many existing works combine large pre-trained
LMs with Graph Neural Networks (GNNs; Scarselli et al.
; Kipf & Welling (2016); Veliˇckovi´c et al.
(2017)), which are shown to achieve prominent QA performance.
These approaches commonly follow a two-step paradigm to process KGs: 1) schema graph grounding and 2) graph modeling for inference .
In Step 1, a schema graph is a retrieved sub-graph of KG related to the QA context and grounded on concepts; such sub-graphs include nodes with concept text, edges with relation types, and their adjacency matrix.
In Step 2, graph modeling is carried out via an elaborately designed graph-based neural module.
For instance, Lin et al.
(2019) uses GCN-LSTM-HPA which combines graph convolutional networks (Kipf & Welling, 2016) and LSTM (Hochreiter & Schmidhuber, 1997) with a hierarchical path-based attention mechanism for path-based relational graph representation.
Feng et al.
(2020) extends the single-hop message passing of RGCN (Schlichtkrull et al., 2018) as multi-hop message passing with structured relational attention to obtain the path-level reasoning ability and intractability, while keeping the good scalability of GNN.
Yasunaga et al.
(2021) uses a
[Figure]
Blue read material outside of his comfort zone because he wanted to gain what?
A. new perspective
C. entertained B. understanding
D. hunger
E. tired eyes
Interpretable Reasoning
https://realpython.com/python-counter/
Counting
Edges
Where could a personal ficus live?
[Figure]
LM to encode QA context as a node in the scheme graph and then utilized graph attention networks (Veliˇckovi´c et al., 2017) to process the joint graph.
Given that today’s QA systems have become more and more complicated, we would like to revisit those systems and ask several basic questions: Are those GNN-based modules under- or overcomplicated for QA?
What is the essential role they play in reasoning over knowledge?
To answer these questions, we ﬁrst analyze current state-of-the-art GNN modules for QA and their reasoning capability.
Building upon our analysis, we then design a simple yet effective graph-based neural counter that achieves even better QA performance on CommonsenseQA and OpenBookQA , two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning.
In the analysis part, we employ Sparse Variational Dropout (SparseVD; Molchanov et al.
as a tool to dissect existing graph network architectures.
SparseVD is proposed as a neural model pruning method in the literature, and its effect of model compressing serves as an indicator to ﬁgure out which part of the model can be pruned out without loss of accuracy.
We apply SparseVD to the inner layers of GNN modules, using their sparse ratio to analyze each layer’s contribution to the reasoning process.
Surprisingly, we ﬁnd that those GNN modules are over-parameterized: some layers in GNN can be pruned to a very low sparse ratio, and the initial node embeddings are dispensable.
Based on our observations, we design Graph Soft Counter (GSC), a very simple graph neural model which basically serves as a counter over the knowledge graph.
The hidden dimension of GSC layers is only 1, thus each edge/node only has a single number as the hidden embedding for graph-based aggregation.
As illustrated in Figure 1, GSC is not only very efﬁcient but also interpretable, since the aggregation of 1-dimensional embedding can be viewed as soft counting of edge/node in graphs.
Although GSC is designed to be a simplistic model, which has less than 1% trainable parameters compared to existing GNN modules for QA, it outperforms state-of-the-art GNN counterparts on two popular QA benchmark datasets.
Our work reveals that the existing complex GNN modules may just perform some simple reasoning such as counting in knowledge-aware reasoning.
The key contributions of our work are summarized as follows:
• Analysis of existing GNN modules: We employ SparseVD as a diagnostic tool to analyze the importance of various parts of state-of-the-art knowledge-aware GNN modules.
We ﬁnd that those GNN modules are over-complicated for what they can accomplish in the QA reasoning process.
• Importance of edge counting: We demonstrate that the counting of edges in the graph plays a crucial role in knowledge-aware reasoning, since our experiments show that even a simple hard counting model can achieve QA performance comparable to state-of-the-art GNN-based methods.
• Design of GSC module: We propose Graph Soft Counter (GSC), a simple yet effective neural module as the replacement for existing complex GNN modules.
With less than 1% trainable parameters compared to existing GNN modules for QA, our method even outperforms those complex GNN modules on two benchmark QA datasets.
[Figure]
Multi
[Figure]
relational Edge
Which ingredients are crucial?
[Figure]
Encode as One-hot Triples
A NALYSIS 2. P RELIMINARIES
The knowledge required by QA systems typically comes from two sources: implicit knowledge in pre-trained language models and explicit knowledge in knowledge graphs.
To use the implicit knowledge, existing works commonly use LMs as the encoder to encode textual input sequence x to contextualized word embeddings, and then pool the embedding of the start token (e.g., [ CLS ] for BERT) as the sentence embedding.
In addition, a MLP (we use a one layer fully connected layer) is used to map the sentence embedding to the score for the choice.
To process the explicit knowledge in knowledge graphs, existing works commonly follow a two-step paradigm: schema graph grounding and graph modeling for inference .
The schema graph is a retrieved sub-graph of KG grounding on concepts related to the QA context.
We deﬁne the sub-graph as a multi-relational graph G =
( V , E ) , where V is the set of entity nodes (concepts) in the KG and
E ⊆
V × R × V is the set of triplet edges that connect nodes in V with relation types in R .
Following prior works (Lin et al., 2019; Yasunaga et al., 2021), we link the entities mentioned in the question q and answer choice a ∈ C to the given sub-graph G .
A wealth of existing works have explored to incorporate pre-trained language models with knowledgeaware graph neural modules.
These methods usually have complex architecture design to process complex input knowledge.
As shown in Figure 2, the processed retrieved sub-graph can be summarized as four main components:
Node embeddings.
Many existing works on GNN-based QA employ external embeddings to initialize the node embeddings in the graph.
For example, Lin et al.
(2019) employs TransE (Wang et al., 2014) with GloVe embeddings (Pennington et al., 2014) as the initial node embeddings.
Feng et al.
(2020) and Yasunaga et al.
(2021) use mean-pooled BERT embeddings of entities across the graph as the initialization of node embeddings.
Relevance score.
Yasunaga et al.
(2021) uses an extra embedding of relevance score in the input node features to estimate the importance of KG nodes relative to the given QA context.
The relevance is computed
Adjacency matrix.
The original adjacency matrix of the sub-graph is asymmetric.
Typically, the adjacency matrix is converted to be symmetric before feeding into the GNN module: each KG relation is double recorded with opposite directions as symmetric edges in the processed adjacency matrix.
Edge embeddings.
Existing works use various ways to obtain the edge embeddings.
Yasunaga et al.
(2021) uses concatenated one-hot vector [ u s , e st , u t ] to encode edge triplets, where u s , u t indicates the node type and e st indicates the edge type.
Lin et al.
(2019) uses TransE (Wang et al., 2014) initialized with GloVe embeddings to generate edge embeddings.
Feng et al.
(2020) designs product-based edge embeddings to approximate the multi-hop relations in the graph.
2. D ISSECTION
To investigate the mechanism of these complex systems and how they use the complex information, we introduce a neural model pruning method named Sparse Variational Dropout (SparseVD)
(Molchanov et al., 2017) as a diagnostic tool to automatically dissect the graph network architecture.
Note that our dissection tool is pruning method agnostic; other pruning schemes (Han et al., 2015; He et al., 2017; Liu et al., 2017) may also be applicable.
Here we choose SparseVD since it prunes not only the weights with smaller scale but also the weights with higher variance, and it is theoretically supported by stochastic variational inference (Hoffman et al., 2013; Kingma & Welling, 2013; Rezende et al., 2014; Kingma et al., 2015).
Despite the fact that SparseVD was originally proposed in the ﬁeld of model compression (Han et al., 2016; He et al., 2018; Lin et al., 2017; Zhou et al., 2018; Wang et al., 2018), we use it to investigate which parts of GNN can be pruned out (sparse ratio to zero) without loss of accuracy, which indicates that part of the model is redundant.
To be speciﬁc, we keep the target model architecture unchanged, and parameterize each part of the weights in the model as a Gaussian distribution.
Afterwards, this probabilistic model will be trained with a cross-entropy loss jointly with a KL-divergence regularization term.
So the joint loss constrains the weights to our pruning prior.
We implement the SparseVD with the default threshold as in Molchanov et al.
Eventually, we get the pruned model with different sparsiﬁed ratios for the layers.
As shown in Table 1, we investigate three representative GNN-based QA methods, and the sparsiﬁed models could achieve the accuracy of their original counterparts.
This indicates that our dissection regularization does not hurt the model performance, so we then dive into each layer to see which parts play the most important role.
[Table]
Edge Embedding + SoftMax
Graph Attention Layer
Node Embedding
+
Node Value +
[Equation]
MLP + aggregate × sparsely dimension =
Adjacency Matrix update edge value update node value 0.2 0.
add in-node value to edge 0.
0.7 output node 0.
aggregate
2. O BSERVATIONS AND H YPOTHESIS
In Figure 3, we plot the sparse ratio of some representative layers during the SparseVD training of
GNN reasoning modules.
Due to the limited space, refer to Appendix A.3 for the full plots of curves.
According to thees plots, we summarize our key observations as follows: 1) the left plot shows that the edge encoder that encodes edge triplets (edge/node type information) preserves a relatively higher sparse ratio, while the node score embedding layer can be fully pruned; 2) the middle plot shows the layers inside GNN, and all the sparse ratios are low while the value layer has a relatively higher sparse ratio than key/query layers; 3) the right plot shows the concept embedding layers of three representative GNN methods, which process the initial node embeddings, can be completely discarded.
Inspired by these ﬁndings, we come up with the following design guidelines for a much simpler yet effective graph neural module for knowledge-aware reasoning:
Node embeddings & relevance score.
We observe that both the initial node (concept) embeddings and the relevance score embeddings are dispensable in our experiments.
Therefore, we may directly remove the initial node embedding layer and the relevance score embedding layer.
Edge embeddings.
The edge encoder layers are hard to prune which indicates that edge/node type information are essential to reasoning, so we further leverage it to get a better representation.
[Table]
Message passing layers.
The linear layers inside GNN layers (query, key, value, etc) can be pruned to a very low sparse ratio, suggesting that GNN may be overparameterized in these systems, and we can use fewer parameters for these layers.
Graph pooler.
The ﬁnal attention-based graph pooler aggregates the node representation over the graph to get a graph representation.
We observe that the key/query layers inside the pooler can be pruned out; as a result, the graph pooler can be reduced to a linear transformation.
G RAPH S OFT C OUNTER
Based on our ﬁndings in Section 2, we design the Graph Soft Counter (GSC), a simplistic graph neural module to process the graph information.
As demonstrated in Algorithm 1, there are only two basic components in GSC to compute the graph score: Edge encoder and Graph Soft Counter layers.
We can get the QA choice score by simply summing up the graph score and context score.
[Table]
[Table]
L |V| L ∆ L (cid:1) O (cid:0) |R| L |V| L ∆ L · D (cid:1)
L -hop MHGRN
O (cid:0) |R| 2 |V| L ∆ (cid:1)
O ( |R||V| L · D ) L -layer QAGNN
O ( |V| L ∆)
O ( |R||V| L · D ) L -layer GSC
O ( |V| L )
O ( |R||V| L ) edge encoder is a two-layer MLP with dimension [46 × 32 × 1] followed by a Sigmoid function to encode the edge triplets to a ﬂoat number in the range of (0 , 1) .
triplets is represented as a concatenated one-hot vector [ u s , e st , u t ] , where u s , u t indicates 4 node types and e st indicates 38 relation types (17 regular relations plus question/answer entity relations and their reversions).
Graph Soft Counter layers are parameter-free and only keep the basic graph operations: Propagation and aggregation.
To overcome overparameterization, one straightforward way is to reduce the hidden size, and the extreme case is reducing it to only 1.
As shown in Figure 4, the GSC layer simply propagates and aggregates the numbers on the edges and node following the two-step scheme: 1) update the edge value with in-node value, and this is done by simply indexing and adding; 2) update the node value by aggregating the edge values, and this is done by scattering edge values to out-node.
Since we reduce the hidden size to only 1, there are basically numbers propagating and aggregating on the graph.
We can interpret these numbers as soft counts representing the importance of the edges and nodes.
The aggregation of GSC can be regarded as accumulating the soft counts, so call it Graph Soft Counter.
In addition, we can also formulate GSC as an extremely efﬁcient variant of current mainstream GNN such as GAT and GCN.
As shown in Table 3, the computation complexity of GSC is much smaller than the baselines regard to both time and space.
Table 2 showing the trainable parameters in GSC is remarkably less than previous methods since our message-passing layers are parameter-free.
The model size of GSC is also extremely small since we do not use any initial node embedding.
To draw a more conclusive conclusion, we handcraft a counting-based feature and use a simple two-layer MLP to map it to graph score.
The feature is computed by counting the occurrence of each possible edge triplet.
Surprisingly, this simple model could achieve similar performance and can even outperform baselines, which further proves that Counter is a basic and crucial part in QA.
E XPERIMENTS 4. S ETTINGS
Datasets and KG.
We conduct extensive experiments on CommonsenseQA (Talmor et al., 2018) and OpenBookQA (Mihaylov et al., 2018), two popular QA benchmark datasets that heavily rely on knowledge-aware reasoning capability.
CommonsenseQA is a 5-way multiple choice QA task that requires reasoning with commonsense knowledge, which contains 12,102 questions.
The test set of
CommonsenseQA is not publicly available, and the model predictions can only be evaluated once every two weeks via the ofﬁcial leaderboard.
Hence, following the data split of Lin et al.
(2019), we experiment and report the accuracy on the in-house dev (IHdev) and test (IHtest) splits.
We also report the accuracy of our ﬁnal system on the ofﬁcial test set.
OpenBookQA is a 4-way multiple choice QA task that requires reasoning with elementary science knowledge, which contains 5,957 questions.
Following Clark et al.
(2019), methods with AristoRoBERTa use textual evidence as an external input of the QA context.
In addition, we use ConceptNet (Speer et al., 2017), a general commonsense knowledge graph, as our structured knowledge source G for both of the above tasks.
Given each QA context (question and answer choice), we retrieve the sub-graph G sub from G following Feng et al.
We only use the concepts that occur in the QA context, as detailed in Table 8.
[Table]
in-house split (controlled experiments).
As the ofﬁcial test is hidden, here we report the in-house dev (IHdev) and test (IHtest) accuracy, following the data split of
Lin et al.
Implementation and training details.
We use a two-layer MLP with hidden dimension 47 × 32 × 1 followed by a Sigmoid function as our edge encoder, and the number of GSC layers is 2.
Since GSC is extremely efﬁcient, we set the dropout (Srivastava et al., 2014) rate to 0.
We use
RAdam (Liu et al., 2020) as the optimizer and set the batch size to 128.
The learning rate is 1e-5 for RoBERTa and 1e-2 for GSC.
The maximum number of epoch is set to 30 for CommonsenseQA and 75 for OpenBookQA .
On a single Quadro RTX
GPU, each GSC training only takes about 2 hours to converge, while other methods often take 10+ hours.
[Table]
Baselines.
For GSC and all the other methods,
RoBERTa-large (Liu et al., 2019) is used for both CommonsenseQA and OpenBookQA , and
AristoRoBERTa (Clark et al., 2019) is used for
OpenBookQA as an additional setting for fair comparison.
We experiment to compare GSC with existing GNN-based QA methods, including RN (Santoro et al., 2017), GconAttn (Wang et al., 2019), RGCN (Schlichtkrull et al., 2018),
KagNet (Lin et al., 2019), MHGRN (Feng et al., 2020) and QA-GNN (Yasunaga et al., 2021), which only differ in the design of GNN reasoning modules.
We report the performance of baselines referring to Yasunaga et al.
(2021) and all the test results are evaluated on the best model on the dev split.
4. R ESULTS
[Table]
CommonsenseQA .
As shown in Table 4,
GSC outperforms the previous best model with 2 .
57% mean accuracy on In-house dev split and 1 .
07% mean accuracy on the inhouse test split.
We observe that the performance variance of GSC is smaller than the baselines, indicating that our GSC are both effective and stable.
On the ofﬁcial leaderboard of CommonsenseQA in Table 5,
GSC also outperforms all the GNN-based
QA systems.
Note that the previous top system UniﬁedQA (11B params) uses T (Raffel et al., 2020) as the pre-trained LM model, which is 30x larger than our model and uses much more pre-training data.
OpenBookQA .
From Table 6 our GSC outperforms the previous best model with 2 .
53% mean test accuracy with normal setting and 3 . 9% test accuracy with the AristoRoBERTa setting.
More remarkably, as shown in Table 7, our GSC ranks top one on the ofﬁcial leaderboard of OpenBookQA , which even surpasses the performance of UniﬁedQA (11B), which is 30x larger than our model.
GSC
GT
QA-GNN 11% 66% 12% 8% 7% 19% 15%
GT
MHGRN 12% 65% 13% 9% 8% 18% 14%
GT
RoBERTa 10% 63% 16% 11% 5% 21% 16%
GT
ALBERT 15% 58% 20% 16% 11% 15% 11% GSC
GSC
GSC GSC
GT
QA-GNN 11% 66% 12% 8% 7% 19% 15%
GT
MHGRN 12% 65% 14% 9% 7% 18% 14%
GT
RoBERTa 10% 63% 16% 11% 5% 21% 16%
GT
ALBERT 15% 58% 20% 16% 11% 15% 11% GSC
GSC
GSC GSC (
[Figure]
GT 9% 69% 5% 20% 17%
GSC (seed2) 6% 8%
GSC
GT
QA-GNN 11% 66% 12% 8% 7% 19% 15%
GT
MHGRN 12% 65% 14% 9% 7% 18% 14%
GT
RoBERTa 10% 63% 16% 11% 5% 21% 16%
GT
ALBERT 15% 58% 20% 16% 11% 15% 11% GSC
[Figure]
[Figure] [Figure]
[Figure] [Figure]
QA-GNN > MHGRN > RoBERTa.
This indicates that the reasoning ability learned by our GSC basically covers that of other GNNs.
4. D ISCUSSION
[Table]
As mentioned above, we observe that the initial node embeddings are dispensable.
Then we start to explore how the maximum number of retrieved nodes (also related to edges) affects the model.
We experiment various numbers of nodes for our GSC, and summarize the results in Table 8.
We ﬁnd that larger maximum number of node does not beneﬁt the model, and the model achieves the best performance when we use all and only the entity nodes directly occur in question and answer, whose number of nodes is generally less than 32.
This indicates that 1-hop retrieval is adequate for our methods, and this could be done super efﬁciently than multi-hop retrieval.
To further analyze the reasoning capacity of
GSC, we draw the Venn diagram for the predictions of different methods and ground truth (GT) in Figure 5.
We ﬁnd that even for the different runs of the same GSC model, the correct overlap is only 69%, showing that the datasets are relatively noisy and there exists decent variance in the prediction results.
We also observe that GSC has a larger overlap for GNN-based systems (e.g., QA-GNN and MHGRN), while at the same time having less overlap for non-GNN methods.
The ALBERT model has the least overlap since all the other methods use RoBERTa as the text encoder.
[Table]
(bottom), showing that 1) the hard counter achieves performance on par with GNN-based methods; 2) GSC works well even when only using entities occurred in
QA context, which typically contains less than 32 nodes.
We also observe that the order of the percentage of overlap of left four exactly matches the order of the performance of each model: GSC > QA-GNN > MHGRN > RoBERTa.
This indicates that GSC has quite similar behaviors versus other GNNbased systems, and the reasoning capability of GSC is on par with existing GNN counterparts.
This further reveals that counting plays an essential role in knowledge-aware reasoning for QA.
To verify our observations, we handcraft a hard edge counting feature feeding into a simple two-layer MLP.
As shown in Table 8, this hard counting model with 2-hop edge feature could achieve a comparable performance of GSC, which even outperforms other GNN baselines.
These impressive results not only show that our GSC is effective, but also prove that counting is an essential functionality in the process of the current knowledge-aware QA systems.
Personal ficus could [Figure] in the front yard.
Where could a personal ficus live?
A.
[Figure]
C. california B. cabin in the woods
D. conservatory
E. tropical forest
[Figure]
tropical
[Figure]
forest
[Figure]
ficus could
[Figure] in the tropical forest [Figure] As shown in Figure 6, for the retrieved sub-graph of each answer choice, we can directly observe the behavior of the GSC by printing out the output edge/node values (soft count) of each layer.
In this way, we can trace back to see why the model scores the answers like that.
We list the runtime soft counts of GSC’s edge encoder in Appendix A.5.
A higher soft count means that the edge/node is more important, and can contribute more to the ﬁnal graph score.
This demonstrates the advantage of
GSC as an interpretable reasoning module.
In addition, GSC only reasons over the edges in the graph so that the concept embedding is not used, which makes our model better generalize to graph with new entities.
R ELATED W ORK
KG-powered QA.
For KG-powered QA, traditional methods use semantic parsers to map the question to logical form (Berant et al., 2013; Yih et al., 2015) or executable program on KG (Liang et al., 2016), which typically require domain-speciﬁc rules and ﬁne-grained annotations, making them difﬁcult to handle the noise in questions and KG.
To address these challenges, various deep neural models have been applied to KG-powered QA, such as memory networks (Kumar et al., 2015;
Sukhbaatar et al., 2015; Miller et al., 2016), neural programmer for knowledge table QA (Neelakantan et al., 2016), neural knowledge-aware reader (Xiong et al., 2019; Sun et al., 2019), etc.
GNN for KG-powered QA.
To further improve the neural reasoning capability, recent studies have explored applying GNNs to KG-powered QA, where GNNs naturally ﬁt the graph-structured knowledge and show prominent results.
KagNet (Lin et al., 2019) proposes GCN-LSTM-HPA for path-based relational graph representation.
MHGRN
(Feng et al., 2020) extend Relation Networks (Santoro et al., 2017) to multi-hop relation scope and uniﬁes both path-based models and RGCN
Schlichtkrull et al.
(2018) to enhance the interpretability and the scalability.
QA-GNN (Yasunaga et al., 2021) proposes a LM+GAT framework to joint reasoning over language and KG.
We do not use conventional GNNs and our GSC is extremely simple and efﬁcient even without parameters inside the GSC layers.
KG embeddings are generally used in these systems.
Zhang et al.
(2018) employs propagation of KG embeddings to perform multi-hop reasoning.
KagNet pre-trains KG
Embedding using TransE initialized with GloVe embeddings.
Recently, MHGRN and QA-GNN are proposed, which leverage pre-trained LM to generate embeddings for KG.
In addition, QA-GNN also scores
KG node relevance using LM as extra node embedding.
We follow the data pre-processing procedure of QA-GNN and MHGRN, but our model does not use any KG embedding or node score embedding.
C ONCLUSION
We investigate state-of-the-art GNN-based QA systems, and discover that they are over-parameterized and over-complex.
Our diagnostic analysis using SparseVD shows that the initial node embeddings and some GNN layers are completely dispensable.
Furthermore, our work reveals that GNN essentially works as a counter in the QA reasoning process.
To verify this point, we design soft / hard counter models, which achieve comparable or even better experimental results than existing
GNN-based methods.
Our work is more of an explorative and investigational research, which points out how far we go in the area of knowledge-powered QA and provides helpful insights for the QA community to enlighten future research.
A A PPENDIX A. M ULTI - HOP QA EXPERIMENTS
To apply our GSC to the task of multi-hop QA, we use a popular dataset MetaQA (Zhang et al., 2018), which contains more than 400k single and multi-hop (up to 3-hop) questions in the domain of movies.
The questions were constructed using the knowledge base provided by the WikiMovies (Miller et al., 2016) dataset.
We use the “vanilla” version of the QA pairs in the dataset, and we use the KG provided by the paper which consists of about 43k entities and 135k triples.
We do not encode the entities information in GSC, since our node embedding is initialized with zero.
The edges in the KG are encoded with concatenated one-hot vectors according to the edge type ( types of relation) and the node type (question/answer entity).
Since the task of multi-hop QA does not provide answer choices, thus there is no context node inside the retrieved knowledge graph.
We then adapt 3 units of GSC to align with this scenario:
Methods
[Table]
Edge encoder We equip each GSC layer with a independent edge encoder to learn edge information for different hop.
The input of the edge encoder is also adapted.
Since various of questions can be asked to the same question entity, which means the retrieved graph for different question may be identical, we concatenate the LM encoded question text representation vector with edge type vector as the input of the edge encoder.
So that the model could learn different edges embedding for different question text.
Except the edge encoder is changed, we keep other parts of or GSC architecture the same.
Output mechanism Since all the retrieved nodes can be the answer entities, we output all the node values as scores and the highest scored entity is our ﬁnal answer.
In the hit@1 setting, if our ﬁnal answer is in the set of all the answer labels, we mark it correct.
Supervision Since there could be multiple correct answers in the KG given a question, we process the labels for each question as a multi-hot vector and use MSE loss instead of Cross Entropy as the supervision to the node scores.
We compared our GSC with three standard baselines KV-Mem (Miller et al., 2016), VRN (Zhang et al., 2018) and GraftNet (Sun et al., 2018) on MetaQA and report the hit@1 accuracy in Table 9.
We can see that our GSC achieves comparable performance with the baselines on the MetaQA dataset, which indicates that our observations and hypotheses keep consistent multi-hop QA.
A. H OW WE CONCLUDE THAT GNN IS A COUNTER
[Table]
The process we conclude that GNN is a counter is not direct, our ﬁnding is based on sufﬁcient experiments and obtained by a step by step process:
We ﬁnd that the initial node embedding is dispensable and the layers inside the GNN preserve relatively low sparse ratio via the SparseVD analysis.
So we remove the initial node embedding layers and try to set different hidden dimension for GNN layers to see how the number of parameters affect the performance.
The experiments result can be found at Table 10, where
we
do
ablation
study on the hidden dimension
size of
the message passing
layers
in the GAT model, where we ﬁnd that the model still works when the hidden size is extremely small, and
the
model get worse when the hidden size is extremely large due to the problem of over-ﬁtting.
And we keep the hidden size of edge encoder with 32 since we ﬁnd if the hidden size of edge encoder is extremely small the model will dis-converge.
Since the hidden dimension size of the message passing layers in the GAT model can not be set to even smaller, we design our own model with hidden dimension equal to 1 and simplify the attention based message passing layers to a non-parameterized operation.
We surprisingly ﬁnd that this model can still work and even work better.
So we further study this model
and ﬁnd it works like a counter, so we name it Graph Soft Counter.
To further verify our perspective that the GNN works like a Counter, we design the hard counter setting, and ﬁnd that it also works and could achieve comparable performance.
We may conclude that current GNNs in QA tasks work like a counter and counting for edges in graph are crucial in reasoning.
A. S PARSE R ATIO C URVE
We attach the details of the training curve of SparseVD dissection for QA-GNN as following.
We ﬁnd the linear query and key layers inside GNN layers and graph pooler generally can be compressed to a relatively low sparse ratio and this is more obvious at the ﬁst three GNN layers.
This indicated the these layers may be over-parameterized and the attention mechanism (Vaswani et al., 2017;
Veliˇckovi´c et al., 2017) may degenerate to linear function in this case if we remove these layers.
As expected, the ﬁnal fully connected layer (FC) preserved the highest sparse ratio, since it serves for directly generating the overall score of a choice and it is very efﬁcient (the output dimension is 1).
Except that, the edge encoder with a two-layer MLP has very high sparse ratio, which can be explained as the edge encoding is essential for GNN reasoning in these tasks.
This is also why we only keep the edge encoder with relatively more learnable parameters and simplify the other components in GNN to design our Graph Soft Counter.
A. O PEN B OOK QA R ESULTS
We ﬁnd a small error on the Table 4 of QA-GNN paper (Yasunaga et al., 2021).
According to the principle that the test results should be evaluated on the best model on the dev split, instead of directly picking the best model on test split.
So we compute the result of QA-GNN on OpenBookQA dataset refer to the ofﬁcial worksheets † released by the author.
We ﬁnd this result is different from the result reported on the paper: We report [seed0: 70.20, seed1: 64.80, seed2: 68.40, mean: 67.80, std: 2.75] as the test results which are evaluated the best model on the dev split, while the original paper report [seed0: 70.20, seed1: 69.60, seed2: 71.80, mean: 70.53, std: 1.14] as the test results which are directly picking the best model on test split.
We mention this minor difference of the reported numbers here in case someone may be confused by it.
And QA-GNN is a excellent method to solve
QA tasks with GNN and LM.
A. S OFT C OUNT OF E DGES
We list the top-30 edge triplets with highest soft counts that encoded by the edge encoder of our GSC as following.
These combinations of edge types and node types have relatively higher counts, which means they can contribute more to the ﬁnal graph score.
A. S PARSE VD
To take a deep scope of the graph neural networks based knowledge-aware systems, we introduce a neural model pruning method Sparse Variational Dropout (SparseVD, (Molchanov et al., 2017)) into this scenario as a dissection tool to automatically dissect the graph network architecture.
We implemented the SparseVD optimization based layer refer to the PyTorch code ‡ released by the author.
To keep the dissection in strict accordance with the theoretical derivation of Molchanov et al.
(2017), we apply regulation to all the linear layers in the GNN modules.
We attach the SparseVD problem formulation as well as the optimization target as following.
Consider a dataset D which is constructed from N pairs of objects ( x n , y n ) Nn =1 .
Our goal is to tune the parameters w of a model p ( y | x, w ) that predicts y given x and w .
In Bayesian Learning we usually have some prior knowledge about weights w , which is expressed in terms of a prior distribution p ( w ) .
After data D arrives, this prior distribution is transformed into a posterior distribution p ( w | D ) =
p ( D | w ) p ( w ) /p
( D ) .
This process is called Bayesian Inference .
Computing posterior distribution
Head Node Type
Relation Type
Head Node Type
Soft Count Value
Q entity desires (inverse)
Q entity 0.
Q entity desires
Q entity 0.
Q entity has subevent (inverse)
Q entity 0.
Q entity is
Q entity 0.
Q entity has property
Q entity 0.
Q entity does not desires
Q entity 0.
Q entity does not desires (inverse)
Q entity 0.
Q entity causes
Q entity 0.
Q entity has property (inverse)
Q entity 0.
A entity is
Q entity 0.
Q entity causes (inverse)
Q entity 0.
Q entity is part of
Q entity 0.
Q entity is capable of (inverse)
Q entity 0.
A entity has property
Q entity 0.
A entity desires
Q entity 0.
Q entity is the antonym of (inverse)
Q entity 0.
Q entity is created by (inverse)
Q entity 0.
Q entity is capable of
Q entity 0.
Q entity is part of (inverse)
Q entity 0.
Q entity desires (inverse)
A entity 0.
Q entity is at location of
Q entity 0.
Q entity is created by
Q entity 0.
Q entity is at location of (inverse)
Q entity 0.
Q entity is
A entity 0.
Q entity has property
A entity 0.
Q entity has subevent (inverse)
A entity 0.
O entity is
Q entity 0.
O entity has property
Q entity 0.
Q entity desires
A entity 0.
Q entity is the antonym of
Q entity 0.
highest soft counts here, and it is generated by the edge encoder of GSC model.
The combination of edge types and node types with a higher count means it can contribute more to the ﬁnal graph score.
using the Bayes rule usually involves computation of intractable multidimensional integrals, so we need to use approximation techniques.
One of such techniques is Variational Inference .
In this approach the posterior distribution p ( w | D ) is approximated by a parametric distribution q φ ( w ) .
The quality of this approximation is measured in terms of the Kullback-Leibler divergence D KL ( q φ ( w ) (cid:107) p ( w | D )) .
The optimal value of variational parameters φ can be found by maximization of the variational lower bound :
[Equation]
It consists of two parts, the expected log-likelihood L D ( φ ) and the KL-divergence
D KL ( q φ ( w ) (cid:107) p ( w )) , which acts as a regularization term.
We formulate the model prunings (parsiﬁcation) problem pruning problem as a variational inference problem and use the Kullback-Leibler Divergence approximation following (Achterhold et al., 2018;
Molchanov et al., 2017) to constrain the model parameters to converge to the pruning prior: Original − D KL was obtained by averaging over 10 7 samples of (cid:15) with less than 2 × 10 − 3 variance of the estimation.
[Equation]
Where σ ( · ) denotes the sigmoid function.
We can see that − D KL term increases with the growth of α .
It means that this regularization term favors large values of α .
The case of α ij → ∞ corresponds to a Binary Dropout rate p ij → 1 (recall α = p 1 − p ).
Intuitively it means that the corresponding weight is almost always dropped from the model.
Therefore its value does not inﬂuence the model during the training phase and is put to zero during the testing phase.
We can also look at this situation from another angle.
Inﬁnitely large α ij corresponds to inﬁnitely large multiplicative noise in w ij .
It means that the value of this weight will be completely random and its magnitude will be unbounded.
It will corrupt the model prediction and decrease the expected log likelihood.
Therefore it is beneﬁcial to put the corresponding weight θ ij to zero in such a way that α ij θ 2 ij goes to zero as well.
It means that q ( w ij | θ ij , α ij ) is effectively a delta function, centered at zero δ ( w ij ) .
[Equation]
In the case of linear regression this fact can be shown analytically.
We denote a data matrix as X N × D and α, θ ∈ R D .
If α is ﬁxed, the optimal value of θ can also be obtained in a closed form.
[Equation]
Assume that ( X (cid:62) X ) ii (cid:54) = 0 , so that i -th feature is not a constant zero.
Then from (5) it follows that θ i =
Θ( α − 1 i ) when α i → + ∞ , so both θ i and α i θ
i tend to 0 .
In our dissection scenario, we optimize the weight θ ij to zero no longer for compressing the model but for ﬁguring out which part of the model can be pruned out (sparse ratio to zero) without loss of accuracy, which indicates that part of model is not important and its output information is unused in optimization.