Hypergraph T ransformer: W eakly-Supervised Multi-hop Reasoning for Knowledge-based Visual Question Answering (2022)

Abstract: Knowledge-based visual question answering (QA) aims to answer a question which requires visually-grounded external knowledge beyond image content itself. Answering complex questions that require multi-hop reasoning under weak supervision is considered as a challenging problem since i) no supervision is given to the reasoning process and ii) high-order semantics of multi-hop knowledge facts need to be captured. In this paper, we introduce a concept of hypergraph to encode high-level semantics of a question and a knowledge base, and to learn high-order associations between them. The proposed model, Hypergraph Transformer, constructs a question hypergraph and a query-aware knowledge hypergraph, and infers an answer by encoding inter-associations between two hypergraphs and intra-associations in both hypergraph itself. Extensive experiments on two knowledge-based visual QA and two knowledge-based textual QA demonstrate the effectiveness of our method, especially for multi-hop reasoning problem. Our source code is available at https://github.com/yujungheo/kbvqa-public .

## introduction visual question answering (vqa) is a semantic reasoning task that aims to answer questions about visual content depicted in images @xcite @xcite , and has become one of the most active areas of research with advances in natural language processing and computer vision. recently, researches for vqa have advanced, from inferring visual properties on entities in a given image, to inferring commonsense or world knowledge about those entities @xcite @xcite @xcite . in this paper, we focus on the task which is called knowledge-based visual question answering, where a massive number of knowledge facts from a general knowledge base (kb) is given with an image-question pair. to answer the given question as shown in figure 1 , a model should understand the semantics of the given question, link visual entities appearing in the given image to the kb, extract a number of evidences from the kb and predict an answer by aggregating semantics of both the question and the extracted evidences. following these, there are two fundamental challenges in this task. i) to answer a complex question, multi-hop reasoning over multiple knowledge evidences is necessary. ii) learning a complex reasoning process is difficult especially in a condition where only qa is provided without extra supervision on how to capture any evidence from the kb and infer based on them. that is, the model should learn which knowledge facts to be attended to and how to combine them to infer the correct answer on its own. following the previous work @xcite , we call this setting under weak supervision. under weak supervision, previous studies proposed memory-based methods @xcite and graph-based methods @xcite to learn to selectively focus on necessary pieces of knowledge. the memory-based methods represent knowledge facts in a form of memory and calculate soft attention scores of each memory with respect to a question. then, it infers an answer by attending to knowledge evidence with high attention scores. on the other hand, to explicitly consider relational structure between knowledge facts, graph-based methods construct a query-aware knowledge graph by retrieving facts from kb and perform graph reasoning for a question. these methods mainly adopt an iterative message passing process to propagate information between adjacent nodes in the graph. however, it is difficult to capture multi-hop relationships containing long-distance nodes from the graph due to the well-known over-smoothing problem, where repetitive message passing process to propagate information across long distance makes features of connected nodes too similar and undiscriminating @xcite . to address the above limitation, we propose a novel method, hypergraph transformer, which exploits hypergraph structure to encode multi-hop relationships and transformer-based attention mechanism to learn to pay attention to important knowledge evidences for a question. we construct a question hypergraph and a knowledge hypergraph to explicitly encode high-order semantics present in the question and each knowledge fact, and capture multi-hop relational knowledge facts effectively. then, we perform hyperedge matching between the two hypergraphs by leveraging transformer-based attention mechanism. we argue that introducing the concept of hypergraph is powerful for multi-hop reasoning problem in that it can encode high-order semantics without the constraint of length and learn cross-modal high-order associations. the main contributions of this paper can be summarized as follows. i) we propose hypergraph transformer which enhances multi-hop reasoning ability by encoding high-order semantics in the form of a hypergraph and learning inter-and intrahigh-order associations in hypergraphs using the attention mechanism. ii) we conduct extensive experiments on two knowledge-based vqa datasets (kvqa and fvqa) and two knowledge-based textual qa datasets (pq and pql) and show superior performances on all datasets, especially multi-hop reasoning problem. iii) we qualitatively observe that hypergraph transformer performs robust in-ference by focusing on correct reasoning evidences under weak supervision. multi-hop knowledge graph reasoning is a process of sequential reasoning based on multiple evidences of a knowledge graph, and has been broadly used in various downstream tasks such as question answering @xcite @xcite , or knowledge-enhanced text generation @xcite @xcite . recent researches have introduced the concept of hypergraph for multi-hop graph reasoning @xcite @xcite @xcite . these models have a similar motivation to the hypergraph transformer proposed in this paper, but core operations are vastly different. these models mainly update node representations in the hypergraph through a message passing process using graph convolution operation. on the contrary, our method update node representations via hyperedge matching of hypergraphs instead of message passing scheme. we argue that this update process effectively learns the high-order semantics inherent in each hypergraph and the high-order associations between two hypergraphs. ## validation for hypergraph transformer we verify the effectiveness of each module in hypergraph transformer. to analyze the performances of the variants in our model, we use kvqa which is a representative and large-scale dataset for knowledge-based vqa. here, we mainly focus on two aspects: i) effect of hypergraph and ii) effect of attention mechanism. to evaluate a pure reasoning ability of the models, we conduct experiments in the oracle setting. ## qualitative analysis figure 3 provides the qualitative analysis on effectiveness of using a hypergraph as an input format to transformer architecture. we present the attention map from the guided-attention block, and visualize top-k attended knowledge facts or entities with the attention scores. in the first example, both model, hypergraph transformer and transformer (sa+ga), infer the correct answer, q5075293. our model responds by focusing on {second ⪯ from ⪯ left} phrase of the question and four facts having a left relation among 86 knowledge hyperedges. in comparison, transformer (sa+ga) strongly attends to the knowledge entities which appear repetitive in the knowledge facts. especially, the model attends to q3476753, q290666 and ireland with the high attention score 0.237, 0.221, and 0.202. in the second example, our model attends to the correct knowledge hyperedges considering the multi-hop facts about place of birth of the people shown in the given image, and infers the correct answer. on the other hand, transformer (sa+ga) strongly attends to the knowledge entity of person (q2439789) presented in the image with undesired attention score 0.788. the second and third attended knowledge entities are the other person (q7141361) and iran. transformer (sa+ga) fails to focus on the multi-hop facts required to answer the given question and predicts the answer with the wrong number at the end. ## discussion and conclusion in this paper, we proposed hypergraph transformer for multi-hop reasoning over knowledge graph under weak supervision. hypergraph transformer adopts hypergraph-based representation to encode high-order semantics of knowledge and questions and considers associations between a knowledge hypergraph and a question hypergraph. here, each node representation in the hypergraphs is updated by inter-and intra-attention mechanisms in two hypergraphs, rather than by iterative message passing scheme. thus, hypergraph transformer can mitigate the well-known over-smoothing problem in the previous graph-based methods exploiting the message passing scheme. extensive experiments on various datasets, kvqa, fvqa, pq, and pql validated that hypergraph transformer conducts accurate inference by focusing on knowledge evidences necessary for question from a large knowledge graph. although not covered in this paper, an interesting future work is to construct heterogeneous knowledge graph that includes more diverse knowledge sources (e.g. documents on web).