Unsupervised Keyword Extraction for Full-Sentence VQA (2020)

Abstract: In the majority of the existing Visual Question Answering (VQA) research, the answers consist of short, often single words, as per instructions given to the annotators during dataset construction. This study envisions a VQA task for natural situations, where the answers are more likely to be sentences rather than single words. To bridge the gap between this natural VQA and existing VQA approaches, a novel unsupervised keyword extraction method is proposed. The method is based on the principle that the full-sentence answers can be decomposed into two parts: one that contains new information answering the question (i.e. keywords), and one that contains information already included in the question. Discriminative decoders were designed to achieve such decomposition, and the method was experimentally implemented on VQA datasets containing full-sentence answers. The results show that the proposed model can accurately extract the keywords without being given explicit annotations describing them.

## introduction visual recognition is one of the most actively researched fields; this research is expected to be applied to real-world systems such as robots. since innumerable object classes exist in the real world, training all of them in advance is impossible. thus, to train image recognition models, it is important for real-world intelligent systems to actively acquire information. one promising approach to acquire information on the fly is learning by asking, i.e., generating questions to humans about unknown objects, and consequently learning new knowledge from the human response @xcite @xcite . this implies that if we can build a visual question answering (vqa) system @xcite that functions in the real given an image, the question, and the full-sentence answer, the keyword extraction model extracts a keyword from the full-sentence answer. in this example, the word "candles" is the most important part, answering the question "what is in front of the animal that looks white?". therefore, "candles" is considered as the keyword of the answer. vqa is a well-known vision and language task which aims to develop a system that can answer a question about an image. one typical dataset used in vqa is the vqa v2 dataset @xcite . the answers in the vqa v2 dataset are essentially single words. this is because the annotators are instructed to keep the answer as short as possible when constructing the dataset. the ultimate goal of the present work is to gain knowledge through vqa that can be easily transferred to other tasks, such as object class recognition and object detection. therefore, the knowledge (vqa answers) should be represented by a single word, such as a class label. however, in real-world dialog, answers are rarely ex-pressed by single words; rather, they are often expressed as complete sentences. in fact, in visdial v1.0 @xcite , a dataset of natural conversations about images that does not have a word limit for answers, the average length of answers is 6.5 words. this is significantly longer than the average length of the answers in the vqa v2 dataset (1.2 words). to bridge the gap between existing vqa research and real-world vqa, a challenging problem must be solved: identifying the word in the sentence that corresponds to the answer to the question. it must also be considered that fullsentence answers provided by humans are likely to follow a variety of sentence structures. thus, the traditional approaches, such as rule-based approaches based on part-of-speech tagging or shallow parsing, require a great deal of work on defining rules in order to extract the keywords. our key challenge is to propose a novel keyword extraction method that leverages information from images and questions as clues, without the heavy work of annotating keywords or defining the rules. this work handles the task of extracting a keyword when a full-sentence answer is obtained from vqa (full-sentence vqa). the simplest approach to this task is to construct a dataset containing full-sentence answers and keyword annotations, and then train a model based on this dataset in a supervised manner. however, the cost of constructing a vqa dataset with full-sentence answers and keyword annotations is very high. if a keyword extraction model can be trained on a dataset without keyword annotations, we can eliminate the high cost of collecting keyword annotations. we propose an unsupervised keyword extraction model using a full-sentence vqa dataset which contains no keyword annotations. here, the principle is based on the intuition that the keyword is the most informative word in the full-sentence answer, and contains the information that is not included in the question (i.e., the concise answer). essentially, the full-sentence answer can be decomposed into two types of words: (1) the keyword information that is not included in the question, and (2) the information that is already included in the question. for example, in the answer "the egg shaped ghost candles are in front of the bear." to the question "what is in front of the animal that looks white?", the word "candles" is the keyword, while the remaining part "the egg shaped ghost something is in front of the bear" is either information already included in the question or additional information about the keyword. in this case, words like "egg," "ghost," and "bear" are also not in the question, making it difficult to find the keyword via naive methods, e.g., rule-based keyword extraction. our proposed model utilizes image features and question features to calculate the importance score for each word in the fullsentence answer. therefore, based on the contents of the image and the question, the model can accurately estimate which words in the full-sentence answer are important. to the best of our knowledge, this is the first attempt at extracting a keyword from full-sentence vqa in an unsupervised manner. the main contributions of this work are as follows: (1) we propose a novel task of extracting keywords from full-sentence vqa with no keyword annotations. (2) we designed a novel, unsupervised keyword extraction model by decomposing the full-sentence answer. (3) we conducted experiments on two vqa datasets, and provided both qualitative and quantitative results that show the effectiveness of our model. ## model this section describes the proposed method in detail. first, the principal concept of the model is shown in figure 2 . to extract the keyword, we intend to obtain two features from the full-sentence answer, each representing the keyword information and the information derived from the question, respectively. to ensure that these two features discriminatively include keyword information and question information, we intend to reconstruct the original questions and answers from the question features and keyword features, respectively. thus, if we successfully extract the keyword and the question information from the full-sentence answer, we can reconstruct original full-sentence answer and the question. essentially, given an image, its corresponding question, and full-sentence answer, our proposed model extracts the keyword of the answer by decomposing the keyword information and the question information in the answer. ## experimental results the experimental results are shown in ## conclusion in this paper, we proposed the novel task of unsupervised keyword extraction from full-sentence vqa. a novel model was designed to handle this task based on information decomposition of fullsentence answers and the reconstruction of questions and answers. both qualitative and quantitative experiments show that our model successfully extracts the keyword of the full-sentence answer with no keyword supervision. in future work, the extracted keywords will be utilized in other tasks, such as vqa, object classification, or object detection. this work could also be combined with recent works on vqg @xcite . in these works, the system generates questions to acquire information from humans. however, they assume that the answers are obtained as single words, which will pose a problem when applying it to the real-world question answering. by combining these studies with our research, an intelligent system can ask humans about unseen objects and learn new knowledge from the answer, even if the answer consists of more than a single word.