Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020)

Abstract: Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights. We present the first study focused on generating natural language rationales across several complex visual reasoning tasks: visual commonsense reasoning, visual-textual entailment, and visual question answering. The key challenge of accurate rationalization is comprehensive image understanding at all levels: not just their explicit content at the pixel level, but their contextual contents at the semantic and pragmatic levels. We present RationaleˆVT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks. In addition, we find that integration of richer semantic and pragmatic visual features improves visual fidelity of rationales.

## introduction explanatory models based on natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans @xcite . in figure 1 , for example, the natural language rationale given in free-text provides a much more informative and conceptually relevant explanation to the given qa problem compared to the non-linguistic explanations that are often provided as localized visual highlights on the image. the latter, while pertinent to what the vision component of the model was attending to, cannot provide the full scope of rationales for such complex reasoning tasks as illustrated in figure 1 . indeed, explanations for higher-level conceptual reasoning can be best conveyed through natural language, as has been studied in recent literature on (visual) nli @xcite , (visual) qa @xcite , playing arcade games @xcite , fact checking @xcite , image classification @xcite , motivation prediction @xcite , and self-driving cars @xcite . in this paper, we present the first focused study on generating natural language rationales across several complex visual reasoning tasks: visual commonsense reasoning, visual-textual entailment, and visual question answering. our study aims to complement the more broadly studied lower-level explanations such as attention weights and gradients in deep neural networks @xcite @xcite . because free-text rationalization is a challenging research question, we assume the gold answer for a given instance is given and scope our investigation to justifying the gold answer. the key challenge in our study is that accurate rationalization requires comprehensive image understanding at all levels: not just their basic content at the pixel level (recognizing "waitress", "pancakes", "people" at the table in figure 1 ), but their contextual content at the semantic level (understanding the structural relations among objects and entities through action predicates such as "delivering" and "pointing to") as well as at the pragmatic level (understanding the "intent" of the pointing action is to tell the waitress who ordered the pancakes). we present rationale vt transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models based on gpt-2 @xcite with visual features. besides commonly used features derived from object detection (fig. 2a ), we explore two new types of visual features to enrich base models with semantic and pragmatic knowledge: (i) visual semantic frames, i.e., the primary activity and entities engaged in it detected by a grounded situation recognizer (fig. 2b ; @xcite , and (ii) commonsense inferences inferred from an image and an optional event predicted from a visual commonsense graph (fig. 2c ; @xcite . 1 we report comprehensive experiments with careful analysis using three datasets with human rationales: (i) visual question answering in vqa-e @xcite , (ii) visual-textual entailment in e-snli-ve @xcite , and (iii) an answer justification subtask of visual commonsense reasoning in vcr @xcite . our empirical findings demonstrate that while free-text rationalization remains a challenging task, newly emerging state-of-the-art models support rationale generation as a promising research direction to complement model interpretability for complex visual-textual reasoning tasks. in particular, we find that integration of richer semantic and pragmatic visual knowledge is important for generating rationales with higher visual fidelity, especially for tasks that require higher-level concepts and richer background knowledge. our code, model weights, and the templates used for human evaluation are publicly available. 2 2 rationale generation with rationale vt transformer our approach to visual-textual rationalization is based on augmenting gpt-2's input with output of external vision models that enable different levels of visual understanding. ## experiments for all experiments, we visually adapt and finetune the original gpt-2 with 117m parameters. we train our models using the language modeling loss computed on rationale tokens. 8 tasks and datasets we consider three tasks and datasets shown in the key difference among the three tasks is the level of required visual understanding. we report here the main observations about how the datasets were collected, while details are in the appendix §a.2. foremost, only vcr rationales are human-written for a given problem instance. rationales in vqa-e are extracted from image captions relevant for question-answer pairs @xcite using a constituency parse tree. to create a dataset for explaining visual-textual entailment, e-snli-ve, @xcite combined the snli-ve dataset @xcite for visual-textual entailment and the e-snli dataset @xcite for explaining textual entailment. we notice that this methodology introduced a data collection artifact for entailment cases. to illustrate this, consider the example in figure 3 . in visual-textual entailment, the premise is the image. therefore, there is no reason to expect that a model will build a rationale around a word that occurs in the textual premise it has never seen ("jumping"). we will test whether models struggle with entailment cases. human evaluation for evaluating our models, we follow @xcite who show that bleu @xcite is not reliable for evaluation of rationale generation, and hence use human evaluation. 9 we believe that other automatic sentence similarity measures are also likely not suitable due to a similar reason; multiple rationales could be plausible, although not necessarily paraphrases of each other (e.g., in figure 4 both generated and human rationales are plausible, but they are not strict paraphrases). 10 future work might consider newly emerging learned evaluation measures, such as bleurt @xcite , that could learn to capture non-trivial semantic similarities between sentences beyond surface overlap. we use amazon mechanical turk to crowdsource human judgments of generated rationales according to different criteria. our instructions are provided in the appendix §a.6. for vcr, we randomly sample one qa pair for each movie in the development split of the dataset, resulting in 244 examples for human evaluation. for vqa and e-snli-ve, we randomly sample 250 examples from their development splits. 11 we did not use any of these samples to tune any of our hyperparameters. each generation was evaluated by 3 crowdworkers. the workers were paid ∼$13 per hour. baselines the main objectives of our evaluation are to assess whether (i) proposed visual features help gpt-2 generate rationales that support a given answer or entailment label better (visual plausibility), and whether (ii) models that generate more plausible rationales are less likely to mention content that is irrelevant to a given image (visual fidelity). as a result, a text-only gpt-2 approach represents a meaningful baseline to compare to. in light of work exposing predictive data artifacts (e.g., @xcite , we estimate the effect of artifacts by reporting the difference between visual plausibility of the text-only baseline and plausibility of its rationales assessed without looking at the image (textual plausibility). if both are high, then there are problematic lexical cues in the datasets. finally, we report estimated plausibility of human rationales to gauge what has been solved and what is next. 12 ## related work rationale generation applications of rationale generation (see §1) can be categorized as text-only, vision-only, or visual-textual. our work belongs to the final category, where we are the first to try to 14 we also study assessing fidelity from phrases that are extracted from a rationale (see appendix b). generate rationales for vcr @xcite . the bottom-up top-down attention (butd) model @xcite has been proposed to incorporate rationales with visual features for vqa-e and e-snli-ve @xcite . compared to butd, we use a pretrained decoder and propose a wider range of visual features to tackle comprehensive image understanding. ## conclusions we present rationale vt transformer, an integration of a pretrained text generator with semantic and pragmatic visual features. these features improve visual plausibility and fidelity of generated rationales for visual commonsense reasoning, visual-textual entailment, and visual question answering. this represents progress in tackling important, but still relatively unexplored research direction; rationalization of complex reasoning for which explanatory approaches based solely on highlighting parts of the input are not suitable. this research was supported in part by nsf (iis1524371, iis-1714566), darpa under the cwc program through the aro (w911nf-15-1-0543), darpa under the mcs program through niwc pacific (n66001-19-2-4031), and gifts from allen institute for artificial intelligence.