DIVE : Towards Descriptive and Diverse Visual Commonsense Generation (2023)

Abstract: Towards human-level visual understanding, visual commonsense generation has been introduced to generate commonsense inferences beyond images. However, current research on visual commonsense generation has overlooked an important human cognitive ability: generating descriptive and diverse inferences. In this work, we propose a novel visual commonsense generation framework, called DIVE, which aims to improve the descriptiveness and diversity of generated inferences. DIVE involves two methods, generic inference filtering and contrastive retrieval learning, which address the limitations of existing visual commonsense resources and training objectives. Experimental results verify that DIVE outperforms state-of-the-art models for visual commonsense generation in terms of both descriptiveness and diversity, while showing a superior quality in generating unique and novel inferences. Notably, DIVE achieves human-level descriptiveness and diversity on Visual Commonsense Graphs. Furthermore, human evaluations confirm that DIVE aligns closely with human judgments on descriptiveness and diversity.

## introduction humans possess a cognitive ability to reason about the rich and complex stories beyond a given visual scene, based on their background commonsense knowledge. visual commonsense reasoning is a key to this cognition-level visual understanding @xcite , which helps humans comprehend the interactions around them. as research towards the human-level visual understanding of machines, visual commonsense generation @xcite has been introduced. this challenging task aims to generate textual commonsense inferences about potential antecedents and consequences, as "work in the music store with person1" "be interested in person1's guitar playing" "see person1 playing" "listen to the music" "meet person1 in the music store" "hear person1 play guitar" "begin talking with person1 about music" well as the present intents of characters. recent works on visual commonsense generation @xcite have progressed to develop vision-language models capable of generating more plausible and relevant inferences. despite considerable efforts in visual commonsense generation, an important aspect of humans' innate cognitive ability has been overlooked in previous studies: humans can make descriptive and diverse inferences by capturing important, specific, and detailed information within a visual scene. this ability is necessary for making precise and informative inferences about various possible scenarios in the world, but it is lacking in existing models. figure 1 illustrates a case where model-generated inferences still fall short of human-written infer-ences in terms of descriptiveness and diversity. existing models often ignore key details in a given scene, leading to the generation of similar and generic inferences such as "walk up to person1 " and "stand behind person1 " that could happen in most contexts and provide minimal specific detail. in contrast, humans can create more descriptive and diverse inferences like "work in the music store with person1 " by considering the image's details, such as many instruments displayed behind and two men wearing employee id cards. we observe that this deficiency can be largely attributed to the skewed distribution of visual commonsense resources. such resources, typically crowd-sourced, often involve many generic inferences as labels, because humans may not use detailed information from a given visual scene when annotating @xcite . for example, more than 60% of images in visual commonsense graphs (vcg) @xcite involve generic inferences as labels foot_0 . as models repeatedly learn these generic inferences, they tend to generate inferences that vaguely describe a situation, failing to capture detailed information that specifies a given scene. this limitation restricts a deeper understanding of visual information by existing vision-language models. in this paper, we introduce dive (descriptive and diverse visual commonsense generation), a novel visual commonsense generation framework for improving the descriptiveness and diversity of commonsense inferences generated by visionlanguage models. firstly, we construct a balanced visual commonsense graph from vcg, using a carefully designed filtering method that removes generic inferences, with a focus on the semantic concentration of images utilizing the representations of clip @xcite . furthermore, we propose a new contrastive retrieval learning method that facilitates a model to recognize specific details of an image. to verify the efficacy of dive, we conduct experiments on vcg @xcite with popular generative models @xcite . our experiments verify that dive generates more descriptive and diverse inferences compared with state-of-the-art visual commonsense generation models. notably, dive achieves human-level descriptiveness and diversity scores on vcg, significantly improving the generation quality of unique and novel inferences. we further conduct human evaluations on the plausibility, descriptiveness, and diversity of generated inferences, which confirm that dive aligns closely with humans' judgment of descriptiveness and diversity. descriptive and diverse text generation. generating meaningful and informative text has been a challenging goal in many nlp fields including response generation, image captioning, and scene graph generation @xcite @xcite @xcite @xcite . in these works, several problems have been identified in current text generation frameworks. first, generation models often prefer "safe" output sentences that use very high-frequency expressions, and they tend to describe only the obvious facts while ignoring key details @xcite . second, the standard training objective function for text generation can promote the generation of generic text without diversity @xcite . third, conventional metrics for text generation, such as bleu and cider, tend to give high scores to inferences with very common n-grams, as they prioritize ngram overlap @xcite . thus, as the first exploration of descriptive and diverse commonsense inferences, we propose filtering and training methods to prevent models from favoring "safe" sentences. additionally, we construct datasets for evaluation, introduce various metrics, and conduct human evaluations to particularly evaluate descriptiveness and diversity of generated inferences. ## methodology in this section, we introduce dive, a framework designed for descriptive and diverse commonsense generation. first, we propose a generic inference filtering method that balances the distribution of visual commonsense resources (section 3.1). then, we propose a contrastive retrieval learning method that facilitates vision-language models to identify information specific to a given image (section 3.2). figure 2 illustrates the overall procedure of dive. ## experiments in this section, we demonstrate the effectiveness of dive by comparing it with existing methods. ## analysis in this section, we conduct analyses of the components and results of dive bart . ## conclusion we have presented dive to improve the descriptiveness and diversity of vision-language models in visual commonsense generation. we have proposed a generic inference filtering method to balance the skewed distribution of visual commonsense resources, based on the frequency and semantic concentration of images. in addition, we have proposed a contrastive retrieval learning method to promote the descriptiveness and diversity of visionlanguage models, by leveraging the structural infor-mation from visual commonsense graphs. through extensive experiments on vcg, we have verified that dive is capable of generating descriptive and diverse inferences about visual scenes, significantly outperforming state-of-the-art visual commonsense generation models. particularly, our human evaluations have confirmed that dive indeed captures specific visual information, leading to improvements in plausibility, descriptiveness, and diversity.