Imagination-Augmented Natural Language Understanding (2022)

Abstract: Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefore, we introduce an Imagination-Augmented Cross-modal Encoder (iACE) to solve natural language understanding tasks from a novel learning perspective—imagination-augmented cross-modal understanding. iACE enables visual imagination with external knowledge transferred from the powerful generative and pre-trained vision-and-language models. Extensive experiments on GLUE and SWAG show that iACE achieves consistent improvement over visually-supervised pre-trained models. More importantly, results in extreme and normal few-shot settings validate the effectiveness of iACE in low-resource natural language understanding circumstances.

## introduction cognitive neuroscience studies reveal neural activation in vision-related brain areas when reading text @xcite and show a tight relationship between brain areas processing linguistic and visual semantic information @xcite . in addition, visual imagery improves comprehension during human language processing @xcite . such imagination empowers hu-1 source code and pre-trained models are publicly available at https://github.com/yujielu10/iace-nlu premise: a senior is waiting at the window of a restaurant that serves sandwiches. hypothesis: a man is waiting in line for the bus. ## our approach we illustrate how we solve the existing text-only learning problem in natural language understanding tasks as the imagination-augmented cross-modal language understanding (iclu) problems in section 3.1. then we give a detailed illustration of our proposed iace's architecture in section 3.2. finally, we describe the procedure and training protocol of the perceptual-enhanced linguistic understanding paradigm in section 3.3. ## conclusion we treat the text-only learning problem in natural language understanding tasks as a cross-modal language understanding problem with generated imagination as supervision. in this scenario, the task aims to bridge the gap between the human and the agent language understanding in both linguistic and perceptual procedures. to address the proposed problem, we devised a model-agnostic learning paradigm iace. specifically, we build the imagination of the downstream dataset using an interactive generative approach with guidance from a self-supervised pre-trained large-scale image and text model. our proposed iace surpassed baselines of two architecture sizes by a large margin in the few-shot setting. the improvement is consistently observed over pure-language baselines (bert and roberta) and visually supervised vo-ken on the glue and swag dataset. the results show the superiority of our iace in language understanding and handling low-resource circumstances.