Visually-Enhanced Phrase Understanding (2023)

Abstract: Large-scale vision-language pre-training has exhibited strong performance in various visual and textual understanding tasks. Recently, the textual encoders of multi-modal pre-trained models have been shown to generate high-quality textual representations, which often outperform models that are purely text-based, such as BERT. In this study, our objective is to utilize both textual and visual encoders of multi-modal pre-trained models to enhance language understanding tasks. We achieve this by generating an image associated with a textual prompt, thus enriching the representation of a phrase for downstream tasks. Results from experiments conducted on four benchmark datasets demonstrate that our proposed method, which leverages visually-enhanced text representations, significantly improves performance in the entity clustering task.

## introduction recent advances in vision-language pre-training have seen the successful alignment of visual and linguistic inputs through the implementation of cross-modal pre-training objectives, such as language modeling and contrastive learning @xcite . these pre-trained models have shown impressive performance on downstream vision-language tasks, validating their crossmodal capabilities @xcite . while most previous studies focused on multimodal tasks, researchers have shown that pretrained cross-modal encoders are equally proficient at uni-modal language understanding, matching the performance of pre-trained text encoders. @xcite were the pioneers in utilizing machine abstract imagination from pre-trained crossmodal encoders, demonstrating improvement on general nlu tasks. @xcite established that the text encoder of clip @xcite surpasses models designed for producing phrase representations, including phrase-bert @xcite and uctopic @xcite . they hypothesized that the visual supervision during pre-training empowers clip to produce visuallygrounded phrase representations, beneficial for language-only tasks. such a phenomenon aligns with neuroscience studies, demonstrating that visual and linguistic semantic representations are coordinated in the human brain @xcite . despite the strong performance of the previous method, it only utilized the text encoder of a crossmodal pre-trained model. in contrast, our study aims to exploit its multi-modal representation capacity, incorporating both text and image encoders. we introduce a visually-enhanced phrase understanding framework to exploit multiple modalities for uni-modal tasks. our framework comprises a text-to-image generator and a text-image crossmodal encoder. we employ a text-to-image generator to produce visual cues for a textual candidate. subsequently, the generated image and the textual prompt are processed by the cross-modal encoder to create visually-enhanced phrase embeddings. unlike @xcite , our method does not require supervised data for downstream tasks, making it more scalable. our approach also differs from voken @xcite , as they generated visual cues in tokens and processed the signal solely on the language side, whereas we employ representations directly from different modalities. therefore, our model can capture more abstract concepts from images, enhancing generalizability. we evaluate our approach on four benchmark phrase understanding datasets. the experiments demonstrate that our proposed visual enhancement significantly outperforms all text-only baselines, demonstrating that abstract visual concepts can provide complementary cues for text understanding. our proposed method is illustrated in figure 1 , where we first generate images associated with phrases using a text-to-image diffusion model. following this, we utilize pre-trained text and image encoders to construct visually-enhanced phrase embeddings for downstream understanding tasks. ## experiments to evaluate whether our visually-enhanced phrase embeddings provide improved semantic cues, we conduct a series of experiments focused on entity clustering, as our primary task is to categorize entity candidates with similar concepts only based on phrase representations in an unsupervised fashion. ## conclusion this work presents a multi-modal framework that leverages a text-to-image model to bridge between language and visual modalities for enhancing text comprehension. the model effectively transforms text inputs into coherent images, enriching phrase representations by merging outputs from different modalities. experimental results show our framework surpassing robust phrase understanding models across diverse domains.