On the Difference of BERT -style and CLIP -style Text Encoders (2023)

Abstract: Masked language modeling (MLM) has been one of the most popular pretraining recipes in natural language processing, e.g. , BERT, one of the representative models. Recently, contrastive language-image pretraining (CLIP) has also attracted attention, especially its vision models that achieve excellent performance on a broad range of vision tasks. However, few studies are dedicated to studying the text encoders learned by CLIP. In this paper, we analyze the difference between BERT-style and CLIP-style text encoders from three experiments: (i) general text understanding, (ii) vision-centric text understanding, and (iii) text-to-image generation. Experimental analyses show that although CLIP-style text encoders underperform BERT-style ones for general text understanding tasks, they are equipped with a unique ability, i.e. , synesthesia , for the cross-modal association, which is more similar to the senses of humans.

## introduction text representation learning provides a feasible solution to extract generic representations from texts, allowing models to better understand and make predictions about texts. normally, to perform this, a language model is pretrained on large-scale text corpora to learn text representation in a self-supervised manner, and it can be further used on downstream tasks, e.g., text classification and question answering @xcite . there are many recipes for pretraining text encoders 1 @xcite @xcite . one of the most popular ways is masked language modeling (mlm), a fill-in-the-blank task where a model uses the context words to predict masked tokens in a sequence. for this type, bert @xcite and its variants @xcite @xcite are the representative encoder models allowing the bidirectional perception of texts. more recently, there has been another framework to produce text encoders, i.e., contrastive languageimage pretraining @xcite . it trains image and text encoders through contrastive learning on a variety of image-text pairs, and the trained vision encoders achieve great success on vision-only tasks, especially its impressive zeroshot transfer results. however, few studies investigated the trained text encoders. in this work, we first conduct a pilot study in §3 to benchmark bert-style and clip-style text encoders in a popular natural language processing benchmark @xcite . it shows that clip-style text encoders significantly underperform bert-style text encoders. therefore, a natural question arises: "are clip text encoders useless byproducts (or in which case can we make use of the clip text encoders?)". our hypothesis for this question is that clip text encoder might additionally learn visual knowledge of textual concepts, which could be complementary to the semantic alignment of textual concepts; the latter is well-captured by bert text encoders. this reminds us of a phenomenon called 'synesthesia' foot_0 . to validate our synesthesia hypothesis, we conduct a side-by-side comparison between bertstyle and clip-style text encoders from two aspects: 1) benchmarking them in vision-centric natural language understanding tasks in §4; and 2) further probing the generated images from their encoded text representation in §5. first, we evaluate the two types of encoders on the cxc dataset @xcite , where the groundtruth similarity is annotated from both textual and visual perspectives; second, we directly generate images based on the two encoded text representations from bert-style and clip-style text encoders, respectively. to achieve this, we train a single linear transformation layer to transfer the two types of text representations as prompts to a frozen image decoder. experimental analyses show that although clip text encoders are not comparable to bert-style text encoders in general text understanding tasks, they have a unique ability, i.e., synesthesia, to associate a text and its visual appearance. this might inspire more studies on text encoders in the future. our codes are available at ## preliminaries in this section, we detail the bert-style and clipstyle text encoders in §2.1 and §2.2, respectively. ## clip-style text encoders capture visual perception for concept similarity motivation the aforementioned experiments show that bert-style text encoders outperform clip-style text encoders on pure text tasks. therefore, a question is "is there any text task for us to testify the superior of clip-style text encoders?". clip-style text encoders are trained under multimodal settings and intuitively, they are better at associating a text with a real-life scenario. to find the answer, we designed a vision-centric text understanding task (described as follows). ## clip-style text encoders capture visual perception for image generation motivation in previous sections, we verify the superiority of clip-style text encoders by associating a text with an image on textural tasks. next, we consider a question "why don't we directly translate a learned text representation to an image to compare their visual perception ability in a more straightforward way?". to this end, we design a text-to-image generation pipeline to probe the association ability. ## conclusion human interaction is multi-modal. starting from the conjecture that text encoders learned from multi-modal data have unique abilities, in this paper, we study the behavioral difference between bert-style and clip-style text encoders. we compare them from three aspects systematically: (i) general (pure) text understanding; (ii) visioncentric text understanding; and (iii) text-to-image generation. experimental analyses show that although clip-style text encoders underperform bert-style text encoders on general text understanding tasks, they have a unique ability, i.e., synesthesia, to associate a text and its visual appearance, which is more similar to human perception.