Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision (2020)

Abstract: Humans learn language by listening, speaking, writing, reading, and also, via interaction with the multimodal real world. Existing language pre-training frameworks show the effectiveness of text-only self-supervision while we explore the idea of a visually-supervised language model in this paper. We find that the main reason hindering this exploration is the large divergence in magnitude and distributions between the visually-grounded language datasets and pure-language corpora. Therefore, we develop a technique named “vokenization” that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images (which we call “vokens”). The “vokenizer” is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks such as GLUE, SQuAD, and SWAG.

## introduction most humans learn language understanding from multiple modalities rather than only from the text and audio, especially using the visual modality. as claimed in @xcite , visual pointing is an essential step for most children to learn meanings of words. however, existing language pretraining frameworks are driven by contextual learning which only takes the language context as selfsupervision. for example, word2vec @xcite takes surrounding bag-of-words; elmo @xcite and gpt @xcite humans learn language by listening, speaking ## visually-supervised language models contextual language representation learning is driven by self-supervision without considering explicit connections (grounding) to the external world. in this section, we illustrate the idea of a visuallysupervised language model and discuss the challenges of creating its visual supervision. ## vokenization in the previous section, we discuss the potential of using vokens (i.e., visualized tokens) as visual supervision to the language model, and also demonstrate the large gap between currently available resources (i.e., annotated dataset) and the desired requirements. hence, in this section, we develop a framework that can generate vokens. as shown in fig. 2 , the general idea is that we learn a "vokenizer" from image-captioning dataset and use it to annotate large language corpora (i.e., english wiki), thus bridging the gap between grounded language and other types of natural language. we start by illustrating the vokenization process and then describe how we implement it. ## conclusion in this paper, we explored the possibility of utilizing visual supervision to language encoders. in order to overcome the challenges in grounded language, we develop the vokenizer with contextual token-image matching models and use it to vokenize the language corpus. supervised by these generated vokens, we observe a significant improvement over the purely self-supervised language model on multiple language tasks. resnext-101-32x8d feature maps are used as features for tokens and the images. for both multilayer perceptrons w mlp θ and x mlp θ , we use two fully-connected layers with relu activation, where the output dimensions of the two layers are 256 and 64, accordingly. we only train the modules marked with θ, i.e., the two backbone models bert @xcite and resnext @xcite are not fine-tuned. since we normalize the features g(w i ; s) and f (v) to be norm-1 vectors, the relevance score thus takes the range from [-1, 1] (from the cauchy inequality). the margin m in hinge loss is set to 0.5. during the vokenization process, we use the faiss @xcite library to speed up the nearest neighbor search. the vokenization runs at a speed of 100k tokens / second with 4 titan v100 gpu. thus the vokenization of the full wikipedia is finished in 8 hours. when transferring vokens to other pre-training frameworks, revokenization does not need the gpu computation and runs as fast as the tokenization. the vokens are retrieved from the visual genome images which are not used in ms coco (our training dataset). we take a voken size of 50000. when pre-training the model on pure language corpus, we unify the training process to avoid possible side effects from different training protocols. we follow previous work to conduct two simplifications: 1. removing the next-sentence-prediction task @xcite 2. using fixed sequence length @xcite of 128. we take the 12-layer bert base model of 768 hidden dimensions and train it on english wikipedia foot_7 for 200k steps from scratch. we also take a reduced 6-layer model and train it on wiki103 foot_8 for 40 epochs (160k steps) from scratch because this reduced model does not fit well on the full wikipedia dataset. the voken classification task will not bring additional parameters to the language encoder (with 110m parameters) but need more computations, we thus adjust the training steps for pure masked-language-model (mlm) training for a fair comparison. it results in around 10% more training steps in pure mlm training. all models take batch sizes of 256 and a learning rate of 2e-4.