On the (In)Effectiveness of Images for Text Classification (2021)

Abstract: Images are core components of multi-modal learning in natural language processing (NLP), and results have varied substantially as to whether images improve NLP tasks or not. One confounding effect has been that previous NLP research has generally focused on sophisticated tasks (in varying settings), generally applied to English only. We focus on text classification, in the context of assigning named entity classes to a given Wikipedia page, where images generally complement the text and the Wikipedia page can be in one of a number of different languages. Our experiments across a range of languages show that images complement NLP models (including BERT) trained without external pre-training, but when combined with BERT models pre-trained on large-scale external data, images contribute nothing.

## introduction combining data from multiple modalities (e.g., text, images, categorical metadata, or user interaction features) has become commonplace in artificial intelligence. in nlp, examples include multimodal machine translation (mmt) @xcite , visual question answering (vqa) @xcite , visual commonsense reasoning (vcr) @xcite , and multi-modal pretraining @xcite . while tasks such as vqa and vcr are multimodal in nature, there has been research on traditionally text-based tasks such as text classification @xcite and word embedding learning @xcite which has demonstrated that the addition of images boosts performance. at the same time, however, there is evidence of images providing no additional information, e.g. @xcite show that mmt models learn to ignore visual content when trained on a parallel corpus of image captions @xcite . these mixed findings raise the question of when visual context is actually useful in nlp. in this work, we take a first step towards answering this question, in focusing on the task of text classification, which has traditionally been addressed using textual data only. we identify two gaps in the literature on multi-modal nlp: (1) no results for pre-trained language models (lms); and ## task description this research is couched in the context of a shared-task dataset released by the shinra project @xcite , aimed at classifying wikipedia pages into fine-grained entity classes. 1 we chose this benchmark as many wikipedia documents contain images, and data is provided for a total of 29 typologically-diverse languages. 2 the task is not trivial as it involves classifying wikipedia documents into a set of 219 classes, with the possibility of multiple labels for a given document. the number of annotated pages for each language in the shinra dataset is shown in the shinra dataset contains only textual information from the original documents. in order to add images, we extract the image links from the english wikipedia dump of june 2020 foot_0 using the zim library. 5 the extracted images are then linked with image links in the source documents in the shinra dataset, foot_2 resulting in about 88% pages being augmented with images (noting that images are generally shared across wikipedia pages for different languages other than english). out of the 30 languages in the original shinra dataset, we experiment primarily with arabic ("ar"), english ("en"), finnish ("fi"), hindi ("hi"), and mandarin chinese ("zh"), selected to span five different language families and where the dataset size is relatively large. from the shinra data, we randomly sample 30k documents for each language, and construct a 80%/10%/10% fixed split for training/development/test in each language. we use a maximum of four images for each document. 7 3 baseline experiments our first set of experiments is aimed at evaluating the empirical utility of images in the absence of pre-trained models. this is in line with previous work over similar text classification tasks @xcite . ## adding a pre-trained textual encoder we next turn to a setting where we employ pretrained textual models. this not only better reflects the state-of-the-art in text classification, but also allows us to investigate the effect of images under such conditions. model as the main backbone, we employ vl-bert @xcite , which uses a transformer to combine textual inputs and image embeddings within a bert-style transformer, and has been shown to perform well on multimodal tasks. the visual embeddings are obtained from the combination of pre-trained faster r-cnn and resnet-101 @xcite , as illustrated in figure 1. for the text modality, the input consists of two parts: the document title, and the opening text of the wikipedia page in the form of the first 300 tokens. the token embeddings are obtained from a pre-trained bert model, which is fine-tuned during training. 8 the full model is plugged into a onelayer feed-forward neural network (ffnn) with a 1,024d hidden layer, and training is performed by minimizing the cross-entropy over the shinra category labels. the model predict one label for each page. for the case of multi-label inputs, we choose one randomly as the "correct" label. ## discussion and conclusion we investigated the utility of images as a supplementary input for a text classification task, and found that although images have empirical utility in traditional supervised learning, when externally pre-trained language models are utilised, any advantage from the visual modality disappears. the results were remarkably consistent across different languages and different volumes of training data. it is important to distinguish between "inherently multi-modal tasks" (e.g. vqa) and "potentially multi-modal tasks" (e.g. text classification) in drawing any broader conclusions about the (in)effectiveness of images. here, a "potentially multi-modal task" in nlp means that the primary modality is text and the task is defined based on that single data modality, but there is potentially the option to include extra modalities such as images. there remain a lot of open questions in more fully determining the (in)effectiveness of images for nlp tasks, even for text classification, such as: