Towards Multi-Modal Text-Image Retrieval to improve Human Reading (2021)

Abstract: In primary school, children’s books, as well as in modern language learning apps, multi-modal learning strategies like illustrations of terms and phrases are used to support reading comprehension. Also, several studies in educational psychology suggest that integrating cross-modal information will improve reading comprehension. We claim that state-of- he-art multi-modal transformers, which could be used in a language learner context to improve human reading, will perform poorly because of the short and relatively simple textual data those models are trained with. To prove our hypotheses, we collected a new multi-modal image-retrieval dataset based on data from Wikipedia. In an in-depth data analysis, we highlight the differences between our dataset and other popular datasets. Additionally, we evaluate several state-of-the-art multi-modal transformers on text-image retrieval on our dataset and analyze their meager results, which verify our claims.

## introduction when we were babies, we learned our native language by combining our parents' words and visual hints. in primary school, children's books, as well as in modern language learning apps, like babble 1 or duolingo 2 , this multi-modal learning strategy continues as illustrations of terms and phrases are used to support reading comprehension also, multiple studies in educational psychology suggest that integrating cross-modal information will improve learning to read @xcite @xcite @xcite @xcite . this paper presents initial research towards leveraging machine learning technology within a language learner context to improve human reading. the most popular training datasets for current models applied on text-image retrieval are ms coco @xcite and flickr30k @xcite . both datasets were created by crowdsourcing workers with the task to find short, simple and descriptive captions for images taken from flickr foot_0 . we argue that sentences slightly advanced language learners might not comprehend are presumably more complex than the captions from coco or flickr30k. hence we further claim that current models will perform poorly on more complex data. the contributions of this work to verify these hypotheses are: a) the collection of a multi-modal dataset based on wikicaps @xcite , which we call wismir (wikicaps subset for multi-modal image-retrieval); b) an in-depth analysis and comparison of wismir to other multimodal datasets used for image-retrieval; c) a textimage retrieval evaluation of state-of-the-art imageretrieval models on wismir. ## related work during the last few years, there were significant breakthroughs in various computer vision tasks and models @xcite as well as in the field of natural language processing. especially with the recent dawn of transformers, models are increasingly capable of understanding text's semantics @xcite @xcite . this progress of uni-modal models also led to a great leap forward in multi-modal visio-linguistic models, which are starting to leverage the power of transformers to work with text and images simultaneously. one of the several multi-modal tasks where these models pushed the boundaries is text-image retrieval, which we want to make use of in our language learner scenario. for this task, the model learns a metric function φ k,l : r |s k |×|i l | → [0, 1] that measures the similarity of sentence s k and image i l . the goal is to find the best matching image i @xmath0,l for a query sentence q from a pool of images p , the similarity scores {φ q,l | @xmath1. the input to multi-modal transformers applied on text-image retrieval are textual tokens of a sentence s k together with region features of an image i l . usually, textual tokens are generated by pre-trained bert tokenizers @xcite . the visual region features are typically computed by pre-trained region and object detection and classification networks such as faster-r-cnn with @xcite @xcite . multi-modal transformer networks can be grouped into so-called "early-fusion" models and "late-fusion" models. in early-fusion models such as uniter @xcite , oscar @xcite , imagebert @xcite , visualbert @xcite , or vl-bert @xcite , tokens of both modalities form the input to the network. self-attention heads in the transformerencoder layers @xcite then compute joint-representations of both modalities, i.e., fine-grained word-region-alignments of the words @xmath2. the similarity function φ k,l is an arbitrary combination of those joint-representations that depends on the respective model. despite their remarkable performance of tasks on typical datasets like coco or flickr30k, early-fusion models are not applicable in real-world information retrieval systems with large pool of images because it would require tremendous computational power and is therefore infeasible in time-critical applications. as opposed to early-fusion models, in latefusion models, the textual and visual modalities get forwarded through separate transformers for each modality. later, the output of the textual transformer and the output of the visual transformer get fused depending on the model's specific implementation. for example, lxmert @xcite and vilbert @xcite ) compute the fused cross-modality output with a third crossmodal transformer that takes the separate and uni-modal transformers' outputs as inputs. other latefusion models specially designed to solve multimodal retrieval tasks like tern @xcite and @xcite use a more computationally efficient way. a significant advantage of late-fusion models over early-fusion models is that the output embeddings of the uni-modal transformers can be pre-computed and indexed. in real-world applications, with a large pool of images, this can save enormous amounts of time. by pre-computing the image embeddings, only the query embedding q has to be computed to measure the similarities φ q,l between the query q and all images i @xmath3. especially in tern and teran, where the multi-modal fusion is not a complex neural network, this leads to short latency and enables real-world multi-modal information retrieval systems. ## dataset collection the most popular datasets for pre-training and fine-tuning multi-modal transformers applied on retrieval tasks are ms coco and flickr30k. both are hand-crafted datasets, with short, descriptive and conceptual captions created by crowdsourcing workers describing mostly non-iconic images from flickr. within a language learner scenario, we argue that the sentences a user does not understand while reading are presumably more complex than the short and relatively simple caption sentences from coco or flickr30k. consequently, we claim that models trained with this data will perform much worse on more complex textual data. an example of a multi-modal dataset containing non-constrained and heterogeneous text-image pairs is wikicaps @xcite , which contains about 3.8 million images and their respective english captions from wikipedia articles. they are non-constrained, because the captions and the images of wikicaps are not the outcomes of a particular (crowdsourcing) task, i.e., there are no constraints besides the terms of use of wikipedia. they are heterogeneous, since the data of wikicaps was randomly crawled from wikipedia and does not follow any pattern, as opposed to coco or flickr30k images and captions, which got carefully collected according to several sophisticated rules. the authors of wikicaps only provide a tabseparated file containing the wikimedia file ids of the image and the respective caption together with a perl script to download the images serially. to make the data more accessible, we developed an efficient python application, which we released on github foot_1 . this tool is capable of collecting corpus statistics based on the captions using different models and frameworks, flexibly filtering the data with user-defined filters, downloading the images in parallel, applying customizable transformations to the images, and finally, persisting the data in an easy to use and efficient format. using the tool, we collected and released two proof-of-concept versions of the dataset, which we refer to as (wikicaps subset for multi-modal information retrieval) version 1 and 2. after inspecting the meager results of several models trained and evaluated on the first version of wismir, we decided to collect the second version containing about twice as many textimage pairs in the training set to check if more data improves model performances. information about the size of the dataset is listed in ## model evaluations in order to verify our claim that models pre-trained solely on coco and flickr30k perform poorly on text-image retrieval with more complex and heterogeneous data like wismir, we trained and evaluated several models on our datasets. ## conclusion in this paper, we verify our claim that state-of-theart multi-modal transformers for image-retrieval, which are pre-trained on common datasets like coco and flickr30k, cannot generalize well on more complex textual data. therefore, we collected a multi-modal image-retrieval dataset based on data from wikipedia, wismir, and conducted several data analysis experiments that underline its differences to coco and flickr30k. additionally, we evaluated two state-of-the-art multi-modal transformers on text-image retrieval on this novel dataset to verify our claim. we discovered significant problems the evaluated models have with our dataset and in the dataset itself, which we will address in future work.