Vision-Language Pretraining: Current Trends and the Future (2022)

Abstract: In the last few years, there has been an increased interest in building multimodal (vision-language) models that are pretrained on larger but noisier datasets where the two modalities (e.g., image and text) loosely correspond to each other (e.g., Lu et al., 2019; Radford et al., 2021). Given a task (such as visual question answering), these models are then often fine-tuned on task-specific supervised datasets. (e.g., Lu et al., 2019; Chen et al.,2020; Tan and Bansal, 2019; Li et al., 2020a,b). In addition to the larger pretraining datasets, the transformer architecture (Vaswani et al., 2017) and in particular self-attention applied to two modalities are responsible for the impressive performance of the recent pretrained models on downstream tasks (Hendricks et al., 2021). In this tutorial, we focus on recent vision-language pretraining paradigms. Our goal is to first provide the background on image–language datasets, benchmarks, and modeling innovations before the multimodal pretraining area. Next we discuss the different family of models used for vision-language pretraining, highlighting their strengths and shortcomings. Finally, we discuss the limits of vision-language pretraining through statistical learning, and the need for alternative approaches such as causal representation learning.

## description in the last few years, there has been an increased interest in building multimodal (vision-language) models that are pretrained on larger but noisier datasets where the two modalities (e.g., image and text) loosely correspond to each other (e.g., @xcite . given a task (such as visual question answering), these models are then often fine-tuned on task-specific supervised datasets. (e.g., @xcite @xcite . in addition to the larger pretraining datasets, the transformer architecture @xcite and in particular self-attention applied to two modalities are responsible for the impressive performance of the recent pretrianed models on downstream tasks @xcite . this approach is appealing for a few reasons: first, the pretraining datasets are often automatically curated from the web, providing huge datasets with negligible collection costs. second, we can train large models once, and reuse them for various tasks. finally, these pretraining approach performs better or on par to previous task-specific models. an interesting question is whether these pretrained models -in addition to their good task performance -learn representations that are better at capturing the alignments between the two modalities. in this tutorial, we focus on recent visionlanguage pretraining paradigms. our goal is to first provide the background on image-language datasets, benchmarks, and modeling innovations before the multimodal pretraining area. next we discuss the different family of models used for vision-language pretraining, highlighting their strengths and shortcomings. finally, we discuss the limits of vision-language pretraining through statistical learning, and the need for alternative approaches such as causal modeling. we believe that the computational linguistics (cl) community will benefit from this tutorial in multiple ways. language grounding research often uses or evaluates the most successful visionlanguage approaches. better understanding of the shortcomings and strengths of these approacheswhich we hope our tutorial provides -will pave the way for building stronger language grounding agents. moreover, vision-language pretraining has been inspired by its parallel in pretraining language models. as a result, the cl community has a special role in thinking about the future of visionlanguage approaches using lessons learned from language pretraining. ## type of the tutorial this is a cutting-edge tutorial focusing on discussing the new trends in vision-language pretraining: if recent models result in better representations and how they contribute to downstream tasks. we plan to mostly discuss recent papers from 2018 and after but will also include influential papers from before 2018 that have played a crucial role in the current vision-language paradigms. ## target audience we expect the target audience to be researchers interested in the intersection of vision and language, such as the language grounding or grounded communication researchers. this tutorial is also of interest for junior students who are starting their career. familiarity with recent architectures such as transformers is a useful but not needed for attending the tutorial. ## breadth of the tutorial we will mainly cover other people's work (as outlined in §4 and §7). more specifically, we expect the tutorial to include less than 15% of instructors' work -speakers will spend at most 10 minutes presenting their prior work. ## diversity considerations we are planning to increase diversity in a few ways: first, the topic of the tutorial is multidisciplinary bringing together researchers from diverse backgrounds (such as language, vision, and representation learning). we also plan to discuss how vision-language pretraining can benefit multilingual applications through grounding multiple languages into vision. second, the instructors are from diverse backgrounds including their career stage (mid-career / junior), geography, gender, as well as their institution (academia / industry). third, we will share our reading list, slides, and the recording of the talk publicly for people who cannot attend the conference in person, and also as a resource for junior researchers who are starting their career. ## reading list for his work at the intersection of computer vision, machine learning, and natural language processing. he was part of the team that won the visual question answering challenge at cvpr 2017, which introduced the bottom-up/top-down attention mechanisms that are now ubiquitous for vision and language. his current research focuses on out-of-distribution generalization and learning methods inspired by causal reasoning. he has given multiple introductory talks on these topics and is a regular invited speaker at workshops and seminars on vision and language (e.g., vqa workshop at cvpr 2021, vision and language workshop at accv 2018). aida nematzadeh [webpage: http://www . aidanematzadeh.me] is a staff research scientist at deepmind. her research interests are in the intersection of computational linguistics, cognitive science, and machine learning. her recent work has focused on multimodal learning and evaluation and analysis of neural representations. she co-instructed a tutorial on "language learning and processing in people and machines" at naacl 2019, and has given numerous invited talks (see http://aidanematzadeh . vision-language systems have many potential applications beneficial for society: but like most other technology, such visionlanguage systems could also be used for potentially harmful applications such as: in addition to the above potentially harmful applications of vision-language systems, there exist ethical concerns around fairness and bias. the vision-language models, as other deep learning based models @xcite , could potentially amplify the biases present in the data they are trained on. since the training data (images and language) captures stereotypical biases present in the society (e.g, the activity of cooking is more likely to be performed by a woman than a man), amplification of such stereotypes by vision-language systems is concerning as it has the potential to harm the users in the relevant groups (based on gender, race, religion etc.) by entrenching existing stereotypes and producing demeaning portrayals @xcite . to raise awareness about such ethical concerns and to promote discussions among researchers, the last part of the tutorial ("beyond statistical learning in vision-language") will focus on such shortcomings of existing models and we will discuss some methods that aim to tackle some of these challenges.