Multi-stage Pre-training over Simplified Multimodal Pre-training Models (2021)

Abstract: Multimodal pre-training models, such as LXMERT, have achieved excellent results in downstream tasks. However, current pre-trained models require large amounts of training data and have huge model sizes, which make them impossible to apply in low-resource situations. How to obtain similar or even better performance than a larger model under the premise of less pre-training data and smaller model size has become an important problem. In this paper, we propose a new Multi-stage Pre-training (MSP) method, which uses information at different granularities from word, phrase to sentence in both texts and images to pre-train a model in stages. We also design several different pre-training tasks suitable for the information granularity in different stage in order to efficiently capture the diverse knowledge from a limited corpus. We take a Simplified LXMERT (LXMERT-S) which is with 45.9% parameters of the original LXMERT model and only 11.44% of the original pre-training data as the testbed of our MSP method. Experimental results show that our method achieves comparable performance to the original LXMERT model in all downstream tasks, and even outperforms the original model in Image-Text Retrieval task.

## introduction self-attention based transformer @xcite effectively overcomes the problem of rnn being difficult to run in parallel, and greatly promotes the development of large-scale pre-training models. the pre-training language models, such as bert @xcite , have achieved excellent performance in many natural language processing tasks. with their big success, researchers have also developed pre-training models on multimodal tasks. a series of multimodal pre-training models have been proposed, such as vilbert @xcite , lxmert @xcite , uniter @xcite etc., and have achieved excellent results in languagevision multimodal tasks. however, the current pre-training models are normally with large-scale parameters, require huge pre-training data and have very high demands on computational resources. for example, the gpt model @xcite has 110 million parameters, gpt-2 @xcite has 1.5 billion parameters, and @xcite has a staggering 175 billion parameters. the same is true for multimodal pre-trained models. for example, lxmert @xcite has 183.5 million parameters and requires 816 titanx gpu hours for training on 9.18 million text-image pairs. the sizes of these models are too huge for them to be deployed in many real-world scenarios. therefore, the study of lightweight pre-training models, which can achieve similar performances to largescale models with smaller parameter scales and training costs, is significantly valuable. there are some types of work on developing lightweight pre-trained models, including the design of the model structure, quantization, pruning and distillation. for example, albert @xcite ) is a lightweight model through structural design such as parameter sharing and parameter decomposition, and achieves better performance than original models; q8bert @xcite compresses the model to 1/4 of the original model but with no more than 1% performance loss by quantizing 32bit floating point into 8bit; @xcite used bert weight pruning to compress the model and found that removing a large number of attention heads would not have a major impact on the model performance; tinybert @xcite reduced the model size by 7.5 times but with no more than 4% performance loss by designing a teacher-student distillation model. all above works are on language pre-training models, and most of them concern scales of model parameters. there are few works on cutting training data and light weighing multimodal pretraining model. in fact, compared with language model, multimodal pre-training models should deal with data from both language and visual modal, which demand larger amounts of data and more computational resources. meanwhile, collections of training data are more difficult. taking for example the size of text-image pairs used for multimodal pre-training, the frequently used ms coco @xcite is a high quality dataset with only 0.82m pairs, while lait @xcite is already a big data with 10m pairs but with average quality. therefore, it is significantly valuable to develop lightweight multimodal pre-training models which can make use of limited data efficiently. existing research on curriculum learning @xcite has shown that imitating the process of human learning by gradually increasing the difficulty of a task from simple to complex in stages helps to make better use of different types of data and effectively improve the performance of learning. many models @xcite use as much as data available but few works have been done on how to arrange the tasks for better making use of limited data. we therefore borrow the idea of curriculum learning on training pre-training models. we construct a pre-training process which makes use of data from smaller units to bigger units in stages, and design appropriate pre-training tasks for each corresponding stage. specifically, we propose a new multi-stage pretraining (msp) method. the first pre-training stage is on the token units, where the text input is the category labels of the objects in the images, and the image input is the object features. an image features random shuffle (ifrs) is designed as a pre-training task for this stage. ifrs randomly shuffles the object features, and the model predicts the original object order based on the text information. the second stage focuses on phrase units. phrase-level descriptions of the image are input on the text side and image features are input on the image side. a topic of image and text for phrase (titp) task is designed for it. the third stage is sentence-based pre-training. sentence-level captions are input on the text side, and image features are input on the image side. a topic of image and text for sentence (tits) task is designed for it. we take a simplified lxmert (lxmert-s) which has fewer parameters and less pre-training data as the testbed of our msp method. experimental results show that our method achieves comparable performance to the original lxmert model in downstream tasks. the main contributions of our work are as follows: (1) we propose a new msp method that allows the model to learn different granularities of text-image correspondence information at different stages; (2) for each stage, we design pre-training tasks suitable for that stage, ifrs task for tokenbased pre-training, titp task for phrase-based pretraining, and tits task for sentence-based pretraining; (3) with less pre-trained data (11.76%), fewer model parameters (45.9%), less resource consumption (25%) and less training time (46.57%), the performances of downstream tasks are comparable to or even exceed that of the original model. ## method the overall structure of our msp method is shown in figure 1 . the pre-training process is divided into three stages based on different granularities of text-image correspondence from token, phrase to sentence. we design corresponding pre-training tasks for the three stages. we perform the above three-stage pre-training on a simplified model of lxmert (lxmert-s). the simplified process of the lxmert model is shown in figure 2 . the cross-modality encoder of lxmert-s is identical to the lxmert. we obtain the simplified lxmert (lxmert-s) by removing the object-relationship encoder and language encoder. the image features and text features are directly input to the cross-modality encoder in the lxmert-s. by removing the single modal coding layer in lxmert, the 12-layer lxmert is simplified to a 5-layer lxmert-s. the amounts of parameters in simplified lxmert-s are only 45.9% of the original model, and the whole experiment can be completed on a single gpu. the three-stage pretraining method is also fully applicable to other pre-training models. ## qualitative analysis we visualize the impact of different pre-training stages on vqa and image-text retrieval task by showing the answers probability distribution. for each example in figure 3 , the left side is the input image of the model, and the right side is the probability distribution of the top3 scoring answers in different pre-training stages. for image-text retrieval task, we select the top 1 caption for visualization. for each sample in figure 4 , the left side is the input image and the right side is the highest scoring caption predicted by the model. from both figure 3 and 4, we can find: (1) token-based pre-training (s vs @xmath0. for example, in the left sample in figure 3 and 4, the model improves its performance on downstream tasks by adding token-based pre-training that makes the model focus on object information such as horses, man and rocks in the images; ## conclusion in this paper, inspired by the idea of curriculum learning, we propose a msp method, which uses information at different granularities from word, phrase to sentence in both texts and images to pre-train a model in stages, we also design pretraining tasks suitable for each stage of pre-training, ifrs task for word-based pre-training, titp task for phrase-based pretraining, and tits task for sentence-based pretraining. experimental results on several vqa datasets as well as one cross-modal retrieval dataset show that our method achieves similar or even better performance than a larger model in terms of accuracy in all downstream tasks under the premise that the model parameters are reduced by 54.1% and the training data set is reduced by 88.24%. in future work, we will add the above training method to other simplified pretrained models to further explore the effectiveness of msp method.