Distilled Dual-Encoder Model for Vision-Language Understanding (2022)

Abstract: On vision-language understanding (VLU) tasks, fusion-encoder vision-language models achieve superior results but sacrifice efficiency because of the simultaneous encoding of images and text. On the contrary, the dual encoder model that separately encodes images and text has the advantage in efficiency, while failing on VLU tasks due to the lack of deep cross-modal interactions. To get the best of both worlds, we propose DiDE, a framework that distills the knowledge of the fusion-encoder teacher model into the dual-encoder student model. Since the cross-modal interaction is the key to the superior performance of teacher model but is absent in the student model, we encourage the student not only to mimic the predictions of teacher, but also to calculate the cross-modal attention distributions and align with the teacher. Experimental results demonstrate that DiDE is competitive with the fusion-encoder teacher model in performance (only a 1% drop) while enjoying 4 times faster inference. Further analyses reveal that the proposed cross-modal attention distillation is crucial to the success of our framework.

## introduction vision-language understanding (vlu) tasks (e.g., visual reasoning @xcite , visual entailment @xcite , visual question answering @xcite ) require the model to understand the cross-modal interactions between images and text. various fusion-encoder vision-language pretrained models @xcite @xcite @xcite are proposed for vlu tasks. as shown in figure 1 (a), these models employ a transformer @xcite network as a cross-modal encoder to capture interactions between different modalities. despite the remarkable performance, the heavy cross-modal encoder remains an efficiency bottleneck due to the simultaneous encoding of images and text, limiting the application in practical scenarios with massive images or text. therefore, it is crucial to find an approach to accelerate inference for vlu. we turn to explore the dual-encoder visionlanguage model (shown in figure 1(b) ), which encodes images and text separately and then applies an extreme lightweight shallow module to model the interactions between modalities. the disentangled encoding paradigm enables off-line computing and caching visual or textual representations on demand, significantly lowering runtime latency. however, the shallow module is insufficient to handle complex vlu tasks, resulting in previous models @xcite falling far behind fusion-encoder models @xcite . can dual-encoder models obtain performance comparable to fusion-encoder models while preserving efficiency? in this work, we propose dide (a knowledge distillation framework for dual-encoder models), where the dual-encoder model (student) is supervised by the fusion encoder models (teacher), as shown in figure 2 . although soft label distillation @xcite is widely applied, our key observation is that cross-modal attention distributions foot_0 are absent in the student, resulting in the inability to model complex crossmodal interactions. thus, only distilling soft labels is not enough for the student to mimic the interactions of teacher deeply. considering that cross-modal interaction is critical for vlu, we introduce a plug-and-play objective cross-modal attention distillation, as finegrained supervision to help the student better learn cross-modal interactions. specifically, besides the soft label distillation, we compute the cross-modal attention of the student model and align it with the distribution in the teacher model during training. the training of dide consists of two-stage distillations. in the pre-training stage, the student learns a general initialization with distillation. in the fine-tuning stage, distillation helps the student learn more task-specific knowledge. experimental results demonstrate that dide performs competitively with the fusion-encoder teacher model in various vlu tasks (retaining 96.9% to 99.9% performance) while having a 4× inference speedup. further analysis indicates that the proposed crossmodal attention distillation yields significant gains compared to distilling only with soft labels or other latent features of the teacher. beyond vlu, dide also shows effectiveness in image-text retrieval. our contributions are summarized as follows: ## method figure 2 gives an overview of our dide framework, a knowledge distillation approach for the dual-encoder model. ## conclusion on vision-language understanding tasks, fusionencoder models obtain superior performance while sacrifice efficiency. in contrast, dual-encoder models have the advantage of efficiency, but previous models are insufficient to handle complex visionlanguage understanding. in this work, to obtain the efficiency-performance trade-off, we propose dide, a knowledge distillation framework for dual-encoder models to improve their performance on vlu tasks while retaining their efficiency. the key of dide is that we employ the cross-modal attention of a fusion encoder model as fine-grained supervision to guide the dual-encoder model for learning complex cross-modal interactions. experimental results on several vision-language understanding tasks show that our dide achieves competitive performance with a four-time speedup over the fusion-encoder teacher model. further analyses verify that distillation with cross-modal attention is critical for dual-encoder models.