Improving Visual-Semantic Embedding with Adaptive Pooling and Optimization Objective (2023)

Abstract: Visual-Semantic Embedding (VSE) aims to learn an embedding space where related visual and semantic instances are close to each other. Recent VSE models tend to design complex structures to pool visual and semantic features into fixed-length vectors and use hard triplet loss for optimization. However, we find that: (1) combining simple pooling methods is no worse than these sophisticated methods; and (2) only considering the most difficult-to-distinguish negative sample leads to slow convergence and poor Recall@K improvement. To this end, we propose an adaptive pooling strategy that allows the model to learn how to aggregate features through a combination of simple pooling methods. We also introduce a strategy to dynamically select a group of negative samples to make the optimization converge faster and perform better. Experimental results on Flickr30K and MS-COCO demonstrate that a standard VSE using our pooling and optimization strategies outperforms current state-of-the-art systems (at least 1.0% on the metrics of recall) in image-to-text and text-to-image retrieval. Source code of our experiments is available at https://github.com/96-Zachary/vse_2ad .

## introduction visual semantic embedding (vse) @xcite is a representation learning method that embeds images and texts for efficient cross-modal retrieval, and typically has the following steps: (1) the image and text features are first extracted by separate visual and text encoders; for feature aggregation, the most commonly used methods are simple pooling aggregators. maxpool @xcite and meanpool @xcite are designed to detect the salient and mean points of features, and k-maxpool @xcite extracts the mean of top-k features. some complex aggregation techniques have been proposed, e.g. localimportance projection @xcite , sequenceto-sequence encoder @xcite , graph convolution network @xcite , exponential adaptive pooling @xcite and selfattention encoder @xcite . however, we found that carefully selected pooling functions can surpass complex methods (see @xcite . motivated by this, our paper proposes an approach that can automatically discover the best pooling functions. specifically, we seek to improve the feature aggregation step by proposing a formulation that parameterizes the different pooling strategies and allows the model to learn the best configuration automatically via its objective, alleviating the need to do manual tuning. in other words, we've turned these hyper-parameters (i.e. choices of pooling functions) into parameters in the model. for optimization, most vse models optimize using the hinge triplet ranking loss with in-batch negative example @xcite . the intuition of the objective is to encourage positive pairs to be embedded in a similar space while widening the distance between a target with the hardest in-batch negative sample @xcite . in practice, however, it is often difficult for the model to find a good negative sample in the early stages of training (as instances are randomly distributed in space), resulting in slow convergence (see appendix a.2). to improve optimization, we propose an adaptive optimization objective that selects multiple in-batch negative samples based on model quality during training. the intuition is that in the early stages of training, we want to sample more negative samples for faster convergence, and in the later stages, we want to sample fewer negative samples for finer tuning. over two public datasets, ms-coco @xcite and flickr30k @xcite , we show that a standard vse model using our proposed feature aggregation and optimization strategies outperforms benchmark models substantially. in particular, our method obtains 1.4% relative gains on rsum for ms-coco and 1.0% for flickr30k. compared with a pre-trained visionlanguage model with similar performance @xcite , our method is 4.3× faster. ## related work depending on whether the image and text features have any form of cross-modal interaction before similarity calculation, existing image-text retrieval can be broadly categorized into two types. the visual semantic embedding (vse) @xcite @xcite methods process the multimodal instances independently before projecting them into a joint embedding space for similarity matching. @xcite design a two-branch neural networks, liwe @xcite considers character-based alignment and embedding methods for language encoder and faghri et al. @xcite extend it by using hard triplet loss for optimization. following these ideas, pvse @xcite and cvse @xcite are proposed to consider intra-modal polysemous and consensus information. recently, @xcite samples instances as probabilistic distributions and achieves further improvement. these vse-based methods are fast as they do not consider cross-modal interaction and as such the visual and text features can be pre-computed. the non-vse methods concentrate on the interaction of modalities. for example, prior methods explore fusing multi-modal information by attention @xcite @xcite , alignment @xcite , multi-view representation @xcite or finegrained reasoning @xcite . the early-interaction methods @xcite , like pre-trained vision-language models @xcite @xcite @xcite , focus on maximizing performance while trading off efficiency. the performance of vse ultimately depends on the quality of the joint embedding space, which is usually learned with simple transformations (e.g. linear projection or multi-layer perceptron) and pooling aggregators (e.g. mean pooling @xcite , max pooling @xcite , or a combination of them @xcite ). compared to these simple aggregation methods, more complex aggregators that introduce a large number of trainable parameters have also been explored, e.g. intermodal attention @xcite and selfattention mechanisms @xcite . @xcite design a cross-modal guided pooling module that attends to local information dynamically. these sophisticated aggregators typically require more time, and don't always outperform simple pooling strategies. perhaps the closest study to our work is gpo (vse∞) @xcite , which builds a generalized operator to learn the best pooling strategy that only considers the position information of the extracted features. some studies focus on improving the optimization objective, and the most widely adopted objective is the hinge-based hard triplet ranking loss @xcite @xcite , which dynamically selects the "hardest" negative sample within a mini-batch. other studies explore solutions that choose multiple negative samples. @xcite introduce a coherence metric to rank the "irrelevant" candidates. extending the idea, @xcite assign different weights for positive and negative pairs. to tackle the issue of noisy labels which impacts multimodal representation, @xcite propose maximizing the mutual information between different modalities. @xcite separate data into "clean" and "noisy" partitions by co-teaching. ultimate though these methods did not explore modifying the behaviour of negative sample selection adaptively according to model performance during training, which we address in this work. inputs all three explosions being audible within the stadium feature extraction feature aggregation convnet seqmodel similarity calculation optimization target vit regions or grids patches plm … … … … all three explosions being audible within the stadium tokens maxpool meanpool k-maxpool aggregate & normalize x cosine joint embedding space anchor text anchor image negative image negative image figure 2 : the framework of vse. the visual and text encoders process the image and text separately at first. the related images and sentences are then directed to a similar space using an appropriate optimization target. ## ms-coco 5-fold 1k test image retrieval caption retrieval vf method metrics. we evaluate cross-modal retrieval performance using recall@k (r@k), where @xmath0, 5, 10}. the evaluation tasks include both caption retrieval (given caption, find images) and image retrieval (given image, find captions). we also compute rsum which is a sum of all r@k metrics across both tasks to assess the overall performance. ## conclusion and discussion in this paper, we propose methods to improve vse's feature aggregation and optimization objective. for the former, we introduce a way to parameterize the aggregation function to allow the visual and text encoders to learn the best way to combine their features to produce a fixed-size embedding. for the latter, we propose a method that dynamically selects many negative samples that allow the vse to converge faster with a better final performance. we compare our enhanced vse model to several baselines and state-of-the-art models over two public datasets and demonstrate that it marries both performance (state-of-the-art retrieval results) and efficiency (orders of magnitude faster than pretrained models). in terms of practical impact, we also tested a real-world application of our method in dianping (a chinese searching platform) for a unimodal retrieval task (semantic textual similarity) and found that it worked very well, demonstrating its applicability to real-world applications. a boston terrier is running on lush green grass in front of a white fence. a boston terrier is running in the grass. a dog runs on the green grass near a wooden fence. a black and white dog is running in a grassy garden surrounded by a white fence. a dog running though a grassy field. a man with gauges and glasses is wearing a blitz hat. the man with pierced ears is wearing glasses and an orange hat. a man wears an orange hat and glasses. a man with glasses is wearing a beer can crocheted hat. a man in an orange hat starring at something. top-1 top-2 top-3 top-4 top-5 a young female student performing a downward kick to break a board held by her karate instructor. top-2 top-1 top-3 top-4 top-5