Direct Metric Optimization for Image Captioning through Reward-Weighted Augmented Data Utilization (2024)

Abstract: While image captioning is an essential field of vision language models (VLM), a lack of continuity between the learning objective and final performance metrics of VLMs complicates their training and optimization. Reinforcement learning (RL) can directly optimize such metrics, but it is accompanied by a significant computational cost, making it difficult to apply to recent large-scale VLMs. In this paper, we propose Direct Metric Optimization (DMO), which is a lightweight final-metric-optimizing training method. We replace the computationally expensive exploration process in RL with an offline, diverse text data augmentation and show that self-supervised training on reward-weighted augmented data leads to direct and stable metric optimization. Our experiments demonstrate that DMO achieves performance comparable to those of the state-of-the-art RL method while saving hundreds of times more model forwarding iterations and greater amounts of computation time. This suggests that DMO constitutes a promising alternative for metric optimization in the era of large-scale VLMs.

## introduction with the advent of clip @xcite , the boundaries between vision and language modalities in machine learning have been dissolved, leading to rapid advancements in research involving these areas. furthermore, the rise of large language models (llm) has led to the emergence of large-scale vision language models (vlm), extending their influence to practical applications. for example, models such as chatgpt @xcite and gemini @xcite generate detailed natural language descriptions from visual information. with the increasing prevalence of vlms, methods for customizing and fine-tuning these models for specific domains or individuals are attracting significant interest and attention @xcite . recent standard captioning models adopt selfsupervised learning for training purposes @xcite @xcite . this method treats the ground truth captions both as inputs and labels, and the model predicts only the next token from the given image and preceding tokens. specifically, recent transformer-based encoder-decoder models can conduct the next token prediction of each step in parallel, significantly enhancing their computational efficiency. however, this approach is subject to certain limitations. typically, the performance of image captioning is evaluated using metrics such as bleu @xcite or cider @xcite ; however, self-supervised learning of language modeling does not necessarily optimize those metrics. to target final metrics directly, reinforcement learning (rl) methods have been employed @xcite @xcite . reinforce-ment learning is a powerful method capable of optimizing even non-differentiable metrics; however, it has certain drawbacks, such as learning instability and significant time and computational costs. with the growing trend of using large pre-trained models, those challenges have become increasingly serious. conducting rl with models containing billions of parameters demands extensive computational time and resources, making the application of rl methods impractical. to bypass the prohibitive computational cost of rl, we propose to replace the expensive exploration process in rl with diverse text data augmentation and reduce rl to simple importanceweighted self-supervised learning. the approach that utilizes previously collected data for rl is known as offline-rl @xcite . particularly in our approach, datasets are augmented by various methods and the augmentation diversity brings a variety of samples of different rewards, enabling the efficient estimation of the optimal caption for the image. we call this metric-optimizing self-supervised training direct metric optimization (dmo). our experiments demonstrate that dmo achieves performance on par with state-of-the-art (sota) rl methods in standard image captioning metrics while retaining lightweight computational efficiency and learning stability. this highlights dmo's significant practical advantages in metric optimization, especially considering the increasing need to tune and customize large-scale vlms. ## experiment implementations this section describes the experiment implementations for the evaluation of our dmo training. ## results we evaluate our proposed method in terms of metric optimization performance, learning stability and computational efficiency. we further investigate how reward-weighting architecture facilitates robust metric optimization by comparing dmo with standard lm training under noisy data and limited data settings. ## conclusion in this paper, we present direct metric optimization (dmo), which is a lightweight final-metricoptimizing training method. we hypothesize that diverse text augmentation can substitute the exploration in rl, and show that self-supervised training on reward-weighted augmented data leads to direct and stable metric optimization. our experiments demonstrate that dmo can directly optimize evaluation metrics across models of various architectures and parameter sizes, and stably achieves performance comparable to the sota rl method while saving hundreds of times more model forwarding iterations and greater amounts of computation time. with these practical advantages of stable and lightweight cost of tuning, dmo emerges as a new promising choice for metric optimization in the era of large-scale vlms.