Synonym relations affect object detection learned on vision-language data (2024)

Abstract: We analyze whether object detectors trained on vision-language data learn effective visual representations for synonyms. Since many current vision-language models accept user-provided textual input, we highlight the need for such models to learn feature representations that are robust to changes in how such input is provided. Specifically, we analyze changes in synonyms used to refer to objects. Here, we study object detectors trained on vision-language data and investigate how to make their performance less dependent on whether synonyms are used to refer to an object. We propose two approaches to achieve this goal: data augmentation by back-translation and class embedding enrichment. We show the promise of such approaches, reporting improved performance on synonyms from mAP@0.5=33.87% to 37.93%.

## introduction in recent years, we have witnessed increased interest in vision-language models @xcite that learn joint image and text representations in a self-supervised way, and that can later be used as building blocks for models fine-tuned on downstream tasks @xcite @xcite . in addition, recent models such as gpt-4 (openai, 2023) and dall-e 3 @xcite are built to accept image and text input provided by end users, with no set constraints on such inputs. thus, models must be robust to variations in how input is provided. we analyze how vision-language models handle the variability in textual inputs. specifically, we investigate variations in synonyms used to refer to objects. we show how such variability negatively affects performance for open-vocabulary object detection, and we propose two ways to help visionlanguage detectors learn better representations for synonyms: data augmentation by back-translation and class embedding enrichment. figure 1 : top: input to an open-vocabulary object detector: images, class embeddings, and captions; and its output: bounding boxes with associated labels. bottom: our approaches. 1) data augmentation by backtranslation: add captions back-translated from a foreign language; 2) class embedding enrichment: consider synonyms and related terms when computing class embeddings. figure 1 illustrates our proposed approaches. with back-translation, we use a machine translation model to translate captions from english to another language, and then we translate them back to english. because the back-translation is not perfect, the original caption and the back-translated one are not the same: they show changes, for instance, in which nouns are used to refer to objects (e.g., synonyms). we hypothesize that adding more synonyms and related terms to the captions used for training will help a model learn better representations for them. with class embedding enrich-ment, we modify the class embeddings that openvocabulary object detectors @xcite @xcite use to match visual embeddings learned for image regions. furthermore, when training with enriched class embeddings, we experiment with enriching them throughout the whole training process, or using a curriculum learning approach: start training with the original embeddings, and finish with the enriched ones. in both our approaches, we modify inputs to the training process (i.e., captions and class embeddings), making them generalizable to different model architectures and training strategies. we show promising results with improved performance on synonyms from map@0.5=33.87% to 37.93%. in summary, our contribution is twofold: (1) we identify an issue with current state-of-the-art (sota) vision-language object detector models (namely, difficulty in detecting objects referred to by synonyms or closely related terms), and (2) we propose two generalizable strategies to train visionlanguage object detectors to learn better representations for such related terms. concept relationships. text embeddings have been shown to encode relationships between concepts such as synonyms and antonyms @xcite . at the same time, studies on adversarial attacks have highlighted how performance of language models varies when the input is changed, even when preserving the semantic meaning of the input text @xcite @xcite . unsurprisingly, when such language models are combined with vision models, similar problems arise, with perfor-mance on vl tasks varying under perturbations of text input @xcite @xcite . our work is related to such studies since we aim to make vl models more robust to text input variations, although we differ from previous work in target task (object detection vs. visual reasoning). further, we do not require changes in how a model is trained, for instance, by defining a new loss function @xcite ; we simply modify inputs to the model, making our approach more general. ## conclusions we considered variations in nouns used to refer to objects (i.e., synonyms and related terms), and how they affect performance of object detectors trained with vision-text data. we highlighted how detecting objects when synonyms are used as input is challenging, and we introduced two approaches to ameliorate this issue, which proved successful at boosting detection performance on synonyms.