VEIL : Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection (2024)

Abstract: The use of large-scale vision-language datasets is limited for object detection due to the negative impact of label noise on localization. Prior methods have shown how such large-scale datasets can be used for pretraining, which can provide initial signal for localization, but is insufficient without clean bounding-box data for at least some categories. We propose a technique to “vet” labels extracted from noisy captions, and use them for weakly-supervised object detection (WSOD), without any bounding boxes. We analyze and annotate the types of label noise in captions in our Caption Label Noise dataset, and train a classifier that predicts if an extracted label is actually present in the image or not. Our classifier generalizes across dataset boundaries and across categories. We compare the classifier to nine baselines on five datasets, and demonstrate that it can improve WSOD without label vetting by 30% (31.2 to 40.5 mAP when evaluated on PASCAL VOC). See dataset at: https://github.com/arushirai1/CLaNDataset.

## introduction freely available vision-language (vl) data has shown great promise in advancing vision tasks @xcite @xcite . unlike smaller, curated vision-language datasets like coco @xcite , captions on the web @xcite @xcite only partially describe the corresponding image, and often describe the context, which could include objects that do not appear in the image. we hypothesize this poses a greater challenge for weakly-supervised object detection (wsod) than learning cross-modal representations for image recognition (e.g. as in clip). wsod involves learning to localize objects, i.e. predict bounding box coordinates along with the corresponding semantic label, from image-level labels only (i.e. using weaker supervision than the outputs expected at test time). so, noise could compound the challenge of implicitly learning localization. wsod has primarily been applied @xcite to smaller, relatively cleaner, paid-for crowdsourced vision-language datasets like coco @xcite and flickr30k @xcite . we argue that extending wsod from paid-for captions to large-scale, in-the-wild captions is not trivial. annotators write captions that faithfully describe an image, however, web captions go beyond a descriptive relationship with their corresponding image. for example, a word can be used literally or metaphorically ("that was a piece of cake") or have multiple senses, of which only one sense is relevant to the object detection vocabulary. a caption could also share a story and include context that goes beyond the visual contents of the image; this context could mention an object name within location names or describe occluded or unpictured interactions with objects as shown in figure 1 . this richness of language is relevant as narration for the image but not as supervision for the precise localization of objects. on the visual side, useruploaded content frequently features diverse object presentations, including intriguing atypical objects, hand-drawn objects, or photos taken from within vehicles ("in my car"). we refer to image-level labels extracted from captions, that are incorrect (object not present in the corresponding image), as visually absent extracted labels (vaels). we show vaels pose a challenge for weakly-supervised object detection. to cope with this challenge, we propose veil, short for vetting extracted image labels, to directly learn whether a label is clean or not from caption context. we first extract potential labels from each caption using substring matching or exact match @xcite . we then use a transformer to predict whether each extracted label is visually present. we refer to this prediction task as extracted label vetting. we bootstrap pseudo-ground-truth visual presence labels for each extracted label or object mention using an ensemble of two pretrained object recognition models @xcite , for a variety of large-scale, noisy datasets: conceptual captions @xcite , redcaps @xcite , and sbucaps @xcite . while these models are trained on coco and similar datasets, they generalize well to estimating extracted label visual presence on in-the-wild vl datasets; however, their predictions are better used as targets for veil, rather than directly for vetting. once we vet the extracted labels, we use them to train a weakly-supervised object detector. we collect and release the caption label noise (clan) dataset with annotations on object visibility (label noise) and object appearance defects (visual noise such as atypical appearance) over three in-thewild datasets. to support using language context to filter object labels, we annotate linguistic indicators of noise that explain why an object is absent from the image but mentioned in the caption. our label vetting method outperforms nine diverse baselines, including standard cross-modal alignment prediction methods (clip), adaptive noise reduction methods, pseudo-label prediction, simple rulebased methods, and no vetting. this means veil produces cleaner wsod training data which leads to an improvement of +10 map over data cleaned using large loss matters @xcite and +3 map improvement over using clip @xcite for filtering. our findings reveal that naively combining noisy sbucaps supervision with clean labels from pascal voc-07 degrades performance (42.06 map) versus using only clean labels (43.48 map); however, vetting with veil improves performance to 51.31 map. lastly, veil's gains persist across datasets, object vocabulary, and scale. to summarize, our contributions are: 1. veil, a transformer-based extracted label, visual presence classifier, and 2. constructing the caption label noise dataset. we find that: 1. veil outperforms language-conditioned, visual-conditioned, and language-agnostic label noise correction approaches in vetting labels from a wide set of in-the-wild datasets for weakly-supervised object detection. 2. veil enables effective combination of extracted noisy and clean labels. 3. even when veil is trained on one dataset/category, but applied to another, it shows advantages over baselines. vision-language datasets include crowdsourced captions @xcite @xcite and alttext written by users to aid visually impaired readers @xcite @xcite are widely used for vision-language grounding due to abundance and high visual-text alignment. there are also large in-the-wild datasets sourced from social media like reddit @xcite and useruploaded captions for photos shared on flickr @xcite . we show the narrative element found in these in-the-wild datasets, captured by the linguistic cues we investigate, impact the ability to successfully train an object detection model. weakly-supervised object detection (wsod) is a multiple-instance learning problem to train a model to localize and classify objects from imagelevel labels @xcite @xcite @xcite . cap2det was the first work to leverage unstructured text accompanying an image for wsod by predicting pseudo imagelevel labels from captions @xcite . however, cap2det cannot operate across novel categories as it directly predicts imagelevel labels and aims to correct false negatives, not visually absent extracted labels. detic @xcite uses weak supervision from imagenet @xcite and extracts labels from conceptual captions (cc) to pretrain an open vocabulary object detection model with a clip classifier head. while these approaches succeed in leveraging relatively clean, crowdsourced datasets like coco, flickr30k and imagenet, both see lower performance in training with cc @xcite . other prior work @xcite uses a pretrained vision-language model to generate pseudo-bounding box annotations, but always requires clean data (coco), and does not explicitly study the contribution of in-the-wild datasets. vision-language pre-training for object detection. image-text grounding has been leveraged as a pretraining task for open vocabulary object detection @xcite @xcite @xcite , followed by bounding box supervision from base classes. some methods distill knowledge from existing pretrained visionlanguage grounding models like clip and align @xcite to get proposals @xcite and supervision for object detection @xcite ; however, these do not study the effect of noisy supervision in a setting without bounding box supervision. in contrast, we perform weakly-supervised object detection (wsod) using noisy image-level labels from captions only. wsod is a distinct task from openvocabulary detection and has the advantage of not requiring expensive bounding boxes. we focus on rejecting labels harmful for localization. adaptive label noise reduction in classification. adaptive methods reject or correct noisy labels ad-hoc during training. these methods exploit a network's ability to learn representations of clean labels earlier in training. this assumes there are no clear visual patterns in the noisy samples corresponding to a particular corrupted label, leading to their memorization later in training @xcite . we instead show diverse real-world datasets contain naturally occurring structured noise, where in many cases there are visual patterns to the corrupted label. large loss matters @xcite is representative of such adaptive noise reduction methods and we find that it struggles with noisy labels extracted from in-the-wild captions. ## label noise analysis and dataset we analyze what makes large in-the-wild datasets a challenging source of labels for object detection. datasets analyzed. redcaps @xcite consists of 12m reddit image-text pairs collected from a curated set of subreddits with heavy visual content. sbucaps @xcite consists of 1 million flickr photos with text descriptions written by their owners. captions were selected if at least one prepositional phrase and 2 matches with a predefined vocabulary were found. conceptual captions (cc) @xcite ) contains 3m image-alt-text pairs after heavy postprocessing: named entities in captions were hypernymized and image-text pairs were accepted if there was an overlap between google cloud vision api class predictions and the caption. extracted object labels. given a vocabulary of object classes, we extract a label for an image if there is an exact match between the object name and the corresponding caption ignoring punctuation. while this strategy will result in some noisy labels, it represents how labels are extracted in prior work @xcite due to the absence of clean annotations. using gold standard labels (defined next), we calculate the precision of the extracted labels. in-the-wild datasets exhibit much lower extracted label precision, with sbu-caps at 0.463, redcaps at 0.596, and cc at 0.737, in stark contrast to coco's 0.948 (refer to tab. 12 for no-vetting precision). gold standard object labels. we use imagelevel predictions from a pretrained image recognition model to estimate visual presence gold standard labels (pseudo-ground-truth) because in-thewild datasets do not have object annotations. we use an object recognition ensemble with the x152-c4 object-attribute model @xcite and ultralytic yolov5-xl @xcite . this ensemble achieves strong accuracy, 82.2% on sbu-caps, 85.6% on redcaps, and 86.8% on cc (see appendix sec. a.1: we annotate a subset to estimate accuracy). for our analysis of visually absent extracted labels (vael), we sample image-caption pairs where the extracted label and gold standard label disagree. note we never use bounding-box pseudo labels, only image-level ones. caption label noise (clan) dataset annotations collected. to understand the label noise distribution, we select 100 vael examples per dataset (redcaps, sbucaps, cc) and annotate four types of information (abbreviations are underlined): two authors provide the annotations, with cohen's kappa agreements of 0.76 for q1, 0.33 for q2, 0.45 for q3, and 0.58 for q4. we calculate cohen's kappa for each option and compute a weighted average for each question, with weights derived from average option counts across annotators and the three datasets. we compute the average disagreement as the number of disagreements divided by the number of samples annotated for each question per dataset, averaged over all datasets. the average disagreement is 25.1% for q2, 25.3% for q3, 14.6% for q4. when comparing similar context ("co-occ" or "sim") vs "no similar context" for q2 and any defects ("occl", "parts", "atyp") vs "no defects" for q3, disagreement is 28.7% for q2, 17.0% for q3. the disagreements are fairly low. in statistics: label noise. we first characterize the visibility of objects flagged as vaels by the recognition ensemble. sbucaps has the highest rate of completely absent images (58.5%), followed closely by redcaps. sbucaps also has the highest rate of partially visible objects (20%). cc has the highest full visibility (32.8%), defined as the object having 75% or more visibility from a given viewpoint. samples with absent and partially-visible objects constitute poor training data for wsod, and their high rate motivates our veil approach. statistics: similar context. certain images with absent objects may be more harmful than others. prior work shows that models exploit cooccurrences between an object and its context to do recognition, but when this context is absent, performance drops @xcite . we hypothesize that including images without the actual object and with this contextual bias could hurt localization when supervising detection implicitly. additionally, semantically similar objects may blur decision boundaries. different annotators may have different references for similarity or co-occurrence frequency, but our annotators achieve fair agreement (κ = 0.33). in statistics: visual defects. we hypothesize there may be visual defects that caused the recognition ensemble to miss fully visible objects. here, we compute the percent of at least one visual defect in fully or partially visible samples: 79% for cc, 87% for sbucaps, and 69% for redcaps. tab. 1 has the distribution by visual defect type; this shows that atypical appearance is the most common defect for redcaps and cc (49% and 57.3%). we argue atypical examples constitute poor training data for wsod, especially when learning from scratch. the caption context (e.g. "acrylic illustration of the funny mouse") may indicate the possibility of a visual defect, further motivating the veil design. statistics: linguistic indicators. noun modifiers are frequently occurring indicators over all datasets. prepositional phrases are significant in sbucaps (40.5%) and cc (31.3%). need for caption context in vetting is motivated by many vaels being mentioned in contexts going beyond the image, e.g.: "just got back from the river. friend sank his truck pulling his boat out. long story short, rip this beast" (redcaps). we find prevalent structured noise (pattern to the images associated with a particular noisy label) for indicators like "noun modifier" and "prepositional phrase" due to high levels of occlusion and similar contexts. while using pretrained object recognition models may appear unfair, bootstrapping this knowledge to train a language model to predict token-level binary visual presence has efficiency benefits (no image input required), can generalize to extracted labels outside of the recognition model's vocabulary (see sec. 5 for generalization experiments), and is realistic for wsod, since detection labels are more limited, whereas many recognition labels exist. during inference, if an extracted label was mapped to multiple tokens (e.g. "teddy bear"), the predicted scores are averaged to a single prediction. ## experiments we show the ability of veil to vet noisy extracted labels, remove structured noise, and outperform language-agnostic filtering and image-based filtering methods. we test generalization ability in veil through cross-dataset and cross-category experiments. lastly, we evaluate weakly-supervised object detection settings using only noisy supervision and a combination of noisy and clean supervision. ## conclusion we released the caption label noise (clan) dataset where we annotated types of visually absent extracted labels and linguistic indicators of noise in 300 image-caption pairs from three in-the-wild datasets. using clan, we find that caption context can be used to vet (filter) extracted labels from caption context. we proposed veil, a lightweight text model which is trained to predict visual presence using pseudo labels sourced from two pretrained models for recognition. veil outperformed nine baselines representative of current noise filtering techniques that could be adapted for captions. we demonstrate three key findings specific to vetting for wsod: (1) there is a distinct advantage in learning to filter as opposed to filtering using pseudo-ground truth visual presence labels;