What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs
Anonymous Author(s)
AffiliationAddress email
Abstract
Given an input image, and nothing else, our method returns the bounding boxes ofobjects in the image and phrases that describe the objects.
This is achieved withinan open world paradigm, in which the objects in the input image may not havebeen encountered during the training of the localization mechanism.
Moreover,training takes place in a weakly supervised setting, where no bounding boxes areprovided.
To achieve this, our method combines two pre-trained networks:
the
CLIP image-to-text matching score and the BLIP image captioning tool.
Trainingtakes place on COCO images and their captions and is based on CLIP.
Then,during inference, BLIP is used to generate a hypothesis regarding various regionsof the current image.
Our work generalizes weakly supervised segmentation andphrase grounding and is shown empirically to outperform the state of the art inboth domains.
It also shows very convincing results in the novel task of weaklysupervised open-world purely visual phrase-grounding presented in our work.
Forexample, on the datasets used for benchmarking phrase-grounding, our methodresults in a very modest degradation in comparison to methods that employ humancaptions as an additional input.
Introduction
What does it mean, to see?
The plain man’s answer (and Aristotle’s too) would be, to know what is where by looking.
In other words, vision is the process of discovering from images what is present in the world, and where it is.
Marr [50]
We address the task of detecting the objects that exist in a given image, without limiting ourselves toa predefined list of objects, of the type that detection algorithms employ.
This property is sometimesreferred to as open-world vision.
Our method assigns a text phrase to each detected object.
Thesimilar task that employs an additional text input describing the image is called phrase-grounding.
We describe our approach as “purely visual” to denote that there is no text input.
Since the taskis performed without the use of bounding boxes in any phase of training, the level of supervisionemployed corresponds to weakly supervised.
As far as we can ascertain, the task of weakly-supervisedopen-world purely visual phrase-grounding is entirely novel.
For short, we refer to this task, which isillustrated in Fig 1, as “what is where by looking” (WWbL).
In order to tackle this task we rely on two recent pretrained vision-language models: CLIP [60] and
BLIP [41].
These models are trained on large corpora of matching images and captions and canproduce a matching score between an input image and an input text.
Trained on millions of such pairs,
CLIP has been extensively used in zero-shot recognition [94], image generation [21, 25, 45], imageediting and manipulation [57, 11, 24, 95], image and video captioning [53, 72, 73] and other tasks[52].
BLIP, a more recent addition, also incorporates into its training phase the ability to generate
[Table]
✓
[Table]
✓ ✓ ✗
What is where by looking (WWbL)
This work ✓ ✓ ✓ image captions.
It also relies on a bootstrapped captioning model to generate matching sentences andon a bootstrapped filtering model to eliminate false matches from its training set.
While BLIP can produce a matching score, we choose to use CLIP for producing such a score and
BLIP for generating plausible captions.
This modular choice allows us to study the matching-basedcomponents (such as generalizing weakly supervised localization methods) separately from the
WWbL. It also emphasizes the modular nature of our method.
Given time and resources, we wouldbe able to evaluate the method using only BLIP or replacing BLIP with CLIP-based image captioningmethods, which are considerably slower than BLIP [73].
Our WWbL method comprises two phases.
During training, an encoder-decoder g is applied toextract a foreground mask for an input image, given an image caption.
This is the same level ofsupervision that is required to train CLIP or BLIP, albeit performed on a much smaller dataset.
Sinceno segmentation mask (or bounding box) is used, this is considered weakly supervised.
Four loss terms are used.
The first one requires that the CLIP matching score of the foreground regionand the matching text be maximal.
The second helps ensure that the background region is unrelatedto the text.
The third term considers the explainability map of CLIP, given the input image and thecaption [13] as a guide for the foreground mask.
Lastly, in order to encourage compact foregroundmasks, a regularization term is added.
A two-stage inference procedure employs the learned foreground segmentation network g and apretrained-BLIP model.
The first stage extracts object proposals using a Selective Search algorithm [74].
The second stage uses BLIP to generate captions for each proposal and find text clustersin CLIP’s embedding space, in order to avoid repetitive descriptions.
Network g is then used to findthe foreground mask for each sentence, given the entire image as input and the sentence.
Our results show that network g , as a weakly supervised localization network, outperforms thestate-of-the-art.
The same model also outperforms the state-of-the-art in phrase grounding.
When theentire method is applied, with the two stages described above, the results obtained for WWbL are onpar with the results obtained using human captions for the task of phrase grounding.
These results areconsiderably better than those obtained by the baselines.
Our main contributions are: (i)
A new state-of-the-art solution for weakly-supervised localizationtasks with text and image as input, and (ii) Tackling, for the first time, as far as we can ascertain, thefundamental WWbL task, which is an open-world detection task that does not rely on any localizationinformation during training.
Related Work
Open-world computer vision is an emerging paradigm that stands in contrast to recognition systemsthat can only address the limited number of classes and types of examples that were observed duringtraining.
Specifically, open-set classification [7, 86, 38] requires the ability to understand when a newsample is not from the observed classes.
Since large visual-language models are trained on hundreds of millions of samples [60], many classesof objects are observed during training.
It is impractical to enumerate all object classes.
Moreover,these models can recognize phrases that were unseen during training, e.g., “a green dove”.
Therefore,the use of the term "open-world" for such models is not based on learned vs. unseen classes, butrather on testing the method without limiting the type of objects observed during the inference.
Open-world localization methods generalize more restricted methods, which rely on a relativelyshort list of detectable objects.
Similarly, weakly supervised methods, which employ labeling at theimage level, generalize fully supervised methods that require bounding boxes during training.
Finally,methods that do not receive as input guiding information regarding the content of the image (we callthese “purely visual”) generalize methods that receive as input a list of objects to detect or text thatdescribes the contents of the image.
Tab. 1 compares the task we tackle to other localization tasks.
As can be seen, the task we tackle-“what is where by looking” (WWbL) - generalizes all other localization tasks.
Unlike supervisedobject detection, it is open-set and weakly supervised.
Weakly supervised localization methods arenot open-set.
Lastly, phrase grounding methods employ a textual input, while WWbL does not.
Object Detection
One of the fundamental tasks of computer vision is object detection, where aclosed set of object categories is localized based on training sets that contain costly bounding-boxannotation [43, 20].
Since the advent of deep learning, many fully supervised algorithms have beenproposed for this task [61, 26, 44].
In contrast to these methods, our approach employs image-levelannotation (global text description) and an open dictionary to describe the detected object.
Phrase
Grounding is a weakly supervised phrase localization task, in which text phrases areassociated with local image annotations and local human-annotation signal [2, 89, 33].
Manymethods extract text embedding from a pretrained language model [16, 63, 35], together with imagerepresentations, to obtain a common semantic space for image-text pairs.
Our model is based on
CLIP [60] as a text-image relation space to extract the localization of objects.
Li et al.
[42] collected27M image-text data points, of which 3M have local human annotation, and used a semi-supervisedtraining method for phrase-grounding, which employs CLIP.
It cannot be compared directly withour work, since it uses additional supervision.
Arbelle et al.
[3] proposed a self-supervision methodscheme for generating a localization map.
Weakly Supervised Localization (WSOL)
Class Activation Map (CAM) explainability methodshave been offered in recent years for solving WSOL tasks [92, 93, 59, 46].
Most of these algorithmstrain a classifier to distinguish between sub-categories of the main object (Birds, Cars, Dog etc),employing a localization loss term for the explainability map [80, 82, 51, 28, 55].
Similarly to us, Shaharabany et al.
[66] employ an external localization network in addition to theclassifier.
During training, the classifier supplies gradients to the localization network.
In contrast toour work: (1) they compare the classifier outputs with and without the foreground mask, while ouralgorithm compares the masked image with the text in the CLIP space, (2) there is no backgroundloss term, and (3) explainability is not used as a signal.
Image Captioning
Captioning is a fundamental vision-language task.
Early methods applied
RNNs [49, 37].
Attention was added to identify relevant salient objects [83, 64].
Subsequently,
Transformers modeled interactions among all image elements with self-attention [75, 56, 22].
Recentworks have shown significant improvements in robustness and generalization by using large-scalevision-language data sets [73, 53, 41, 34, 77].
Our method employs BLIP [41], which is a visionlanguage pre-training framework that is trained on images and their caption, employing a methodto filter out noisy data.
The framework also provides an image captioning model and shows goodperformances in diverse image domains and for a large variety of real-world objects.
Explainability
Many methods generate a heatmap that indicates relevancy for CNNs, e.g., [65, 8,47, 67].
Most relevant to our work is the use of GradCAM [65] relevancy maps as a cue for weaklysupervised segmentation methods [54, 79], including recent contributions that employ GradCAMheatmaps as pseudo labels for weakly supervised localization and segmentation algorithms[78, 28].
The literature on transformer explainability is sparse and is mostly limited to pure self-attentionarchitectures [1, 12].
A recent work by Chefer et al.
[12] is the first to fully address bi-modaltransformer networks, and is the method employed to obtain relevancy maps.
The solution is based on
Layer-wise Relevance Propagation [5], with gradient integration for the self-attention and co-attentionlayers.
[Figure]
Method
In this section, we describe our method for generating a localization map based on image andtext input signals without local annotation.
We use our method on three different tasks (i)
Weaklysupervised object localization (WSOL) (ii) Weakly supervised phrase-grounding (WSG) (iii)
What iswhere by looking (WWbL).
At the core of our method lies a segmentation network g , which is trained in a weakly supervisedmanner.
The network g has two inputs: an input image I ∈ R 3 × W × H , and an embedding of the inputtext Z T .
For the simpler task of WSOL, network g receives only the first input I as input signal.
Theloss terms used in both cases are the same, and, for the sake of generality we write the equations withg that has two inputs.
The training of g does not involve any ground truth localization information.
Instead, it relies on twotypes of signals: (1) comparing the CLIP score of the extracted segment to that of the entire image,and (2) including the heatmap of the CLIP network, as obtained with the relevancy method of Cheferet al.
[14], in the segmented region.
Four-loss terms are used, using the input image I , the text t , and the relevancy heat-map of CLIP given
I and t , which we denote as H .
The terms are (1) a foreground loss L fore ( I, t ) (2) a backgroundloss L back ( I, t ) , (3) a relevancy heatmap loss L rmap ( I, H ) , and (4) a regularization loss L reg ( I, g ) .
The foreground loss encourages the segmentation network g to output a map that increases thesimilarity between the text and the masked image g
( I, t ) ⊙ I , ⊙ being the Hadamard product.
[Equation]
where CLIP ( J, s ) denotes the CLIP matching score between an image J and the text s .
The background loss helps ensure that the complementary part to that returned by g ( I, Z T ) isirrelevant, in the sense that its CLIP matching score with the text is low.
[Equation]
The relevancy map H obtained by the method of Chefer et al.
[14] on CLIP ( I, t ) provides us withanother important cue.
While it is not entirely reliable in the sense that irrelevant regions are oftenmarked and those regions often do not include the entire relevant part, we find that it often highlightssome of the relevant parts.
Since both H and g ( I, Z T ) have the same range of values ([0,1]), weemploy the squared Euclidean norm of the difference.
[Equation]
The background loss may encourage g ( I, Z T ) to be as inclusive as possible, so that the complementis minimal.
Additionally, the relevancy map loss L rmap may encourage the obtained foregroundmask to be overly inclusive.
To encourage g to be more spatially limited, we add a regularization loss
[Equation]
The combined loss is defined as L = λ 1 ∗ L fore ( I, t ) + λ 2 ∗ L back ( I, t ) + λ 3 ∗ L rmap ( I, H ) + λ 4 ∗
L reg ( I ) , where λ 1 , ..., λ 4 are fixed weighting parameters for all datasets, which were determinedafter a limited hyperparameters search on the CUB [76] validation set to be 1, 1, 4, 1 respectively.
3.1 Inference Tasks
Weakly supervised object localization (WSOL)
The goal of the WSOL task is to localize a singleobject in the image I without local annotation.
For each fine-grained dataset, there is a single typeof object (birds, cars, dogs, etc), and we do not guide the segmentation process with any type ofconditioning.
The text is used only in the training time for the loss functions, and is simply the nameof the object (“birds”, “cars”, “dogs”).
For converting the obtained continuous segmentation map g ( I ) , which is a sigmoid output between 0to 1, to a binary mask, we follow the binarization used, e.g., by Qin et al.
[59, 18, 17, 66], which isthe standard in the benchmark.
Namely, low-value pixels, below the threshold of 0.1, are zeroed out.
Then, the method of Suzuki et al.
[70] is used to extract contours.
From the largest contour found, thealgorithm selects the bounding box of the object.
Weakly supervised phrase-grounding (WSG)
The aim of weakly supervised phrase-grounding isto generate a localization map, given an image I and a textual phrase t for a specific object in theimage.
To extract the objects from the output map of M = g ( I, Z T ) , we first zero low-value pixels,using a threshold of 0.5.
Next, we find contours with the method of Suzuki et al.
For eachcontour, we extract the proper bounding box.
The score for each box is calculated from the outputmap in the positions of the detected box.
In the final stage, we apply non-maximal suppression overthe boxes with 0.3 IoU, and filtering boxes with a low score (50% less) compared to the maximumscore.
See Appendix E for the visualization of these steps.
WWbL Inference
The WWbL task is an open-world localization task, in which the only input is agiven image.
The task consists of localizing and describing all the elements composing the scene.
To solve this task, we propose a two-stage algorithm based on a selective search [74] algorithm toextract object proposals to describe local regions in the image and incorporate the learned network g ,the image captioning network BLIP, and the CLIP model.
Captioning models usually tend to describe the main element of the scene, neglecting other objects orthe background.
To address this issue, our proposed algorithm extracts local regions from the imagewith the Selective Search [74] procedure, which first over-segments the image into super-pixels, andthen groups those in a bottom-up manner to propose candidate bounding boxes, see line 2 of Alg.
Each region is passed through BLIP to generate a local caption (line 3).
To avoid duplicate captions, the algorithm projects the captions to CLIP space (line 4) and applies the
Community Detection (Cd) algorithm
[9] to find caption clusters (line 5).
The clustering algorithmuses cosine similarity in CLIP space between each caption such that a matrix of size M × M isobtained, where M is the number of captions.
The algorithm employs two hyperparameters: (i) cosinesimilarity threshold for items in the same cluster and (ii) minimum size of a cluster such that, for eachcaption (row in the matrix), we find the number of captions above the threshold and check whether itis above the minimum size for a cluster.
The final stage of Cd filters out overlapping clusters, givingpriority to larger clusters.
We set the cosine similarity threshold to be 0.85, and the minimum size tobe 2.
This algorithm returns a caption for each cluster and its embedding in CLIP space.
For eachcluster, g generates a localization map.
Next, the bounding boxes are extracted like in Sec.3.1.
3.2 Architecture
Our network g is used in two types of tasks where (1) image captioning is used only for the lossfunction (WSOL) and g receives only the image (2) image captioning is used as input to g and in theloss function (WSG and WWbL).
In this section, we describe each mode.
Require: Input image
I 1: Load trained networks g , BLIP and E t 2: { P i } ni =1 =
SelectiveSearch ( I ) 3: { T i } ni
=1 = BLIP ( { P i } ni =1 ) 4: { Z Ti } ni
=1 = E t ( { T i } ni =1 ) 5: { T i , Z Ti } mi =1 =
Cd ( { Z Ti } ni
=1 , 2 , 0 . 85) 6: D ← ϕ 7: for i = 1 ....m do 8:
M i =
g ( I, Z i )
▷ Generate map 9:
B i ← BE ( M i )
▷ Extract bounding box 10:
D
= D ∪ ( B i , T i )
▷ Add object 11: return D a person riding a tractor the sky with clouds in the background a dog a woman skating on the ice
WSOL Architecture
For WSOL, Network g is based on a U-Net [62] architecture, with skipconnections between the encoder and the decoder for signals with the same spatial resolution.
Thedecoder of g employs five upsampling layers, with each block containing two convolutional layerswith batch normalization after the last convolutional layer before the activation function.
The lastactivation function is Sigmoid, while the others are Relu.
Multi-Model Architecture
For WSG and WWbL, Network g is based on an encoder-decoderarchitecture adapted to support text-based conditioning.
This is illustrated in Fig.
The encoding Z t for the input text t is produced by the CLIP text encoder, where the size of thevector Z t is 512.
The image encoder of g is a VGG network [68], in which the receptive field is ofsize 16 × 16 , i.e., the image is downscaled four times.
The number of channels of the obtained map
Z I is also 512.
The VGG network is initialized using an ImageNet pre-trained model.
We then consider the vector in R 512 associated with each spatial location of the tensor Z I .
It isnormalized to have a norm of one.
The dot product with the vector Z t (which in CLIP is alsonormalized) is computed.
Performing this over all spatial locations, we obtain a map Z s with valuesbetween -1 and 1.
Z s has a single channel and the same spatial dimensions as Z I .
Z s can be viewed as a text-conditioned importance map.
All channels of Z I are multiplied by Z sand the result is passed to the decoder part of g .
This decoder consists of three upsampling blocks,each with a sampling factor of two.
Each block contains two convolutional layers with a kernel sizeequal to 3 and zero padding equal to one.
Batch normalization is used after the last convolution layer,before the activation function.
The first layer’s activation function is a ReLU, while the last layer’sactivation function is a Sigmoid.
Experiments
We present our results for the three tasks: (i) weakly supervised object localization (WSOL) , (ii)weakly supervised phrase grounding (WSG), with training on either MSCOCO 2014 [43] or the
Visual Genome (VG) dataset [40], and (iii) the new task we present (WWbL).
For the first task,we employ three fine-grained localization datasets, and for the other two, we use the three datasetscommonly used in WSG.
Datasets For the task of WSOL we evaluate our model on fine-grained datasets often used for thistask.
CUB-200-2011 [76] contains 200 birds species, with 11,788 images divided into 5994 trainingimages and 5794 test images.
Stanford Car [39] contains 196 categories of cars, with 8144 samplesin the training set and 8041 samples in the test set.
Stanford dogs [36] consists of 20,580 images,with a split of 12,000 for training and 8580 for testing, where the data has 120 classes of dogs.
For the task of WSG and WWbL, our model trains on the split of MSCOCO and VG, and we evaluateit on the test splits of Flickr30k, ReferIt, and VG.
MSCOCO 2014 [43], using the splits of Akbari etal.
[2], consists of 82,783 training images and 40,504 validation images.
Each image has five captionsdescribing it.
VG [40] contains 77,398 training, 5000 validation, and 5000 test images.
Each imagecomes with a set of free-form text annotated bounding boxes.
Flickr30k [58] Entities, which is based on Flickr30k, contains 224K phrases describing objects inover 31K images, each described by 5 captions.
For evaluation, we use the same 1k images from the
[Table] [Table]
[Table] [Table]
test split of Akbari et al.
ReferIt [27, 15] contains 20K images and 99,535 segmented images thatalso contain a description for the entire image.
These image regions were collected in a two-playergame with approximately 130K isolated entity descriptions.
We use the split of Akbari et al.
Implementation details
In WSOL, our Algorithm receives an input image of size 224 × 224.
During training, the image is first resized to 256 × 256 and then a random crop of size 224 × 224 isextracted.
During the evaluation, the image is resized to 224 × 224.
Network g for WSOL is based on an Imagenet pretrained visual encoder.
We compare the performance obtained using VGG16 [68], MobileNetV1 [30], Resnet50 [29], InceptionV3 [71] and
DenseNet161 [31] is applied for CUB dataset to compare performances with the state-of-the-artbaselines, while for the other datasets we used Resnet50, since that is what the literature baselineshave published.
A SGD optimizer with a batch size of 48 and an initial learning rate of 0.0003 for 100epochs is used.
The optimizer momentum of 0.9 and weight decay of 0.0001 are also used.
Duringthe training, a random horizontal flip with 0.5 probability is applied.
All models are trained on asingle GeForce RTX 2080Ti Nvidia GPU.
In WSG training (the same network is used for inference in both WSG and WWbL), following Akbariet al.
[2], network g receives an input image of size 299 × 299 and generates a localization map of thesame size.
During training, the image is also resized to 224 × 224, which is the input size of CLIP.
The relevancy map H is generated at this resolution and is resized to 299 × 299.
[Table]
Network g for WSG and WWbL is VGG16 [68], to ensure a fair comparison with the baselines.
An
SGD optimizer (batch size of 32 and an initial learning rate of 0.0003) is used for 100 epochs.
Theoptimizer momentum of 0.9 and weight decay of 0.0001 are also used.
During training, a randomhorizontal flip is applied, with 0.5 probability.
All models are trained on a double GeForce RTX2080Ti Nvidia GPU, while all experiments of Arbelle et al.
[3] were conducted on four V100 GPUs.
Evaluations and Results
For the WSOL task, the main accuracy metric measures whether theintersection over union (IoU) of the ground-truth bounding box and the detector’s output are above0.5.
As can be seen in Tab. 2, our method obtains state-of-the-art performance for the CUB [76]dataset, for all five different backbones.
The method also outperforms GAE [12] with CLIP as abackbone.
Our method improves the GAE map, which it uses during training, by more than 25% inthe accuracy metric.
The results listed in Tab.
3 show that our method also achieves state-of-the-artperformance for the fine-grained localization datasets of dogs and cars.
Following previous work,a Resnet-50 is used for these methods, except for GAE, which is applied with a CLIP backbone.
Appendix F presents sample results.
For the WSG task, the algorithm is evaluated with respect to the accuracy of the pointing game [89],which is calculated from the output map by finding the maximum-value location for the given queryand checking whether this point is located in the region of the object.
Another metric we report(B-Box accuracy) compares the extracted bounding-boxes with the bounding-box annotations in thesame manner as the WSOL task above.
For a fair comparison, we use the same training/validation/testsplits as Akbari et al.
For ReferIt, Visual Genome and Flickr30K, we treat each query as a singlesentence.
Tab.
6 summarises the results for the Flickr30k, ReferIt and Visual Genome datasets forthe WSG task.
Evidently, our method is superior to all baselines, whether training takes place over
VG or MS-COCO.
Tab.
7 presents bounding box accuracy results for the WSG task.
Here, too, ourmethod outperforms the baseline methods.
In appendix B we present sample output maps.
For the WWbL task, we use the same metrics (pointing game and bounding-box accuracy).
For eachground-truth pair of bounding box and caption, we select the closest caption in CLIP space fromthe output predictions and compare the output map to the bounding box using the pointing accuracymetric.
In addition, bounding boxes are extracted for the heat-map, as described at 3.1, and comparedto the selected ground-truth bounding boxes with the same accuracy metric as above.
Tab.7 presents the results for the WWbL task.
In addition to our method (Alg. 1) that utilizes networkg to generate a heat-map, given an image and a text (line 8), we also test two alternative heatmapgeneration methods.
The first one is the MG method of Akbari et al.
[2] for the WSG task, and thesecond is the GAE explainability method [12].
As can be seen in the results listed in Tab. 7, ourproposed method obtains the best results among the three.
For comparison, we also provide WSGresults using Alg.
1 and the ground truth text for the same three heatmap methods.
Interestingly, theg that is trained on the VG dataset shows better performance for WSG, while in WWbL the resultsare mixed.
Fig 3 and Appendix D present sample WWbL results.
Ablation Study
In this section, we compare the accuracy of g with different modes of the lossfunction and present the experiment used to determine the values of λ 1 , 2 , 3 , 4 .
In addition, we alsocompare the performance of two backbone families, Resnet [29] and HarDnet [10].
All experimentswere conducted in the WSOL settings for the CUB[76] dataset.
Tab. 5 presents the sensitivity of our network g to each loss term, as evaluated on the test set, usingthe Resnet50 backbone.
The first part of the table compares the performance without L r map , usingdifferent combinations of the other loss terms.
As can be seen, the best performance is achieved when
[Table]
we use the three terms L fore , L back , L reg together.
Each term improves the performance:
L foreimproves it by 21%, L back by 5% and L reg by 48%.
The second part compares the performancewith L rmap which improved by more than 6%.
As can be seen, each of the other terms improves theperformance: L fore improves it by 4%, L back by 2%, and L reg by 5.5%.
Also, training only with
L rmap decreases the performance by more than 9% compared to performance using all loss terms.
The backbone ablation was conducted with two backbone families and is summarised in Tab. 4.
Forboth families, performance improves as the number of parameters is increased.
Appendix C, presents the sensitivity of the network g to the λ 3 coefficient, which is the weight of
L rmap and the only coefficient that is not set to 1 in our experiments.
The method seems to be largelyinsensitive to this parameter and it is stable for a wide range of values.
The ability of g to generalize well under different pre-trained CLIP models is examined in App. H.
Evidently, the variance in the performance of g is much lower than the variance in the zero-shotclassification performance of the CLIP models used to train g .
Discussion and limitations
When biological systems see, the input is seldom limited to single images.
Although our WWbLmethod was developed in the context of stills, it can be naturally extended to perform onlinelocalization of objects in video.
For this, the detected regions and their matching concepts need tobe tracked and must evolve over time, with the ability to correct previous hypotheses.
We leave thiseffort for future research, since benchmarking this ability with the current datasets is challenging.
The common computer vision terminology makes the distinction between object localization andobject detection.
The former aims to locate the main object of a certain type in the input image, whilethe latter aims to find all the relevant objects and their individual boundaries.
Our WWbL methodaims to extract every existing object and can, therefore, detect all image objects.
The method employs the selective search method [74].
This can be avoided by identifying one regionat a time and iterating the process.
Such a method is presented in Appendix A. It uses g to recover theobject map at each stage.
The results reported in this Appendix are somewhat lower than the methodbased on a selective search.
Since the image datasets and pre-trained models our method employs may be biased towards westernmedia concepts, it is culturally biased.
For similar reasons, it may display discriminatory behavior.
Another limitation is that our weakly supervised learning scheme does not distinguish betweenmultiple instances of the same object.
While Algorithm 1 can be improved to somewhat mitigate this,by separating multiple objects that have the same caption, building such a solution robustly may bechallenging without additional supervision.
App.
G present visualization for this scenario.
Conclusions
Through the power of large pre-trained transformers and by integrating explainability cues, we buildan effective weakly supervised phrase-grounding network g .
By combining an off-the-shelf imagecaptioning engine, we are able to identify and localize the objects within an input image.
The taskwe solve is an open-world one and the setting generalizes multiple existing tasks.
It is convincinglydemonstrated that our weakly supervised solution is on par with the fully supervised alternatives thatexist for the phrase-grounding task.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Will be includedin the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A WWbl algorithm without selective search
In this section, we present an algorithm for wwbl without the selective search method for describingthe objects in the image.
The algorithm runs N iterations with a break option when no object was found.
In line 2 of the listedalgorithm, we obtain the caption of the input image T 0 = BLIP ( I ) .
Lines 4-16 of Alg.
2 describe the iterative algorithm used to extract objects from a single image.
Thefirst operation is to encode the text in the current iteration with the CLIP text encoder.
The next stepis to generate a map with g .
From the output map M i we extract the largest bounding box ( B i ), usingthe algorithm described in section 3.2.
For the extracted bounding box, the algorithm crops a relativepatch from the image ( P i ), and updates the image, such that the patch pixels are zero.
The newimage is used for generating the new caption T Pi =
BLIP ( P i ) .
Next, the algorithm calculates thesimilarity between E t ( T 0 ) and E t ( T Pi ) .
Only if the similarity is above τ = 0 .
6 does the algorithmadd this bounding box to the predicted output and continues with the iterations.
Otherwise, it stopsthe routine and returns the predicted output.
Tab. 8 compares the performances of algorithm 1 and 2.
Evidently, while Alg. 2 addresses WWbL tosome degree, Alg. 1 is considerably better.
Algorithm 2 WWbL Inference
Require: Input image
I 1: Load networks g , CLIP, BLIP 2: T 0 = BLIP ( I ) 3:
D ← ϕ
▷ Empty output dictionary 4: for i =
....n − 1 do 5:
Z i =
E t ( T i )
▷ CLIP text encoder 6:
M i =
g ( I, Z i )
▷ Generate map 7:
B
i ← C ( M i )
▷ Extract bounding box 8:
P i ← I
[ B i ] ▷ Crop patch 9:
T Pi = BLIP ( P i )
▷ Caption patch 10:
S = CLIP ( T Pi , T 0 ) 11: if S ≥ 0 .
6 then 12:
D
= D ∪ ( B i , T Pi )
▷ Add object 13:
I [ B i ] =
▷ Delete the object 14:
T i +1 =
BLIP ( I )
▷ New caption 15: else 16: break 17: return D
[Table]
Algorithm
CLIP+VGG
VG 35.
Algorithm
CLIP+VGG
VG 43.
Algorithm
CLIP+VGG
MS-COCO 32.
Algorithm
CLIP+VGG
MS-COCO 44.
B Visualization of baselines - task WSG
[Figure]
We present visualization of our network g for different scenarios, together with the GAE method,to emphasize the refinement of our network, as can be seen at Fig.
The proposed loss terms fortraining g encourage the output mask to capture the whole shape of the object instead of specificregions of GAE which tend to focus more on discriminate regions.
[Figure]
C λ 3 sensitivity
The proposed loss terms have four coefficients that balance the final loss function.
To avoid anexcessive search for hyperparameters, we set λ 1 , λ 2 and λ 4 to one and only set λ 3 at a different value.
In Tab.
9 we study the effect of λ 3 on the performance of the learned g for the WSOL task on the
CUB benchmark [76] and for WSG on Filckr30K.
Evidently, the value of λ 3 = 4 is optimal forboth experiments (this is the value used throughout our work).
However, performance is relativelystable in a wide range of values.
As noted in the ablation experiments of the paper, a value of zero(canceling this term), leads to a drop of 6% for WSOL on CUB.
The current results indicate an evenlarger drop for Phrase grounding.
[Table]
D WWbL Visualization
In this section, we show visualizations of the proposed task - "what is where by looking (WWbL)".
For each image, our algorithm describes the objects and generates a localization map.
Fig.
5 presentsresults for our method from flickr dataset.
Fig.
6 shows results for our method from web imagessamples.
Fig.
7 shows results for our method from BBC - Week in pictures from the recent weeks(April-May 2022).
E Visualization of bounding box extraction from localization map
In this section, we present visualization for bounding box extraction, which is used for WWbl, andalso for WSG.
We first set each pixel in the localization map that is below 50% under the maximumvalue in the map to zero.
Next, we apply the algorithm of Suzuki et al.
[70] to find the contours of allobjects in the map.
We consider the bounding box of each contour.
We then calculate the sum ofall values of the map g inside the bounding box, which we refer to as “energy”.
The final stage is tofilter out bounding boxes via non-maximal suppression using the energy values, for all overlappingbounding boxes with an IOU larger than 0.3.
The bounding boxes are then filtered further, keepingonly the ones with energy values of at least 50% of that of the bounding box with the maximal energy.
Fig.
8 presents a visualization of the method where (a) the input image (b) the localization map(c) localization map after thresholding with bounding box proposals (d) localization map afterthresholding with the final bounding boxes (e) the input image with the final bounding boxes.
As canbe seen, the first step of zeroing low-value pixels allows the method to focus on the relevant regionsin the image based on the text.
The importance of filtering non-relevant bounding boxes based on
NMs and low energy also rises from the next examples.
[Figure]
F WSOL visualization
In this section, we present a visualization of WSOL task for fine-grained datasets where the first rowis the Stanford-cars dataset, the second row is the CUB dataset, and the last row is Stanford-dogs.
Fig.
9 present our results together with GAE[12] which emphasizes our improvements for the relevantmap of the GAE (this is used, during training only, as one of our losses) for the localization task.
G Multiple instances of the same object
In Fig.
10, we present our method’s results for images with multiply instances of the same object,such as apples and dogs.
On the left side, we present the results of the WWbL method.
On the rightside, we present results for grounding specific sentences that are provided to study the output ofnetwork g .
As can be seen, WWbL does not typically select sentences that distinguish between the variousobjects.
However, network g can separate between different objects of the same class given specificcaptions.
When there are multiple objects of the same type, e.g., multiple green apples, g marks allof them.
We note that g ’s heatmap does peak at specific parts of the objects, which may facilitateinstance separation
[Figure]
H Employing different pre-trained CLIP models
In this section, we examine the ability of g to perform weakly supervised localization on CUB and Filckr30K for WSG task, when employing CLIP that is pre-trained on different datasets.
The models were taken from the OpenCLIP open-source repository [32], which provides both thepre-trained CLIP models and the obtained imagenet1K zero-shot accuracy.
As can be seen in Tab. 10, while the zero-shot classification varies considerably, the performanceof g on the localization task varies much less.
Also, there does not seem to be a strong correlationbetween the amount of training data used for training CLIP and the localization performance of g .
A similar experiment was conducted for the phrase grounding task, with similar results.
As Tab. 10indicates, the ResNet50 model slightly outperforms the results obtained by the ViT32 model, whenboth are trained on the openai dataset.
Other datasets, both bigger and smaller, do not lead to betterperformance.
We note that the gap observed for both WSOL and Phrase Grounding is much narrower than the gapthat we see in top-1 accuracy for zero-shot classification. This indicates that our method is relativelystable concerning the CLIP model used during training.
[Figure]
[Table]
image of a lady image of a child image of a golfer
[Figure] [Figure]