What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs
Anonymous Author(s)
AffiliationAddress email
Abstract
Given an input image, and nothing else, our method returns the bounding boxes ofobjects in the image and phrases that describe the objects.
This is achieved withinan open world paradigm, in which the objects in the input image may not havebeen encountered during the training of the localization mechanism.
Moreover,training takes place in a weakly supervised setting, where no bounding boxes areprovided.
To achieve this, our method combines two pre-trained networks:
the
CLIP image-to-text matching score and the BLIP image captioning tool.
Trainingtakes place on COCO images and their captions and is based on CLIP.
Then,during inference, BLIP is used to generate a hypothesis regarding various regionsof the current image.
Our work generalizes weakly supervised segmentation andphrase grounding and is shown empirically to outperform the state of the art inboth domains.
It also shows very convincing results in the novel task of weaklysupervised open-world purely visual phrase-grounding presented in our work.
Forexample, on the datasets used for benchmarking phrase-grounding, our methodresults in a very modest degradation in comparison to methods that employ humancaptions as an additional input.
Introduction
What does it mean, to see?
The plain man’s answer (and Aristotle’s too) would be, to know what is where by looking.
In other words, vision is the process of discovering from images what is present in the world, and where it is.
Marr [47]
We address the task of detecting the objects that exist in a given image, without limiting ourselves toa predefined list of objects, of the type that detection algorithms employ.
This property is sometimesreferred to as open-world vision.
Our method assigns a text phrase to each detected object.
Thesimilar task that employs an additional text input describing the image is called phrase-grounding.
We describe our approach as “purely visual” to denote that there is no text input.
Since the taskis performed without the use of bounding boxes in any phase of training, the level of supervisionemployed corresponds to weakly supervised.
As far as we can ascertain, the task of weakly-supervisedopen-world purely visual phrase-grounding is entirely novel.
For short, we refer to this task, which isillustrated in Fig 1, as “what is where by looking” (WWbL).
In order to tackle this task we rely on two recent pretrained vision-language models: CLIP [57] and
BLIP [38].
These models are trained on large corpora of matching images and captions and canproduce a matching score between an input image and an input text.
Trained on millions of such pairs,
CLIP has been extensively used in zero-shot recognition [89], image generation [21, 25, 42], imageediting and manipulation [54, 11, 24, 90], image and video captioning [50, 69, 70] and other tasks[49].
BLIP, a more recent addition, also incorporates into its training phase the ability to generate
[Figure]
[Table]
What is where by looking (WWbL)
[Table]
✓ ✓ ✓ image captions.
It also relies on a bootstrapped captioning model to generate matching sentences andon a bootstrapped filtering model to eliminate false matches from its training set.
While BLIP can produce a matching score, we choose to use CLIP for producing such a score and
BLIP for generating plausible captions.
This modular choice allows us to study the matching-basedcomponents (such as generalizing weakly supervised localization methods) separately from the
WWbL. It also emphasizes the modular nature of our method.
Given time and resources, we wouldbe able to evaluate the method using only BLIP or replacing BLIP with CLIP-based image captioningmethods, which are considerably slower than BLIP [70].
Our WWbL method comprises two phases.
During training, an encoder-decoder g is applied toextract a foreground mask for an input image, given an image caption.
This is the same level ofsupervision that is required to train CLIP or BLIP, albeit performed on a much smaller dataset.
Sinceno segmentation mask (or bounding box) is used, this is considered weakly supervised.
Four loss terms are used.
The first one requires that the CLIP matching score of the foreground regionand the matching text be maximal.
The second helps ensure that the background region is unrelatedto the text.
The third term considers the explainability map of CLIP, given the input image and thecaption [13] as a guide for the foreground mask.
Lastly, in order to encourage compact foregroundmasks, a regularization term is added.
A two-stage inference time procedure employs the learned foreground segmentation network gand a pretrained-BLIP model.
The first stage extracts object proposals using a Selective Searchalgorithm [71].
The second stage uses BLIP to generate captions for each proposal and find textclusters in CLIP’s embedding space, in order to avoid repetitive descriptions.
Network g is then usedto find the foreground mask for each sentence, given the entire image as input and the sentence.
Our results show that network g , as a weakly supervised localization network, outperforms thestate-of-the-art.
The same model also outperforms the state-of-the-art in phrase grounding.
When theentire method is applied, with the two stages described above, the results obtained for WWbL are onpar with the results obtained using human captions for the task of phrase grounding.
These results areconsiderably better than those obtained by the baselines.
Related Work
Open-world computer vision is an emerging paradigm that stands in contrast to recognition systemsthat can only address the limited number of classes and types of examples that were observed duringtraining.
Specifically, open-set classification [7, 81, 35] requires the ability to understand when a newsample is not from the observed classes.
Since large visual-language models are trained on hundreds of millions of samples [57], many classesof objects are observed during training.
It is impractical to enumerate all object classes.
Moreover,these models can recognize phrases that were unseen during training, e.g., “a green dove”.
Therefore, the use of the term "open-world" for such models is not based on learned vs. unseen classes, butrather on testing the method without limiting the type of objects observed during the inference test.
Open-world localization methods generalize more restricted methods, which rely on a relativelyshort list of detectable objects.
Similarly, weakly supervised methods, which employ labeling at theimage level, generalize fully supervised methods that require bounding boxes during training.
Finally,methods that do not receive as input guiding information regarding the content of the image (we callthese “purely visual”) generalize methods that receive as input a list of objects to detect or text thatdescribes the contents of the image.
Tab. 1 compares the task we tackle to other localization tasks.
As can be seen, the task we tackle-“what is where by looking” (WWbL) - generalizes all other localization tasks.
Unlike supervisedobject detection, it is open-set and weakly supervised.
Weakly supervised localization methods arenot open-set.
Lastly, phrase grounding methods employ a textual input, while WWbL does not.
Object Detection
One of the fundamental tasks of computer vision is object detection, where aclosed set of object categories is localized based on training sets that contain costly bounding-boxannotation [40, 20].
Since the advent of deep learning, many fully supervised algorithms have beenproposed for this task [58, 26, 41].
In contrast to these methods, our approach employs image-levelannotation (global text description) and an open dictionary to describe the detected object.
Phrase Grounding
is a weakly supervised phrase localization task, in which text phrases areassociated with local image annotations and local human-annotation signal [2, 84, 32].
Manymethods extract text embedding from a pretrained language model [16, 60], together with imagerepresentations, to obtain a common semantic space for image-text pairs.
Our model is based on
CLIP [57] as a text-image relation space to extract the localization of objects.
Li et al.
[39] collected27M image-text data points, of which 3M have local human annotation, and used a semi-supervisedtraining method for phrase-grounding, which employs CLIP.
It cannot be compared directly withour work, since it uses additional supervision.
Arbelle et al.
[3] proposed a self-supervision methodscheme for generating a localization map.
Weakly Supervised Localization (WSOL)
Class Activation Map (CAM) explainability methodshave been offered in recent years for solving WSOL tasks [87, 88, 56, 43].
Most of these algorithmstrain a classifier to distinguish between sub-categories of the main object (Birds, Cars, Dog etc),employing a localization loss term for the explainability map [76, 78, 48, 28, 52].
Similarly to us, Shaharabany et al.
[63] employ an external localization network in addition to theclassifier.
During training, the classifier supplies gradients to the localization network.
In contrast toour work: (1) they compare the classifier outputs with and without the foreground mask, while ouralgorithm compares the masked image with the text in the CLIP space, (2) there is no backgroundloss term, and (3) explainability is not used as a signal.
Image Captioning
Captioning is a fundamental vision-language task.
Early methods applied
RNNs [46, 34].
Attention was added to identify relevant salient objects [79, 61].
Subsequently,
Transformers modeled interactions among all image elements with self-attention [72, 53, 22].
Recentworks have shown significant improvements in robustness and generalization by using large-scalevision-language data sets [70, 50, 38].
Our method employs BLIP [38], an image captioning modelthat shows good performances for diverse image domains and a large variety of real-world objects.
Explainability
Many methods generate a heatmap that indicates relevancy for CNNs, e.g., [62, 8,44, 64].
Most relevant to our work is the use of GradCAM [62] relevancy maps as a cue for weaklysupervised segmentation methods [51, 75].
The literature on transformer explainability is sparse andis mostly limited to pure self-attention architectures [1, 12].
A recent work by Chefer et al.
[12] is thefirst to fully address bi-modal transformer networks, and is the method employed to obtain relevancymaps.
The solution is based on Layer-wise Relevance Propagation [5], with gradient integration forthe self-attention and co-attention layers.
Method
In this section, we describe our method for generating a localization map based on image andtext input signals without local annotation.
We use our method on three different tasks (i)
Weakly
[Figure]
CLIP
[Figure] CLIP CLIP CLIP Explainability supervised object localization (WSOL) (ii) Weakly supervised phrase-grounding (WSG) (iii)
What iswhere by looking (WWbL).
At the core of our method lies a segmentation network g , which is trained in a weakly supervisedmanner.
The network g has two inputs: an input image I ∈ R 3 × W × H , and an embedding of the inputtext Z T .
For the simpler task of WSOL, network g receives only the first input I as input signal.
Theloss terms used in both cases are the same, and, for the sake of generality we write the equations withg that has two inputs.
The training of g does not involve any ground truth localization information.
Instead, it relies on twotypes of signals: (1) comparing the CLIP score of the extracted segment to that of the entire image,and (2) including the heatmap of the CLIP network, as obtained with the relevancy method of Cheferet al.
[14], in the segmented region.
Four-loss terms are used, using the input image I , the text t , and the relevancy heat-map of CLIP given
I and t , which we denote as H .
The terms are (1) a foreground loss L fore ( I, t ) (2) a backgroundloss L back ( I, t ) , (3) a relevancy heatmap loss L rmap ( I, H ) , and (4) a regularization loss L reg ( I, g ) .
The foreground loss encourages the segmentation network g to output a map that increases thesimilarity between the text and the masked image g
( I, t ) ⊙ I , ⊙ being the Hadamard product.
[Equation]
where CLIP ( J, s ) denotes the CLIP matching score between an image J and the text s .
The background loss helps ensure that the complementary part to that returned by g ( I, Z T ) isirrelevant, in the sense that its CLIP matching score with the text is low.
[Equation]
The relevancy map H obtained by the method of Chefer et al.
[14] on CLIP ( I, t ) provides us withanother important cue.
While it is not entirely reliable in the sense that irrelevant regions are oftenmarked and those regions often do not include the entire relevant part, we find that it often highlightssome of the relevant parts.
Since both H and g ( I, Z T ) have the same range of values ([0,1]), weemploy the squared Euclidean norm of the difference.
[Equation]
The background loss may encourage g ( I, Z T ) to be as inclusive as possible, so that the complementis minimal.
Additionally, the relevancy map loss L rmap may encourage the obtained foregroundmask to be overly inclusive.
To encourage g to be more spatially limited, we add a regularization loss
L reg ( I, g )) = ∥ g ( I, Z T ) ∥ (4)
The combined loss is defined as L = λ 1 ∗ L fore ( I, t ) + λ 2 ∗ L back ( I, t ) + λ 3 ∗ L rmap ( I, H ) + λ 4 ∗
L reg ( I ) , where λ 1 , ..., λ 4 are fixed weighting parameters for all datasets, which were determinedafter a limited hyperparameters search on the CUB [73] validation set to be 1, 1, 4, 1 respectively.
3.1 Inference Tasks
Weakly supervised object localization (WSOL)
The goal of the WSOL task is to localize a singleobject in the image I without local annotation.
For each fine-grained dataset, there is a single typeof object (birds, cars, dogs, etc), and we do not guide the segmentation process with any type ofconditioning.
The text is used only in the training time for the loss functions, and is simply the nameof the object (“birds”, “cars”, “dogs”).
For converting the obtained continuous segmentation map g ( I ) , which is a sigmoid output between 0to 1, to a binary mask, we follow the binarization used, e.g., by Qin et al.
[56, 18, 17, 63], which isthe standard in the benchmark.
Namely, low-value pixels, below the threshold of 0.1, are zeroed out.
Then, the method of Suzuki et al.
[67] is used to extract contours.
From the largest contour found, thealgorithm selects the bounding box of the object.
Weakly supervised phrase-grounding (WSG)
The aim of weakly supervised phrase-grounding isto generate a localization map, given an image I and a textual phrase t for a specific object in theimage.
To extract the objects from the output map of M = g ( I, Z T ) , we first zero low-value pixels,using a threshold of 0.5.
Next, we find contours with the method of Suzuki et al.
For eachcontour, we extract the proper bounding box.
The score for each box is calculated from the outputmap in the positions of the detected box.
In the final stage, we apply non-maximal suppression overthe boxes with 0.3 IoU, and filtering boxes with a low score (50% less) compared to the maximumscore.
See Appendix E for the visualization of these steps.
WWbL Inference
The WWbL task is an open-world localization task, in which the only input is agiven image.
The task consists of localizing and describing all the elements composing the scene.
To solve this task, we propose a two-stage algorithm based on a selective search [71] algorithm toextract object proposals to describe local regions in the image and incorporate the learned network g ,the image captioning network BLIP, and the CLIP model.
Captioning models usually tend to describe the main element of the scene, neglecting other objects orthe background.
To address this issue, our proposed algorithm extracts local regions from the imagewith the Selective Search [71] procedure, which first over-segments the image into super-pixels, andthen groups those in a bottom-up manner to propose candidate bounding boxes, see line 2 of Alg.
Each region is passed through BLIP to generate a local caption (line 3).
To avoid duplicate captions, the algorithm projects the captions to CLIP space (line 4) and applies the
Community Detection (Cd) algorithm
[9] to find caption clusters (line 5).
The clustering algorithmuses cosine similarity in CLIP space between each caption such that a matrix of size M × M isobtained, where M is the number of captions.
The algorithm employs two hyperparameters: (i) cosinesimilarity threshold for items in the same cluster and (ii) minimum size of a cluster such that, for eachcaption (row in the matrix), we find the number of captions above the threshold and check whether itis above the minimum size for a cluster.
The final stage of Cd filters out overlapping clusters, givingpriority to larger clusters.
We set the cosine similarity threshold to be 0.85, and the minimum size tobe 2.
This algorithm returns a caption for each cluster and its embedding in CLIP space.
For eachcluster, g generates a localization map.
Next, the bounding boxes are extracted like in Sec.3.1.
3.2 Architecture
Our network g is used in two types of tasks where (1) image captioning is used only for the lossfunction (WSOL) and g receives only the image (2) image captioning is used as input to g and in theloss function (WSG and WWbL).
In this section, we describe each mode.
WSOL Architecture
For WSOL, Network g is based on a U-Net [59] architecture, with skipconnections between the encoder and the decoder for signals with the same spatial resolution.
Thedecoder of g employs five upsampling layers, with each block containing two convolutional layers
Require: Input image
I 1: Load trained networks g , BLIP and E t 2: { P i } ni =1 =
SelectiveSearch ( I ) 3: { T i } ni
=1 = BLIP ( { P i } ni =1 ) 4: { Z Ti } ni
=1 = E t ( { T i } ni =1 ) 5: { T i , Z Ti } mi =1 =
Cd ( { Z Ti } ni
=1 , 2 , 0 . 85) 6: D ← ϕ 7: for i = 1 ....m do 8:
M i =
g ( I, Z i )
▷ Generate map 9:
B i ← BE ( M i )
▷ Extract bounding box 10:
D
= D ∪ ( B i , T i )
▷ Add object 11: return D a person riding a tractor the sky with clouds in the background a dog a woman skating on the ice with batch normalization after the last convolutional layer before the activation function.
The lastactivation function is Sigmoid, while the others are Relu.
Multi-Model Architecture
For WSG and WWbL, Network g is based on a encoder-decoderarchitecture adapted to support text-based conditioning.
This is illustrated in Fig.
The encoding Z t for the input text t is produced by the CLIP text encoder, where the size of thevector Z t is 512.
The image encoder of g is a VGG network [65], in which the receptive field is ofsize 16 × 16 , i.e., the image is downscaled four times.
The number of channels of the obtained map
Z I is also 512.
The VGG network is initialized using an ImageNet pre-trained model.
We then consider the vector in R 512 associated with each spatial location of the tensor Z I .
It isnormalized to have a norm of one.
The dot product with the vector Z t (which in CLIP is alsonormalized) is computed.
Performing this over all spatial locations, we obtain a map Z s with valuesbetween -1 and 1.
Z s has a single channel and the same spatial dimensions as Z I .
Z s can be viewed as a text-conditioned importance map.
All channels of Z I are multiplied by Z sand the result is passed to the decoder part of g .
This decoder consists of three upsampling blocks,each with a sampling factor of two.
Each block contains two convolutional layers with a kernel sizeequal to 3 and zero padding equal to one.
Batch normalization is used after the last convolution layer,before the activation function.
The first layer’s activation function is a ReLU, while the last layer’sactivation function is a Sigmoid.
Experiments
We present our results for the three tasks: (i) weakly supervised object localization (WSOL) , (ii)weakly supervised phrase grounding (WSG), with training on either MSCOCO 2014 [40] or the
Visual Genome (VG) dataset [37], and (iii) the new task we present (WWbL).
For the first task,we employ three fine-grained localization datasets, and for the other two, we use the three datasetscommonly used in WSG.
Datasets For the task of WSOL we evaluate our model on fine-grained datasets often used for thistask.
CUB-200-2011 [73] contains 200 birds species, with 11,788 images divided into 5994 trainingimages and 5794 test images.
Stanford Car [36] contains 196 categories of cars, with 8144 samplesin the training set and 8041 samples in the test set.
Stanford dogs [33] consists of 20,580 images,with a split of 12,000 for training and 8580 for testing, where the data has 120 classes of dogs.
For the task of WSG and WWbL, our model trains on the split of MSCOCO and VG, and we evaluateit on the test splits of Flickr30k, ReferIt, and VG.
MSCOCO 2014 [40], using the splits of Akbari etal.
[2], consists of 82,783 training images and 40,504 validation images.
Each image has five captionsdescribing it.
VG [37] contains 77,398 training, 5000 validation, and 5000 test images.
Each imagecomes with a set of free-form text annotated bounding boxes.
Flickr30k [55] Entities, which is based on Flickr30k, contains 224K phrases describing objects inover 31K images, each described by 5 captions.
For evaluation, we use the same 1k images from thetest split of Akbari et al.
ReferIt [27, 15] contains 20K images and 99,535 segmented images that
[Table] [Table]
[Table] [Table]
also contain a description for the entire image.
These image regions were collected in a two-playergame with approximately 130K isolated entity descriptions.
We use the split of Akbari et al.
Implementation details
In WSOL, our Algorithm receives an input image of size 224 × 224.
During training, the image is first resized to 256 × 256 and then a random crop of size 224 × 224 isextracted.
During the evaluation, the image is resized to 224 × 224.
Network g for WSOL is based on an Imagenet pretrained visual encoder.
We compare the performance obtained using VGG16 [65], MobileNetV1 [30], Resnet50 [29], InceptionV3 [68] and
DenseNet161 [31] is applied for CUB dataset to compare performances with the state-of-the-artbaselines, while for the other datasets we used Resnet50, since that is what the literature baselineshave published.
An SGD optimizer with a batch size of 48 and an initial learning rate of 0.0003for 100 epochs is used.
The optimizer momentum of 0.9 and weight decay of 0.0001 are also used.
During the training, a random horizontal flip with 0.5 probability is applied.
All models are trainedon a single GeForce RTX 2080Ti Nvidia GPU.
In WSG training (the same network is used for inference in both WSG and WWbL), following Akbariet al.
[2], network g receives an input image of size 299 × 299 and generates a localization map of thesame size.
During training, the image is also resized to 224 × 224, which is the input size of CLIP.
The relevancy map H is generated at this resolution and is resized to 299 × 299.
[Table]
Network g for WSG and WWbL is VGG16 [65], to ensure a fair comparison with the baselines.
An
SGD optimizer (batch size of 32 and an initial learning rate of 0.0003) is used for 100 epochs.
Theoptimizer momentum of 0.9 and weight decay of 0.0001 are also used.
During training, a randomhorizontal flip is applied, with 0.5 probability.
All models are trained on a double GeForce RTX2080Ti Nvidia GPU, while all experiments of Arbelle et al.
[3] were conducted on four V100 GPUs.
Evaluations and Results
For the WSOL task, the main accuracy metric measures whether theintersection over union (IoU) of the ground-truth bounding box and the detector’s output are above0.5.
As can be seen in Tab. 2, our method obtains state-of-the-art performance for the CUB [73]dataset, for all five different backbones.
The method also outperforms GAE [12] with CLIP as abackbone.
Our method improves the GAE map, which it uses during training, by more than 25% inthe accuracy metric.
The results listed in Tab.
3 show that our method also achieves state-of-the-artperformance for the fine-grained localization datasets of dogs and cars.
Following previous work,a Resnet-50 is used for these methods, except for GAE, which is applied with a CLIP backbone.
Appendix F presents sample results.
For the WSG task, the algorithm is evaluated with respect to the accuracy of the pointing game [84],which is calculated from the output map by finding the maximum-value location for the given queryand checking whether this point is located in the region of the object.
Another metric we report(B-Box accuracy) compares the extracted bounding-boxes with the bounding-box annotations in thesame manner as the WSOL task above.
For a fair comparison, we use the same training/validation/testsplits as Akbari et al.
For ReferIt, Visual Genome and Flickr30K, we treat each query as a singlesentence.
Tab.
6 summarises the results for the Flickr30k, ReferIt and Visual Genome datasets forthe WSG task.
Evidently, our method is superior to all baselines, whether training takes place over
VG or MS-COCO.
Tab.
7 presents bounding box accuracy results for the WSG task.
Here, too, ourmethod outperforms the baseline methods.
In appendix B we present sample output maps.
For the WWbL task, we use the same metrics (pointing game and bounding-box accuracy).
For eachground-truth pair of bounding box and caption, we select the closest caption in CLIP space fromthe output predictions and compare the output map to the bounding box using the pointing accuracymetric.
In addition, bounding boxes are extracted for the heat-map, as described at 3.1, and comparedto the selected ground-truth bounding boxes with the same accuracy metric as above.
Tab.7 presents the results for the WWbL task.
In addition to our method (Alg. 1) that utilizes networkg to generate a heat-map, given an image and a text (line 8), we also test two alternative heatmapgeneration methods.
The first one is the MG method of Akbari et al.
[2] for the WSG task, and thesecond is the GAE explainability method [12].
As can be seen in the results listed in Tab. 7, ourproposed method obtains the best results among the three.
For comparison, we also provide WSGresults using Alg.
1 and the ground truth text for the same three heatmap methods.
Interestingly, theg that is trained on the VG dataset shows better performance for WSG, while in WWbL the resultsare mixed.
Fig 3 and Appendix D present sample WWbL results.
Ablation Study
In this section, we compare the accuracy of g with different modes of the lossfunction and present the experiment used to determine the values of λ 1 , 2 , 3 , 4 .
In addition, we also [Table] compare the performance of two backbone families, Resnet [29] and HarDnet [10].
All experimentswere conducted in the WSOL settings for the CUB[73] dataset.
Tab. 5 presents the sensitivity of our network g to each loss term, as evaluated on the test set, usingthe Resnet50 backbone.
The first part of the table compares the performance without L r map , usingdifferent combinations of the other loss terms.
As can be seen, the best performance is achieved whenwe use the three terms L fore , L back , L reg together.
Each term improves the performance:
L foreimproves it by 21%, L back by 5% and L reg by 48%.
The second part compares the performance with L rmap which improved by more than 6%.
As can beseen, each of the other terms improves the performance: L fore improves it by 4%, L back by 2%, and L reg by 5.5%.
Also, training only with L rmap decreases the performance by more than 9% comparedto performance using all loss terms.
The backbone ablation was conducted with two backbone families and is summarised in Tab. 4.
Forboth families, performance improves as the number of parameters is increased.
Appendix C, presentsthe sensitivity of the network g to the λ 3 coefficient, which is the weight of L rmap .
The methodseems to be largely insensitive to this parameter (it is stable for a wide range of values).
The sameoptimal set of hyperparameters used for WSOL is applied for WSG training, to avoid additionaltuning experiments. The state-of-the-art performance obtained without changing the hyperparametersis an indicator of the robustness of our method.
Discussion and limitations
When biological systems see, the input is seldom limited to single images.
Although our WWbLmethod was developed in the context of stills, it can be naturally extended to perform onlinelocalization of objects in video.
For this, the detected regions and their matching concepts need tobe tracked and must evolve over time, with the ability to correct previous hypotheses.
We leave thiseffort for future research, since benchmarking this ability with the current datasets is challenging.
The common computer vision terminology makes the distinction between object localization andobject detection.
The former aims to locate the main object of a certain type in the input image, whilethe latter aims to find all the relevant objects and their individual boundaries.
Our WWbL methodaims to extract every existing object and can, therefore, detect all image objects.
The method employs the selective search method [71].
This can be avoided by identifying one regionat a time and iterating the process.
Such a method is presented in Appendix A. It uses g to recover theobject map at each stage.
The results reported in this Appendix are somewhat lower than the methodbased on a selective search.
Conclusions
Through the power of large pre-trained transformers and by integrating explainability cues, we buildan effective weakly supervised phrase-grounding network g .
By combining an off-the-shelf imagecaptioning engine, we are able to identify and localize the objects within an input image.
The taskwe solve is an open-world one and the setting generalizes multiple existing tasks.
It is convincinglydemonstrated that our weakly supervised solution is on par with the fully supervised alternatives thatexist for the phrase-grounding task.
[-▷ Empty output dictionary 4: for i = 
-]