Paper: FlexCap: Generating Rich, Localized, and  Flexible Captions in Images (ICLR-2024)

Reviewer comment (Experiments):
Another concern is the precision or accuracy of the proposed large scale dataset. The authors claimed they use the filtered n-grams as text queries for pre-trained region proposal models (i.e. OWL- ViT (Minderer et al., 2022)) to extract boxes and select text-box pairs based on the similarity score (> 0.1). Therefore, the upper bound of precision of this dataset is determined by OWL- ViT, this might restrict the downstream tasks on the dataset.

Author response:
There is an inherent trade-off between precise, small-scale, well curated data (e.g. COCO captions) and large-scale diverse but noisy data (e.g. CLIP training set). The existing research on full image captioning (e.g. CLIP, CoCa, CAPPA) shows that large-scale noisy data could be more effective even if it is noisy data, mainly because it has diversity and enables training at scale. In this work we also choose to explore using a large-scale albeit noisy dataset as we want to generate a rich caption for any region in an image. While there will be a certain level of noise when using internet caption data, we find we can still train models with strong object localization using this data as our results demonstrate.

Reviewer comment (Theory):
Why is the 'length conditioning' works? The length conditioning is implemented simply by adding an additional token that indicates the desired length of the output caption, and there is no training constrains. How to guarantee the additional token constrains the length?

Author response:
We train the entire model including the text decoder from scratch for the task of length conditioned caption generation. For dataset generation, we associate n-gram textual captions with bounding boxes (refer fig 3). During training, the length token is prefixed to each textual caption to indicate to the model the desired length of the caption. The training constraint of adhering to the length conditioning exists in the loss function. Throughout training the model needs to predict outputs of a certain length and then “<eos>” token (end-of-sentence) or it will result in a high loss. 

In Section 5.3, We measure how often the trained model complies to the length constraint. We observe that if we train with length conditioning the model complies with the desired length most of the time (97 out of 100). The detailed statistics on compliance can be viewed in Table 4. Our length-conditioning claims are mainly backed with strong empirical evaluations.  

Let us start with our training data. In our dataset, for each box there exist multiple captions of varying lengths. In Fig 3, for the bounding box enclosing the dog there are 3 captions: dog (length 1), brown dog (length 2) and brown dog playing (length 3). To make it clear to the model that we expect captions of different lengths, we produce 3 ground truth sequences: [$Length_1$, dog,  <eos>], [$Length_2$, brown, dog, <eos>], [$Length_3$, brown, dog, playing <eos>]. The model is now trained to predict the caption given the image, bounding box, and the length conditioning prompt. If the model predicts a sentence that is longer than one word when conditioned with $Length_1$ token, the model incurs a high loss. The model will only incur a low loss if it learns to predict <eos> after a caption of length 1. If the caption is longer or shorter it incurs a high loss. 

Given the 32B triplets of image-box-caption as training data, the model sees a variety of captions of varying lengths. In Fig 11, we show the length distribution in the dataset which shows the diversity of length in our captions. 

After training exclusively on this objective, the model learns to predict captions by choosing the appropriate amount of  description for that particular length (as indicated by the length conditioning token). In Fig 5, we get a sense how the model is achieving this goal. Sometimes the model incorporates articles (elephant, *an* elephant), adjectives (cat, *typing* cat), verbs (computer with cat, cat *sleeping* on a computer), and more contextual information (bird kite in blue sky, colorful *flamingo* *beach* kite) to produce length-conditioned captions.

Reviewer comment (Novelty):
In introduction, the authors claim that they diverge from Flamingo and BLIP2 (which use latent representations to connect LLMs), and use textual representation of images. However, the idea of using textual description of images to prompt LLMs for vision-language tasks is not new.

Author response:
We agree with the reviewer that “using textual description of images to prompt LLMs for vision-language tasks” is not new. What is new in our work is providing more informative textual inputs in the form of localized captions produced in a controllably-rich manner. To achieve this, we also had to create a large-scale dataset of localized captions that enables the training of such a model. We will clarify this in the paper.

Reviewer comment (Experiments):
The major contribution, comparing to prior works, is producing a desired length of caption. Thus a baseline is missing: Simply use a caption model to predict (1) whole image description, (2) a detection model to predict the object in the box, and prompt a LLM conditioned on (1) and (2) to generate desired length of captions. This baseline circumvents the collecting of a new dataset and training a model like FlexCap.

Author response:
We would like to point out that besides producing a caption of desired length, a key contribution of our work is producing a localized caption that describes the content in any given bounding box using the context of the whole image. The captioning model needs to describe objects and regions of all sizes in the image. 

We thank the reviewer for this suggestion. In theory, this is a sound approach. However, we found that the current leading image captioning models (such as BLIP2) do not recover all the objects in an image. We experimented initially with this approach while trying to localize objects with captions generated by VLMs. However image captioning models usually only generate captions mentioning salient objects in an image. This is what inspired us to work on a model that generates descriptions for any region in an image instead of matching existing descriptions with regions.

As an example we ran BLIP2 on the same images for which we have FlexCap results here:

* [Image 1](https://flex-cap.github.io/images/00.html):

    * BLIP2 captioner returns the following output: *two people walking down a sidewalk.*
    * BLIP2 VQA with the prompt “List all the objects in the image.” returns the following output: *A man walking down the sidewalk, a man on a skateboard, a man on a bike, a man on a bench, a man on.*
    * Objects that FlexCap described in the image (subset shown here): *brick-paved street, a black trash can, window, orange jacket, trees lining the street, blue jeans, a man is wearing a orange jacket, a bench on the sidewalk, a yellow car, red and white sign on the side of the building, a white pole on the sidewalk, a white street sign, a black bag on a man’s shoulder, car on the street.*

* [Image 2](https://flex-cap.github.io/images/09.html):

    * BLIP2 captioner returns the following output: *a group of people playing a video game.*
    * BLIP2 VQA with “List all the objects in the image.” returns the following output: *people, a dog, a cat, a tv, a computer, a couch, a window, a door, a chair, a table.*
    * Objects that FlexCap described in the image (subset shown here): *red shirt on a woman, brown couch, wii controller, watch, black glasses, cap, magazines, framed picture on the wall, white blinds on window, papers on table, a watch on man’s wrist, a pair of black shorts, remote strap, a stuffed animal on the couch, a stack of magazines, a framed picture on the wall.*

As the first part of the suggested approach misses on detecting a large number of objects, an LLM will be unable to produce reasonable descriptions of varying lengths for all objects in the image.

Reviewer comment (Experiments):
In fact, I am wondering if other carefully filtered/designed localized captions could better prompt the LLM for those tasks. The work fails to compare or propose other baselines to demonstrate the superiority of FlexCap.

Author response:
We thank the reviewer for this feedback. We are also optimistic about using better localized captions to prompt LLMs. In Section 5, we explore how FlexCap’s outputs (localized descriptions) can be used to prompt existing LLMs for various VQA tasks. 
We also compare the usefulness of such localized captions for downstream tasks against other baselines such as Flamingo, BLIPv2 and ViperGPT. We think these are strong baselines that use LLMs in different ways for VQA tasks. We are happy to include comparisons with any particular baseline that the reviewer has in mind.

Reviewer comment (Reproducibility):
Will the pre-trained model (weights) be released?

Author response:
We are actively looking into ways to release the model weights.