Original: L EARNING VISUAL - LINGUISTIC ADEQUACY , FIDELITY , AND FLUENCY FOR NOVEL OBJECT CAPTIONING
Revised: L EARNING VISUAL - LINGUISTIC ADEQUACY , FIDELITY , AND FLUENCY FOR NOVEL OBJECT CAPTIONING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Novel object captioning (NOC) learns image captioning models for describing objects or visual concepts which are unseen (i.e., novel ) in the training captions.
Revised: Novel object captioning (NOC) learns image captioning models for describing objects or visual concepts which are unseen (i.e., novel ) in the training captions.

Original: Such captioning models need to sufﬁciently describe such visual data with ﬂuent and natural language expression.
Revised: Such captioning models need to sufﬁciently describe such visual data with ﬂuent and natural language expression.

Original: In other words, we expect the produced captions being linguistically ﬂuent, containing novel objects of interest, and ﬁtting the visual concept of the image.
Revised: In other words, we expect the produced captions being linguistically ﬂuent, containing novel objects of interest, and ﬁtting the visual concept of the image.

Original: The above three aspects thus correspond to ﬂuency , ﬁdelity , and adequacy , respectively.
Revised: The above three aspects thus correspond to ﬂuency , ﬁdelity , and adequacy , respectively.

Original: However, most novel object captioning models are not explicitly designed to address the aforementioned properties due to the absence of caption annotations.
Revised: However, most novel object captioning models are not explicitly designed to address the aforementioned properties due to the absence of caption annotations.

Original: In this paper, we start by providing an insight into the relationship between the above properties and existing visual/language models.
Revised: In this paper, we start by providing an insight into the relationship between the above properties and existing visual/language models.

Original: Then, we present VLAF2 , for learning Visual-Linguistic Adequacy, Fidelity, and Fluency, which utilizes linguistics observed from captions for describing visual information of images with novel objects.
Revised: Then, we present VLAF2 , for learning Visual-Linguistic Adequacy, Fidelity, and Fluency, which utilizes linguistics observed from captions for describing visual information of images with novel objects.

Original: More speciﬁcally, we revisit
Revised: More speciﬁcally, we revisit

Original: BERT and CLIP, and explain how we leverage the intrinsic language knowledge from such popular models to reward captions with precise and rich visual content associated with novel images.
Revised: BERT and CLIP, and explain how we leverage the intrinsic language knowledge from such popular models to reward captions with precise and rich visual content associated with novel images.

Original: To validate the effectiveness of our framework, we conduct extensive experiments on the nocaps dataset.
Revised: To validate the effectiveness of our framework, we conduct extensive experiments on the nocaps dataset.

Original: Our method not only performs favorably against state-of-the-art novel captioning models in all caption evaluation metrics, but also surpasses the SPICE scores of human baseline.
Revised: Our method not only performs favorably against state-of-the-art novel captioning models in all caption evaluation metrics, but also surpasses the SPICE scores of human baseline.

Original: We perform quantitative and qualitative analysis to demonstrate how our model generates novel object captions with improved ﬂuency, ﬁdelity, and adequacy.
Revised: We perform quantitative and qualitative analysis to demonstrate how our model generates novel object captions with improved ﬂuency, ﬁdelity, and adequacy.

Original: Implementation details and code are available in the supplementary materials.
Revised: Implementation details and code are available in the supplementary materials.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Novel Object Captioning (NOC) (Agrawal et al., 2019) requires captioning models to accurately capture images containing novel objects unseen during training captions, and to describe such data with ﬂuent and grammatically correct sentences.
Revised: Novel Object Captioning (NOC) (Agrawal et al., 2019) requires captioning models to accurately capture images containing novel objects unseen during training captions, and to describe such data with ﬂuent and grammatically correct sentences.

Original: Despite impressive benchmark performance on
Revised: Despite impressive benchmark performance on

Original: COCO Captions (Chen et al., 2015) and Flickr (Young et al., 2014), existing image captioning models (Gao et al., 2019; Huang et al., 2019; Wang et al., 2019; Guo et al., 2020; Pan et al., 2020;
Revised: COCO Captions (Chen et al., 2015) and Flickr (Young et al., 2014), existing image captioning models (Gao et al., 2019; Huang et al., 2019; Wang et al., 2019; Guo et al., 2020; Pan et al., 2020;

Original: Cornia et al., 2020; Zhou et al., 2020) or unpaired image captioning (Gu et al., 2019;
Revised: Cornia et al., 2020; Zhou et al., 2020) or unpaired image captioning (Gu et al., 2019;

Original: Feng et al., 2019) generalize poorly to this task since the NOC task covers a larger variety of visual concepts, where nearly 400 object classes barely have any associated training captions.
Revised: Feng et al., 2019) generalize poorly to this task since the NOC task covers a larger variety of visual concepts, where nearly 400 object classes barely have any associated training captions.

Original: Existing work of novel object captioning typically rely on the object detection results to either ﬁll in the generated slotted sentences (Lu et al., 2018; Wu et al., 2018) or learn the visual vocabulary for novel objects directly (Hu et al., 2020).
Revised: Existing work of novel object captioning typically rely on the object detection results to either ﬁll in the generated slotted sentences (Lu et al., 2018; Wu et al., 2018) or learn the visual vocabulary for novel objects directly (Hu et al., 2020).

Original: However, these methods do not consider the semantics and linguistics of the entire sentences comprehensively.
Revised: However, these methods do not consider the semantics and linguistics of the entire sentences comprehensively.

Original: Speciﬁcally, most existing works do not exhibit the abilities in assuring the produced captions with correct and rich visual content, or with sufﬁcient natural and ﬂuent expression.
Revised: Speciﬁcally, most existing works do not exhibit the abilities in assuring the produced captions with correct and rich visual content, or with sufﬁcient natural and ﬂuent expression.

Original: In other words, the three aspects or challenges of NOC still need to be addressed.
Revised: In other words, the three aspects or challenges of NOC still need to be addressed.

Original: First of all, ﬂuency is expected in the output caption which is linguistically natural and grammatically correct.
Revised: First of all, ﬂuency is expected in the output caption which is linguistically natural and grammatically correct.

Original: Fidelity reﬂects the novel objects to be described, while adequacy encourages output captions to properly capture the visual concept of the input image.
Revised: Fidelity reﬂects the novel objects to be described, while adequacy encourages output captions to properly capture the visual concept of the input image.

Original: Unfortunately, the above properties cannot be easily achieved by standard image captioning models due to the absence of ground truth caption annotations.
Revised: Unfortunately, the above properties cannot be easily achieved by standard image captioning models due to the absence of ground truth caption annotations.

Original: In this paper, we revisit two popular pre-trained visual/language models of BERT (Devlin et al., 2018) and CLIP (Radford et al., 2021),
Revised: In this paper, we revisit two popular pre-trained visual/language models of BERT (Devlin et al., 2018) and CLIP (Radford et al., 2021),

Original: [Figure]
Revised: [Figure]

Original: and we provide the associated connection to ﬂuency, ﬁdelity, and adequacy for NOC.
Revised: and we provide the associated connection to ﬂuency, ﬁdelity, and adequacy for NOC.

Original: As shown in
Revised: As shown in

Original: Fig.
Revised: Fig.

Original: 1, we present VLAF2 for learning Visual-Linguistic Adequacy, Fidelity, and Adequacy, which leverages the intrinsic knowledge of BERT and CLIP for utilizing linguistics observed from captions for describing visual information of images with novel objects.
Revised: 1, we present VLAF2 for learning Visual-Linguistic Adequacy, Fidelity, and Adequacy, which leverages the intrinsic knowledge of BERT and CLIP for utilizing linguistics observed from captions for describing visual information of images with novel objects.

Original: With BERT pre-trained to excel at various linguistic tasks and CLIP to associate visual-language data at instance levels, we provide insights to these models and present technical details on how such models can be utilized for performing NOC, while the goals of adequacy, ﬁdelity and ﬂuency can be jointly achieved.
Revised: With BERT pre-trained to excel at various linguistic tasks and CLIP to associate visual-language data at instance levels, we provide insights to these models and present technical details on how such models can be utilized for performing NOC, while the goals of adequacy, ﬁdelity and ﬂuency can be jointly achieved.

Original: For the evaluation part of our work, we conduct extensive experiments on benchmark nocaps datasets, conﬁrming that our model is able to produce state-of-the-art results in terms of metrics linking to adequacy, ﬁdelity, and ﬂuency.
Revised: For the evaluation part of our work, we conduct extensive experiments on benchmark nocaps datasets, conﬁrming that our model is able to produce state-of-the-art results in terms of metrics linking to adequacy, ﬁdelity, and ﬂuency.

Original: In addition, by a variety of ablation studies, we further provide analysis on how BERT and CLIP are utilized and thus be preferable in solving NOC tasks.
Revised: In addition, by a variety of ablation studies, we further provide analysis on how BERT and CLIP are utilized and thus be preferable in solving NOC tasks.

Original:  C ONNECTING FLUENCY , FIDELITY AND ADEQUACY WITH B ERT AND C LIP
Revised:  C ONNECTING FLUENCY , FIDELITY AND ADEQUACY WITH B ERT AND C LIP

Original: We ﬁrst discuss how ﬂuency, ﬁdelity, and adequacy can be fundamentally and technically related to the visual/language models of BERT (Devlin et al., 2018) and CLIP (Radford et al., 2021), which will be utilized in our proposed learning framework.
Revised: We ﬁrst discuss how ﬂuency, ﬁdelity, and adequacy can be fundamentally and technically related to the visual/language models of BERT (Devlin et al., 2018) and CLIP (Radford et al., 2021), which will be utilized in our proposed learning framework.

Original: For caption ﬂuency, one expects that the image caption output to be linguistically natural and grammatically correct.
Revised: For caption ﬂuency, one expects that the image caption output to be linguistically natural and grammatically correct.

Original: It not only requires to capture the co-occurrence of novel-object vocabularies, the associated collocations such as verbs or modiﬁers are expected to be properly utilized.
Revised: It not only requires to capture the co-occurrence of novel-object vocabularies, the associated collocations such as verbs or modiﬁers are expected to be properly utilized.

Original: We deﬁne the context containing novel-object vocabularies as ˜y and the associated collocations as ˆy .
Revised: We deﬁne the context containing novel-object vocabularies as ˜y and the associated collocations as ˆy .

Original: Fluency can be deﬁned as how many collocations are observed by captioning models given a novel image context p (ˆy | ˜y) .
Revised: Fluency can be deﬁned as how many collocations are observed by captioning models given a novel image context p (ˆy | ˜y) .

Original: This is exactly the masked language modeling (MLM) objective adopted in BERT, without the log function explicitly calculated.
Revised: This is exactly the masked language modeling (MLM) objective adopted in BERT, without the log function explicitly calculated.

Original: This is the reason why we adopt BERT to learn the co-occurrence of novel-object vocabularies and their associated collocations to improve the linguistic ﬂuency.
Revised: This is the reason why we adopt BERT to learn the co-occurrence of novel-object vocabularies and their associated collocations to improve the linguistic ﬂuency.

Original: We now relate ﬁdelity and adequacy in image caption outputs to the model of CLIP.
Revised: We now relate ﬁdelity and adequacy in image caption outputs to the model of CLIP.

Original: We start by deﬁning objects appearing in the caption as X , and objects mentioned by the captions as Y .
Revised: We start by deﬁning objects appearing in the caption as X , and objects mentioned by the captions as Y .

Original: Relevant objects ( X , Y ) are deﬁned as objects that are both included in the images and described by the captions.
Revised: Relevant objects ( X , Y ) are deﬁned as objects that are both included in the images and described by the captions.

Original: Fidelity assesses whether the visual content presented in the produced caption is correct, and can be deﬁned as the fraction of relevant objects among objects in captions p ( X|Y )
Revised: Fidelity assesses whether the visual content presented in the produced caption is correct, and can be deﬁned as the fraction of relevant objects among objects in captions p ( X|Y )

Original: =
Revised: =

Original: p ( X , Y ) p ( Y ) .
Revised: p ( X , Y ) p ( Y ) .

Original: Similarly, adequacy evaluates whether sufﬁcient visual details have been expressed by captions, and we can deﬁne it as the fraction of relevant objects among objects in images p ( Y|X )
Revised: Similarly, adequacy evaluates whether sufﬁcient visual details have been expressed by captions, and we can deﬁne it as the fraction of relevant objects among objects in images p ( Y|X )

Original: =
Revised: =

Original: p ( X , Y ) p ( X ) .
Revised: p ( X , Y ) p ( X ) .

Original: Based on the observation of Friedman (2017), these two quantities p ( X|Y ) and p ( Y|X ) represent the association between X and Y .
Revised: Based on the observation of Friedman (2017), these two quantities p ( X|Y ) and p ( Y|X ) represent the association between X and Y .

Original: While CLIP is trained to associate image and text data at the instance level, it can be further applied for guiding image captioning models for NOC, boosting the desirable ﬁdelity and adequacy.
Revised: While CLIP is trained to associate image and text data at the instance level, it can be further applied for guiding image captioning models for NOC, boosting the desirable ﬁdelity and adequacy.

Original:  M ETHOD
Revised:  M ETHOD

Original: Before presenting the learning framework of VLAF2 for novel object captioning, we determine the notations and settings for the sake of completeness.
Revised: Before presenting the learning framework of VLAF2 for novel object captioning, we determine the notations and settings for the sake of completeness.

Original: Given a small set of caption-labeled images
Revised: Given a small set of caption-labeled images

Original: X l , the corresponding captions Y l , as well as a large set of uncaptioned images X u , our goal is to generate the associated captions ˆ Y u for X u using the captioning model C θ , where θ is the parameters of captioning model C .
Revised: X l , the corresponding captions Y l , as well as a large set of uncaptioned images X u , our goal is to generate the associated captions ˆ Y u for X u using the captioning model C θ , where θ is the parameters of captioning model C .

Original: To achieve this, we propose a two-stage learning framework of VLAF2, guided by two pre-trained visual-language models of BERT (Devlin et al., 2018) and CLIP (Radford
Revised: To achieve this, we propose a two-stage learning framework of VLAF2, guided by two pre-trained visual-language models of BERT (Devlin et al., 2018) and CLIP (Radford

Original: [Figure]
Revised: [Figure]

Original: Take Fig. 1(a) for example, when the captioning model generates a caption for a given image,
Revised: Take Fig. 1(a) for example, when the captioning model generates a caption for a given image,

Original: VLAF2 would regularize and substitute the verb wound for going , ensuring the corresponding linguistic ﬂuency.
Revised: VLAF2 would regularize and substitute the verb wound for going , ensuring the corresponding linguistic ﬂuency.

Original: Followed by next training stage in which we randomly samples multiple captions from the same image with reward captions properly designed, VLAF2 would further produce captions with improved ﬁdelity and adequacy.
Revised: Followed by next training stage in which we randomly samples multiple captions from the same image with reward captions properly designed, VLAF2 would further produce captions with improved ﬁdelity and adequacy.

Original: For example, in Fig. 1(b), the caption in the lower box accurately describes the object red scarf and thus receives a higher reward from CLIP.
Revised: For example, in Fig. 1(b), the caption in the lower box accurately describes the object red scarf and thus receives a higher reward from CLIP.

Original: This encourages the model to correctly describe the visual content and capture the associated visual concept of the input image.
Revised: This encourages the model to correctly describe the visual content and capture the associated visual concept of the input image.

Original: How the pre-trained models of BERT and CLIP would guide the learning of our
Revised: How the pre-trained models of BERT and CLIP would guide the learning of our

Original: VLAF2 will be detailed in the following subsections.
Revised: VLAF2 will be detailed in the following subsections.

Original: 3. D ESCRIBING NOVEL OBJECTS WITH LINGUISTIC FLUENCY
Revised: 3. D ESCRIBING NOVEL OBJECTS WITH LINGUISTIC FLUENCY

Original: By observing image-caption pairs ( x l , y l ) , the image captioning model in Fig. 1(a) would learn the visual grounding (i.e., localization of known objects and referring their expressions) as well the linguistics of captions.
Revised: By observing image-caption pairs ( x l , y l ) , the image captioning model in Fig. 1(a) would learn the visual grounding (i.e., localization of known objects and referring their expressions) as well the linguistics of captions.

Original: With uncaptioned images X u containing novel objects, we adopt BERT to reﬁne the wordings for captions containing novel objects, followed by CLIP to assess the quality of the resulting caption outputs.
Revised: With uncaptioned images X u containing novel objects, we adopt BERT to reﬁne the wordings for captions containing novel objects, followed by CLIP to assess the quality of the resulting caption outputs.

Original: While initialized by pre-trained models of VIVO (Hu et al., 2020), our captioning model in Fig.
Revised: While initialized by pre-trained models of VIVO (Hu et al., 2020), our captioning model in Fig.

Original: 1(a) would recognize images with novel objects but lack sufﬁcient ability in “describing” them in terms of captions.
Revised: 1(a) would recognize images with novel objects but lack sufﬁcient ability in “describing” them in terms of captions.

Original: To solve this problem, we impose a conventional sequence-to-sequence objective L s 2 s , which requires supervision of image-caption pairs ( x l , y l ) .
Revised: To solve this problem, we impose a conventional sequence-to-sequence objective L s 2 s , which requires supervision of image-caption pairs ( x l , y l ) .

Original: That is, we have
Revised: That is, we have

Original: [Equation]
Revised: [Equation]

Original: where ˆ y l = C θ ( x l ) denotes the predicted caption.
Revised: where ˆ y l = C θ ( x l ) denotes the predicted caption.

Original: As for uncaptioned images X u , while we do not observe ground truth caption for images containing novel objects, the collocations of the associated novel objects would be discovered via exploiting the intrinsic knowledge of BERT.
Revised: As for uncaptioned images X u , while we do not observe ground truth caption for images containing novel objects, the collocations of the associated novel objects would be discovered via exploiting the intrinsic knowledge of BERT.

Original: Thus, the linguistic ﬂuency of resulting captions can be further improved.
Revised: Thus, the linguistic ﬂuency of resulting captions can be further improved.

Original: We now detail the learning process for the aforementioned uncaptioned image data.
Revised: We now detail the learning process for the aforementioned uncaptioned image data.

Original: As illustrated in
Revised: As illustrated in

Original: Fig. 1, given an uncaptioned image x u , the captioning model C θ generates a caption ˆ y cu
Revised: Fig. 1, given an uncaptioned image x u , the captioning model C θ generates a caption ˆ y cu

Original: =
Revised: =

Original: C θ ( x u ) , ˆ y cu = { w c 1 , w c 2 , ..., w cT } , where w ci denotes the i th word and T is the caption length.
Revised: C θ ( x u ) , ˆ y cu = { w c 1 , w c 2 , ..., w cT } , where w ci denotes the i th word and T is the caption length.

Original: The superscript c represents it is generated by our captioning model.
Revised: The superscript c represents it is generated by our captioning model.

Original: We then obtain a masked caption ˆ y m u
Revised: We then obtain a masked caption ˆ y m u

Original: = { w c 1 , w c 2 , ..., w mM , ..., w cT } with m indicating the mask index, by randomly masking out the words in the caption.
Revised: = { w c 1 , w c 2 , ..., w mM , ..., w cT } with m indicating the mask index, by randomly masking out the words in the caption.

Original: We note that, we do not mask nouns in the above process, since they are viewed as relating to objects grounded in the visual content, and language models like BERT are not designed to handle such information.
Revised: We note that, we do not mask nouns in the above process, since they are viewed as relating to objects grounded in the visual content, and language models like BERT are not designed to handle such information.

Original: Finally, BERT takes the masked sequence ˆ y mu as input and recovers the masked word conditioned on the semantics of the entire sentence, producing the reﬁned caption ˆ y bu = BERT (ˆ y mu ) , ˆ y bu = { w c 1 , w c 2 , ..., w b M , ..., w c T } .
Revised: Finally, BERT takes the masked sequence ˆ y mu as input and recovers the masked word conditioned on the semantics of the entire sentence, producing the reﬁned caption ˆ y bu = BERT (ˆ y mu ) , ˆ y bu = { w c 1 , w c 2 , ..., w b M , ..., w c T } .

Original: [Figure]
Revised: [Figure]

Original: s a m p
Revised: s a m p

Original: [Figure]
Revised: [Figure]

Original: n g
Revised: n g

Original: [Figure]
Revised: [Figure]

Original: It is worth pointing out that, however, not every word substitution from BERT is semantically correct.
Revised: It is worth pointing out that, however, not every word substitution from BERT is semantically correct.

Original: This is the reason why we require CLIP to validate each replacement output.
Revised: This is the reason why we require CLIP to validate each replacement output.

Original: Speciﬁcally, if the reﬁned caption comprises a more accurate and associated word that human generally uses to describe the scene, then a higher CLIP score would be obtained (than that of the original caption).
Revised: Speciﬁcally, if the reﬁned caption comprises a more accurate and associated word that human generally uses to describe the scene, then a higher CLIP score would be obtained (than that of the original caption).

Original: Thus, we propose the objective function L BERT for learning wordings, which calculates the crossentropy loss for the replaced words, gated by the comparison of the CLIP scores of the two captions.
Revised: Thus, we propose the objective function L BERT for learning wordings, which calculates the crossentropy loss for the replaced words, gated by the comparison of the CLIP scores of the two captions.

Original: More precisely, L BERT is derived as:
Revised: More precisely, L BERT is derived as:

Original: [Equation]
Revised: [Equation]

Original: where CLIP ( x, y ) calculates the association between an image x and its caption y , indicating how well the captions match the images.
Revised: where CLIP ( x, y ) calculates the association between an image x and its caption y , indicating how well the captions match the images.

Original: More details can be referred to Radford et al.
Revised: More details can be referred to Radford et al.

Original: 3. L EARNING NOVEL OBJECT CAPTIONS WITH FIDELITY AND ADEQUACY
Revised: 3. L EARNING NOVEL OBJECT CAPTIONS WITH FIDELITY AND ADEQUACY

Original: Preserving the adequacy and ﬁdelity of captions for images with novel objects is another challenging task.
Revised: Preserving the adequacy and ﬁdelity of captions for images with novel objects is another challenging task.

Original: Recall that, as discussed in Sec. 2, ﬁdelity veriﬁes the correctness of visual content presented in the generated caption, while adequacy assesses whether sufﬁcient visual details have been expressed in it.
Revised: Recall that, as discussed in Sec. 2, ﬁdelity veriﬁes the correctness of visual content presented in the generated caption, while adequacy assesses whether sufﬁcient visual details have been expressed in it.

Original: Conventional sequence-to-sequence model training with cross-entropy loss might not reﬂect the above desirable properties.
Revised: Conventional sequence-to-sequence model training with cross-entropy loss might not reﬂect the above desirable properties.

Original: This is because that, models trained with word-by-word supervision tend to imitate the sentence patterns of the training images instead of relating caption data to the visual content, leading to less optimal caption generation.
Revised: This is because that, models trained with word-by-word supervision tend to imitate the sentence patterns of the training images instead of relating caption data to the visual content, leading to less optimal caption generation.

Original: To tackle the above issues, we utilize the captioning evaluation metric of CIDEr as the reward in our learning framework, since it encourages the generated caption to be consistent with that of the human annotated ones in the word level.
Revised: To tackle the above issues, we utilize the captioning evaluation metric of CIDEr as the reward in our learning framework, since it encourages the generated caption to be consistent with that of the human annotated ones in the word level.

Original: However, while CIDEr can be easily computed for captioning labeled images X l , it cannot be explicitly calculated for captioning uncaptioned images X u due to the absence of ground-truth captions.
Revised: However, while CIDEr can be easily computed for captioning labeled images X l , it cannot be explicitly calculated for captioning uncaptioned images X u due to the absence of ground-truth captions.

Original: Instead, we propose to optimize for ﬁdelity and adequacy of the generated captions via the association between images and captions calculated by CLIP, encouraging captions which precisely describe the objects with plentiful visual details, as discussed in
Revised: Instead, we propose to optimize for ﬁdelity and adequacy of the generated captions via the association between images and captions calculated by CLIP, encouraging captions which precisely describe the objects with plentiful visual details, as discussed in

Original: Sec. 2.
Revised: Sec. 2.

Original: However, we observe that captioning models would achieve improved association by simply repeating the same object that occurs in the image, which undermines the linguistic ﬂuency of the captions.
Revised: However, we observe that captioning models would achieve improved association by simply repeating the same object that occurs in the image, which undermines the linguistic ﬂuency of the captions.

Original: For example, the image caption “a group of cans of soda and other items on a table” can be replaced by “a pile of cans and bottles of soda on a counter with cans of cans ” with a higher
Revised: For example, the image caption “a group of cans of soda and other items on a table” can be replaced by “a pile of cans and bottles of soda on a counter with cans of cans ” with a higher

Original: CLIP score.
Revised: CLIP score.

Original: Therefore, we additionally impose a repetition penalty to avoid such trivial solutions.
Revised: Therefore, we additionally impose a repetition penalty to avoid such trivial solutions.

Original: We now describe how the rewards are calculated and how do we update the captioning models.
Revised: We now describe how the rewards are calculated and how do we update the captioning models.

Original: Rewards for the generated captions.
Revised: Rewards for the generated captions.

Original: In order to realize the above properties, we design the following rewards to reﬂect the quality of the generated captions.
Revised: In order to realize the above properties, we design the following rewards to reﬂect the quality of the generated captions.

Original: For image-caption pairs ( x l , y l ) , we directly calculate the CIDEr reward for the predicted caption ˆ y l (i.e., r CIDEr = CIDEr (ˆ y l , y l ) ).
Revised: For image-caption pairs ( x l , y l ) , we directly calculate the CIDEr reward for the predicted caption ˆ y l (i.e., r CIDEr = CIDEr (ˆ y l , y l ) ).

Original: To encourage the generated captions for X =
Revised: To encourage the generated captions for X =

Original: X l ∪ X u with ﬁdelity and adequacy, we exploit CLIP to compute the association between X =
Revised: X l ∪ X u with ﬁdelity and adequacy, we exploit CLIP to compute the association between X =

Original: X l ∪ X u and ˆ Y =
Revised: X l ∪ X u and ˆ Y =

Original: ˆ Y l ∪ ˆ Y u (i.e., r CLIP =
Revised: ˆ Y l ∪ ˆ Y u (i.e., r CLIP =

Original: CLIP ( x, ˆ y ) ).
Revised: CLIP ( x, ˆ y ) ).

Original: As for repetition penalty to preserve linguistic ﬂuency of the generated captions ˆ y u
Revised: As for repetition penalty to preserve linguistic ﬂuency of the generated captions ˆ y u

Original: [Equation]
Revised: [Equation]

Original: { w 1 , w 2 , ..., w T } for x u , we formulate it as a linear assignment problem, where every word is assigned to the most similar one in the sentence except for itself.
Revised: { w 1 , w 2 , ..., w T } for x u , we formulate it as a linear assignment problem, where every word is assigned to the most similar one in the sentence except for itself.

Original: Then, we calculate the similarity between such pairs for each sentence.
Revised: Then, we calculate the similarity between such pairs for each sentence.

Original: Intuitively, repetitive captions would have high similarity scores, since the repeated words will be assigned to the exact same words.
Revised: Intuitively, repetitive captions would have high similarity scores, since the repeated words will be assigned to the exact same words.

Original: We deﬁne the assignment ˆ α as the one that maximizes the average pairwise similarity of a sentence.
Revised: We deﬁne the assignment ˆ α as the one that maximizes the average pairwise similarity of a sentence.

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: where α ( i ) is the index of the word assigned to the i -th word in the caption, and C ( w i , w j ) is the cosine similarity between the GloVe (Pennington et al., 2014)
Revised: where α ( i ) is the index of the word assigned to the i -th word in the caption, and C ( w i , w j ) is the cosine similarity between the GloVe (Pennington et al., 2014)

Original: word representation of two words.
Revised: word representation of two words.

Original: Since a desirable captioning model would encourage captions with low repetition (i.e., low average pairwise similarity), the reward for repetition penalty is deﬁned as follows:
Revised: Since a desirable captioning model would encourage captions with low repetition (i.e., low average pairwise similarity), the reward for repetition penalty is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that we do not calculate repetition penalty for ˆ Y l since they are regularized by the aforementioned CIDEr rewards.
Revised: Note that we do not calculate repetition penalty for ˆ Y l since they are regularized by the aforementioned CIDEr rewards.

Original: With the above discussions, the total reward for caption-labeled data would be r (ˆ y l ) =
Revised: With the above discussions, the total reward for caption-labeled data would be r (ˆ y l ) =

Original: r CIDEr (ˆ y l , y l ) + r CLIP ( x l , ˆ y l ) , and the total reward for uncaptioned data would be r (ˆ y u ) =
Revised: r CIDEr (ˆ y l , y l ) + r CLIP ( x l , ˆ y l ) , and the total reward for uncaptioned data would be r (ˆ y u ) =

Original: r CLIP ( x u , ˆ y u ) + r rep (ˆ y u ) .
Revised: r CLIP ( x u , ˆ y u ) + r rep (ˆ y u ) .

Original: Back-propagation via reinforce algorithm.
Revised: Back-propagation via reinforce algorithm.

Original: Unfortunately, computation of the aforementioned rewards is non-differentiable.
Revised: Unfortunately, computation of the aforementioned rewards is non-differentiable.

Original: Thus, we adopt reinforce algorithm (Williams, 1992) to optimize the learning of our model.
Revised: Thus, we adopt reinforce algorithm (Williams, 1992) to optimize the learning of our model.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, for an image x we randomly sample the caption ˆ y s from the word distribution and use greedy decoding to obtain the baseline result ˆ y g .
Revised: 1, for an image x we randomly sample the caption ˆ y s from the word distribution and use greedy decoding to obtain the baseline result ˆ y g .

Original: If the sampled captions possess higher linguistic ﬂuency or cross-modal association than the baseline caption, they will be encouraged by positive rewards and vice versa.
Revised: If the sampled captions possess higher linguistic ﬂuency or cross-modal association than the baseline caption, they will be encouraged by positive rewards and vice versa.

Original: We follow Rennie et al.
Revised: We follow Rennie et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: and deﬁne the objective function as follows:
Revised: and deﬁne the objective function as follows:

Original: [Equation]
Revised: [Equation]

Original: where d indicates the source of the image, θ being the parameters of captioning model, and p θ (ˆ y s ) represents the predicted word logits for the generated captions.
Revised: where d indicates the source of the image, θ being the parameters of captioning model, and p θ (ˆ y s ) represents the predicted word logits for the generated captions.

Original: With the objective functions deﬁned in equations (1), (2), and (5), our captioning model C θ can be trained accordingly.
Revised: With the objective functions deﬁned in equations (1), (2), and (5), our captioning model C θ can be trained accordingly.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: Datasets & Implementation Details.
Revised: Datasets & Implementation Details.

Original: The training data for the nocaps benchmark comprises the
Revised: The training data for the nocaps benchmark comprises the

Original: Open Images V4 (Kuznetsova et al., 2020) object detection training set (1.7M images annotated with bounding boxes for 600 object classes), plus the image-caption pairs from the COCO Captions 2017 (Chen et al., 2015) training set (118K images containing 80 object classes).
Revised: Open Images V4 (Kuznetsova et al., 2020) object detection training set (1.7M images annotated with bounding boxes for 600 object classes), plus the image-caption pairs from the COCO Captions 2017 (Chen et al., 2015) training set (118K images containing 80 object classes).

Original: No additional image-caption pairs are provided for training.
Revised: No additional image-caption pairs are provided for training.

Original: We refer the images from the Open Images dataset to the uncaptioned images X u we deﬁne in Sec. 3, and the image-caption pairs from COCO Captions are deﬁned as ( X l , Y l ) .
Revised: We refer the images from the Open Images dataset to the uncaptioned images X u we deﬁne in Sec. 3, and the image-caption pairs from COCO Captions are deﬁned as ( X l , Y l ) .

Original: We only use the Open Images datasets during VIVO pre-training but leverage both datasets for training as described in Sections 3.1 and 3.2.
Revised: We only use the Open Images datasets during VIVO pre-training but leverage both datasets for training as described in Sections 3.1 and 3.2.

Original: We evaluate our model on the validation and test set of nocaps, which comprises 4500 and 10600 images from the Open Images validation and test sets, respectively.
Revised: We evaluate our model on the validation and test set of nocaps, which comprises 4500 and 10600 images from the Open Images validation and test sets, respectively.

Original: For the architecture of our captioning model, we follow (Hu et al., 2020; Li et al., 2020; Zhang et al., 2021) to use a BERT-base (Devlin et al., 2018) model.
Revised: For the architecture of our captioning model, we follow (Hu et al., 2020; Li et al., 2020; Zhang et al., 2021) to use a BERT-base (Devlin et al., 2018) model.

Original: As for CLIP and BERT, we directly exploit the pre-trained models released by their authors.
Revised: As for CLIP and BERT, we directly exploit the pre-trained models released by their authors.

Original: The architecture for BERT is BERT-Large, and the version for CLIP is ViT/B-32.
Revised: The architecture for BERT is BERT-Large, and the version for CLIP is ViT/B-32.

Original: Due to page limits, hyperparameters and other training details can be found in the Appendix A.
Revised: Due to page limits, hyperparameters and other training details can be found in the Appendix A.

Original: 4. E VALUATION METRICS
Revised: 4. E VALUATION METRICS

Original: CIDEr.
Revised: CIDEr.

Original: Similar to evaluation metrics (Papineni et al., 2002; Lin, 2004; Banerjee & Lavie, 2005) for NLP tasks, Consensus-based Image Description Evaluation (CIDEr) calculates the similarity
Revised: Similar to evaluation metrics (Papineni et al., 2002; Lin, 2004; Banerjee & Lavie, 2005) for NLP tasks, Consensus-based Image Description Evaluation (CIDEr) calculates the similarity

Original: [Table] [Table]
Revised: [Table] [Table]

Original: between the reference and generated caption by word n-gram overlap in a rule-based manner.
Revised: between the reference and generated caption by word n-gram overlap in a rule-based manner.

Original: To capture human consensus for image captioning evaluation, it introduces the tf-idf weight to reduce the matching weight of the n-grams that are common in all image captions.
Revised: To capture human consensus for image captioning evaluation, it introduces the tf-idf weight to reduce the matching weight of the n-grams that are common in all image captions.

Original: SPICE.
Revised: SPICE.

Original: Semantic Propositional Image Caption Evaluation (SPICE) (Anderson et al., 2016b) matches the semantics between sentences, such as objects, relations, and attributes of objects.
Revised: Semantic Propositional Image Caption Evaluation (SPICE) (Anderson et al., 2016b) matches the semantics between sentences, such as objects, relations, and attributes of objects.

Original: Speciﬁcally, it converts sentences into semantic scene graphs, which allows evaluation to break grammatical constraints and focuses on propositional semantic content.
Revised: Speciﬁcally, it converts sentences into semantic scene graphs, which allows evaluation to break grammatical constraints and focuses on propositional semantic content.

Original: It reﬂects the accuracy of the visual content and considers less about linguistic properties.
Revised: It reﬂects the accuracy of the visual content and considers less about linguistic properties.

Original: Fluency.
Revised: Fluency.

Original: To quantitatively evaluate ﬂuency, we remove the effect of the visual information and focus on the quality of linguistic properties in the conventional caption evaluation metrics.
Revised: To quantitatively evaluate ﬂuency, we remove the effect of the visual information and focus on the quality of linguistic properties in the conventional caption evaluation metrics.

Original: Speciﬁcally, we remove all the objects and nouns from the captions and report BLEU@4 (Papineni et al., 2002) and CIDEr scores calculated by the removed version of reference and candidate captions.
Revised: Speciﬁcally, we remove all the objects and nouns from the captions and report BLEU@4 (Papineni et al., 2002) and CIDEr scores calculated by the removed version of reference and candidate captions.

Original: Note that the ﬂuency experiment is conducted on a subset of the nocaps validation set, which containsimages whose caption annotations are available on the ofﬁcial website of the nocaps dataset.
Revised: Note that the ﬂuency experiment is conducted on a subset of the nocaps validation set, which containsimages whose caption annotations are available on the ofﬁcial website of the nocaps dataset.

Original: Fidelity & Adequacy.
Revised: Fidelity & Adequacy.

Original: Fidelity and adequacy evaluate how well the captions are associated with images.
Revised: Fidelity and adequacy evaluate how well the captions are associated with images.

Original: As deﬁned in Sec. 2, ﬁdelity stands for the fraction of relevant objects described in captions among all the objects in captions, and adequacy is the fraction of relevant instances that were retrieved.
Revised: As deﬁned in Sec. 2, ﬁdelity stands for the fraction of relevant objects described in captions among all the objects in captions, and adequacy is the fraction of relevant instances that were retrieved.

Original: These two properties are analogous to the deﬁnition of precision and recall, respectively.
Revised: These two properties are analogous to the deﬁnition of precision and recall, respectively.

Original: Therefore, we extract the objects mentioned in the captions and the ground-truth objects in the images and calculate the precision (for ﬁdelity), recall (for adequacy), and F1 (for overall association) scores.
Revised: Therefore, we extract the objects mentioned in the captions and the ground-truth objects in the images and calculate the precision (for ﬁdelity), recall (for adequacy), and F1 (for overall association) scores.

Original: The experiment is performed on the validation set of nocaps.
Revised: The experiment is performed on the validation set of nocaps.

Original: Following Agrawal et al.
Revised: Following Agrawal et al.

Original: (2019), we further split the dataset into three subsets for evaluation: indomain images only contain the seen objects that have been described in the training captions, outof-domain images only with unseen (i.e., novel) objects presented, and near-domain ones containing both seen and unseen objects.
Revised: (2019), we further split the dataset into three subsets for evaluation: indomain images only contain the seen objects that have been described in the training captions, outof-domain images only with unseen (i.e., novel) objects presented, and near-domain ones containing both seen and unseen objects.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /o
Revised: /o

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: 4. Q UANTITATIVE A NALYSIS
Revised: 4. Q UANTITATIVE A NALYSIS

Original: For performance comparisons, we choose UpDown (Agrawal et al., 2019) without SCST optimization (Rennie et al., 2017) and Oscar (Li et al., 2020) as baselines, as well as VinVL (Zhang et al., 2021) that achieves SOTA results on the benchmark of nocaps.
Revised: For performance comparisons, we choose UpDown (Agrawal et al., 2019) without SCST optimization (Rennie et al., 2017) and Oscar (Li et al., 2020) as baselines, as well as VinVL (Zhang et al., 2021) that achieves SOTA results on the benchmark of nocaps.

Original: In addition, VIVO (Hu et al., 2020) is a pre-training technique for captioning models, allowing them to recognize the novel objects.
Revised: In addition, VIVO (Hu et al., 2020) is a pre-training technique for captioning models, allowing them to recognize the novel objects.

Original: Since
Revised: Since

Original: VinVL did not report the numbers with Constrained Beam Search (CBS) exploited during inference (CBS is known to improve model performance on out-of-domain data), we reproduce VinVL following details stated in the original paper.
Revised: VinVL did not report the numbers with Constrained Beam Search (CBS) exploited during inference (CBS is known to improve model performance on out-of-domain data), we reproduce VinVL following details stated in the original paper.

Original: For more details please refer to Appendix A. For a comprehensive comparison, we conduct experiments on the validation and test set of nocaps.
Revised: For more details please refer to Appendix A. For a comprehensive comparison, we conduct experiments on the validation and test set of nocaps.

Original: In addition, we compare our method with VinVL in ﬂuency, ﬁdelity, and adequacy to demonstrate the improvement in terms of these properties.
Revised: In addition, we compare our method with VinVL in ﬂuency, ﬁdelity, and adequacy to demonstrate the improvement in terms of these properties.

Original: We also evaluate on the COCO Caption dataset and report in Appendix B.
Revised: We also evaluate on the COCO Caption dataset and report in Appendix B.

Original: The nocaps datasets.
Revised: The nocaps datasets.

Original: The results on nocaps are shown in Table 1.
Revised: The results on nocaps are shown in Table 1.

Original: From this table, we see that our model performed favorably against baselines and SOTAs across different metrics.
Revised: From this table, we see that our model performed favorably against baselines and SOTAs across different metrics.

Original: We observe that CBS slightly decreased model performance on seen object captioning, since it forces captioning models to describe the detected objects without considering detection correctness.
Revised: We observe that CBS slightly decreased model performance on seen object captioning, since it forces captioning models to describe the detected objects without considering detection correctness.

Original: Nevertheless, for near or out-of-domain images, CBS still beneﬁts the captioning performances.
Revised: Nevertheless, for near or out-of-domain images, CBS still beneﬁts the captioning performances.

Original: It is worth noting that our model largely increased the performance in SPICE score for every data domain, which veriﬁes that our method is able to generate captions with the improved image-language association.
Revised: It is worth noting that our model largely increased the performance in SPICE score for every data domain, which veriﬁes that our method is able to generate captions with the improved image-language association.

Original: Fluency, ﬁdelity, and adequacy.
Revised: Fluency, ﬁdelity, and adequacy.

Original: As described in Sec.
Revised: As described in Sec.

Original: 4.1, we design additional experiments for evaluating ﬂuency, ﬁdelity, and adequacy and report the numbers in Table 2.
Revised: 4.1, we design additional experiments for evaluating ﬂuency, ﬁdelity, and adequacy and report the numbers in Table 2.

Original: For ﬂuency, we remove all the objects and nouns in the captions since they relate less to the linguistics of the captions.
Revised: For ﬂuency, we remove all the objects and nouns in the captions since they relate less to the linguistics of the captions.

Original: We then calculate the BLEU@4 (B@4) and CIDEr (C) scores for the captions after removal.
Revised: We then calculate the BLEU@4 (B@4) and CIDEr (C) scores for the captions after removal.

Original: For ﬁdelity and adequacy, they indicate that captions should accurately (high precision) describe sufﬁcient (high recall) visual details.
Revised: For ﬁdelity and adequacy, they indicate that captions should accurately (high precision) describe sufﬁcient (high recall) visual details.

Original: Therefore, we report the object precision and recall in this table, and object F scores represent the overall association between captions and images.
Revised: Therefore, we report the object precision and recall in this table, and object F scores represent the overall association between captions and images.

Original: One can see that our method surpass previous methods by a visible margin on all tasks except for in-domain object precision., which further veriﬁes our model improves novel object captioning on ﬂuency, ﬁdelity, and adequacy.
Revised: One can see that our method surpass previous methods by a visible margin on all tasks except for in-domain object precision., which further veriﬁes our model improves novel object captioning on ﬂuency, ﬁdelity, and adequacy.

Original: 4. R EMARKS ON BERT AND CLIP FOR CAPTION FIDELITY , ADEQUACY AND FLUENCY
Revised: 4. R EMARKS ON BERT AND CLIP FOR CAPTION FIDELITY , ADEQUACY AND FLUENCY

Original: We conduct ablation studies on the nocaps validation set, with the aim to verify the necessity of integrating BERT and CLIP for learning NOC models.
Revised: We conduct ablation studies on the nocaps validation set, with the aim to verify the necessity of integrating BERT and CLIP for learning NOC models.

Original: Following the same evaluation procedures described in Sec. 4.1, we discuss the contributions of these models in terms linguistic and semantic level metrics in Tables 3 and 4.
Revised: Following the same evaluation procedures described in Sec. 4.1, we discuss the contributions of these models in terms linguistic and semantic level metrics in Tables 3 and 4.

Original: Detailed ablation analysis of every objective can be further found in
Revised: Detailed ablation analysis of every objective can be further found in

Original: Appendix B.2.
Revised: Appendix B.2.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Ours
Revised: Ours

Original: [Table]
Revised: [Table]

Original: BERT.
Revised: BERT.

Original: As shown in Table 3, the captioning model without BERT would observe a performance drop in CIDEr for linguistic ﬂuency, but such drops for the metric of SPICE (related to visual content) would be less signiﬁcant.
Revised: As shown in Table 3, the captioning model without BERT would observe a performance drop in CIDEr for linguistic ﬂuency, but such drops for the metric of SPICE (related to visual content) would be less signiﬁcant.

Original: Similarly, as observed in Table 4, removing BERT would result in the lowest BLEU and CIDEr scores.
Revised: Similarly, as observed in Table 4, removing BERT would result in the lowest BLEU and CIDEr scores.

Original: These results conﬁrm our motivation and model design discussed in Sec. 2, since BERT is utilized to improve caption quality at the linguistics level.
Revised: These results conﬁrm our motivation and model design discussed in Sec. 2, since BERT is utilized to improve caption quality at the linguistics level.

Original: CLIP.
Revised: CLIP.

Original: As shown in Table 3, the model without CLIP showed signiﬁcant drops in captioning metrics of CIDEr and SPICE.
Revised: As shown in Table 3, the model without CLIP showed signiﬁcant drops in captioning metrics of CIDEr and SPICE.

Original: However, the performance decrease in SPICE is expected, since CLIP is particularly deployed in our framework for preserving visual content in captions.
Revised: However, the performance decrease in SPICE is expected, since CLIP is particularly deployed in our framework for preserving visual content in captions.

Original: As for CIDEr, its decrease is mainly due to the deterioration of missing visual content in captions.
Revised: As for CIDEr, its decrease is mainly due to the deterioration of missing visual content in captions.

Original: This is also conﬁrmed by Table 4, in which mainly the metrics reﬂecting ﬁdelity and adequacy (i.e., at the visual semantics level) would observe signiﬁcant drops for model trained without CLIP rewards.
Revised: This is also conﬁrmed by Table 4, in which mainly the metrics reﬂecting ﬁdelity and adequacy (i.e., at the visual semantics level) would observe signiﬁcant drops for model trained without CLIP rewards.

Original: Repetition penalty.
Revised: Repetition penalty.

Original: Recall that, in Sec. 3.2, this penalty is to alleviate association between images and captions with redundant visual information.
Revised: Recall that, in Sec. 3.2, this penalty is to alleviate association between images and captions with redundant visual information.

Original: As seen in Table 3, the model without this penalty observed a signiﬁcant performance drop in CIDEr scores.
Revised: As seen in Table 3, the model without this penalty observed a signiﬁcant performance drop in CIDEr scores.

Original: We did not see such trends for SPICE.
Revised: We did not see such trends for SPICE.

Original: This is because that, repetitive words in captions mainly violate linguistic structures rather then semantic accuracy, and thus the performance related to linguistic ﬂuency would be more sensitive to the deployment of this penalty.
Revised: This is because that, repetitive words in captions mainly violate linguistic structures rather then semantic accuracy, and thus the performance related to linguistic ﬂuency would be more sensitive to the deployment of this penalty.

Original: 4. Q UALITATIVE A NALYSIS
Revised: 4. Q UALITATIVE A NALYSIS

Original: We now empirically show captions in Fig.
Revised: We now empirically show captions in Fig.

Original: 4, which are generated by our model and VinVL, with both pretrained from VIVO for novel object detection.
Revised: 4, which are generated by our model and VinVL, with both pretrained from VIVO for novel object detection.

Original: In this ﬁgure, wordings that are less accurate or incorrectly describe the associated visual content are marked in bold.
Revised: In this ﬁgure, wordings that are less accurate or incorrectly describe the associated visual content are marked in bold.

Original: And, our wording improvements are highlighted in red.
Revised: And, our wording improvements are highlighted in red.

Original: From this ﬁgure, one can see that for ﬂuency, our model generated vivid captions with more proper wordings.
Revised: From this ﬁgure, one can see that for ﬂuency, our model generated vivid captions with more proper wordings.

Original: Take the upper-left image for example, our model particularly described “fan hanging from the ceiling in a room” instead of “fan in a room” .
Revised: Take the upper-left image for example, our model particularly described “fan hanging from the ceiling in a room” instead of “fan in a room” .

Original: As for ﬁdelity, our model is designed to capture the visual content in an image.
Revised: As for ﬁdelity, our model is designed to capture the visual content in an image.

Original: Speciﬁcally, take the second image in the ﬁrst row for example, we correctly described the number of gloves and the novel object red scarf , while VinVL failed to do so.
Revised: Speciﬁcally, take the second image in the ﬁrst row for example, we correctly described the number of gloves and the novel object red scarf , while VinVL failed to do so.

Original: As for adequacy, take the bottom-right image for example, our model was able to recover visual details in the image (i.e., “people playing the accordion” and “sitting in a church” ).
Revised: As for adequacy, take the bottom-right image for example, our model was able to recover visual details in the image (i.e., “people playing the accordion” and “sitting in a church” ).

Original: For more qualitative examples, please refer to Appendix B.3.
Revised: For more qualitative examples, please refer to Appendix B.5.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Image captioning.
Revised: Image captioning.

Original: Recent progress of image captioning focuses on different model architectures and learning methods.
Revised: Recent progress of image captioning focuses on different model architectures and learning methods.

Original: Gao et al.
Revised: Gao et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: Pan et al.
Revised: Pan et al.

Original: ; Cornia et al.
Revised: ; Cornia et al.

Original: (2020) design different attention mechanisms for image captioning.
Revised: (2020) design different attention mechanisms for image captioning.

Original: Rennie et al.
Revised: Rennie et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2020) adopt reinforcement learning to improve the performance.
Revised: (2020) adopt reinforcement learning to improve the performance.

Original: On the other hand, some researchers consider more challenging settings, such as partially supervised (Liu et al., 2018; Kim et al., 2019) or unpaired image captioning (Gu et al., 2019;
Revised: On the other hand, some researchers consider more challenging settings, such as partially supervised (Liu et al., 2018; Kim et al., 2019) or unpaired image captioning (Gu et al., 2019;

Original: Feng et al., 2019).
Revised: Feng et al., 2019).

Original: However, these methods are restricted to the assumption that the unpaired images and captions share the same set of object class, and the number of object class is limited as well, which make them inapplicable to our task.
Revised: However, these methods are restricted to the assumption that the unpaired images and captions share the same set of object class, and the number of object class is limited as well, which make them inapplicable to our task.

Original: Novel object captioning.
Revised: Novel object captioning.

Original: Previously, novel object captioning approaches (Anderson et al., 2016a; 2018; Hendricks et al., 2016) were only tested on a restrictive dataset with only eight novel object classes held out from the COCO dataset.
Revised: Previously, novel object captioning approaches (Anderson et al., 2016a; 2018; Hendricks et al., 2016) were only tested on a restrictive dataset with only eight novel object classes held out from the COCO dataset.

Original: Their extensions to large-scale image data with various novel objects are not sufﬁciently studied.
Revised: Their extensions to large-scale image data with various novel objects are not sufﬁciently studied.

Original: Recent studies mainly rely on object detection results to improve the performance on novel object captioning.
Revised: Recent studies mainly rely on object detection results to improve the performance on novel object captioning.

Original: Lu et al.
Revised: Lu et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2018) generate slotted caption templates, which are later ﬁlled in with visual concepts identiﬁed by object detectors.
Revised: (2018) generate slotted caption templates, which are later ﬁlled in with visual concepts identiﬁed by object detectors.

Original: Yao et al.
Revised: Yao et al.

Original: (2017) exploits a copying mechanism to assemble words corresponding to object detector predictions to generate captions.
Revised: (2017) exploits a copying mechanism to assemble words corresponding to object detector predictions to generate captions.

Original: Similarly, Constrained Beam Search (CBS) (Anderson et al., 2016a) is an architecture-agnostic decoding algorithm that can be exploited during inference to enforce the inclusion of novel object classes in the captions.
Revised: Similarly, Constrained Beam Search (CBS) (Anderson et al., 2016a) is an architecture-agnostic decoding algorithm that can be exploited during inference to enforce the inclusion of novel object classes in the captions.

Original: Instead of explicitly using detection results, Hu et al.
Revised: Instead of explicitly using detection results, Hu et al.

Original: (2020) learns the relationship between image and text by aligning object detection tags with their corresponding image region features.
Revised: (2020) learns the relationship between image and text by aligning object detection tags with their corresponding image region features.

Original: Recently, Wang et al.
Revised: Recently, Wang et al.

Original: (2021) indicate that a desirable caption should comprise properties of ﬂuency, ﬁdelity, and adequacy.
Revised: (2021a) indicate that a desirable caption should comprise properties of ﬂuency, ﬁdelity, and adequacy.

Original: Nevertheless, most existing NOC approaches are not designed to handle language expression and cross-modal association with the above properties preserved.
Revised: Nevertheless, most existing NOC approaches are not designed to handle language expression and cross-modal association with the above properties preserved.

Original: Vision and Language Pre-training (VLP).
Revised: Vision and Language Pre-training (VLP).

Original: Existing VLP works can be classiﬁed into two streams.
Revised: Existing VLP works can be classiﬁed into two streams.

Original: One is based on the dual-encoder architecture, such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), utilizing features encoded in each modality followed by contrastive learning (Oord et al., 2018) for alignment purposes.
Revised: One is based on the dual-encoder architecture, such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), utilizing features encoded in each modality followed by contrastive learning (Oord et al., 2018) for alignment purposes.

Original: On the other hand, models like Zhou et al.
Revised: On the other hand, models like Zhou et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2020); Zhang et al.
Revised: (2020); Zhang et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2021) exploit multiple cross-attention layers to learn the relationship between images and text, and show impressive performances on image-text matching tasks.
Revised: (2021) exploit multiple cross-attention layers to learn the relationship between images and text, and show impressive performances on image-text matching tasks.

Original: Due to the efﬁciency of the former models in handling data across modalities, we adopt CLIP for cross-modal association in this paper.
Revised: Due to the efﬁciency of the former models in handling data across modalities, we adopt CLIP for cross-modal association in this paper.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: A visual-linguistic learning framework with improved adequacy, ﬁdelity, and ﬂuency (VLAF2) is presented for novel object captioning.
Revised: A visual-linguistic learning framework with improve d adequacy, ﬁdelity, and ﬂuency (VLAF2) is presented for novel object captioning.

Original: We fundamentally quantify the above properties and relate them to the models of BERT and CLIP.
Revised: We fundamentally quantify the above properties and relate them to the models of BERT and CLIP.

Original: We propose objectives and rewards reﬂecting the desirable linguistic ﬂuency and visual semantics for NOC.
Revised: We propose objectives and rewards reﬂecting the desirable linguistic ﬂuency and visual semantics for NOC.

Original: Guided by BERT, our model learns to reﬁne wordings of novel objects.
Revised: Guided by BERT, our model learns to reﬁne wordings of novel objects.

Original: Via reinforce algorithms, we have CLIP-based rewards assess the correctness of visual content described in the generated caption.
Revised: Via reinforce algorithms, we have CLIP-based rewards assess the correctness of visual content described in the generated caption.

Original: Empirically, we showed that our model achieved SOTA results on the nocaps benchmark.
Revised: Empirically, we showed that our model achieved SOTA results on the nocaps benchmark.

Original: We further provided analyses on both BERT and
Revised: We further provided analyses on both BERT and

Original: CLIP, verifying the necessity of their integration for learning novel object captions.
Revised: CLIP, verifying the necessity of their integration for learning novel object captions.

Original: A future direction of this work would be utilizing both captioned and uncaptioned images with self-supervised learning strategies for training NOC models.
Revised: A future direction of this work would be utilizing both captioned and uncaptioned images with self-supervised learning strategies for training NOC models.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: We acknowledge that all authors of this work have read and commit to adhering to the ICLR code of ethics.
Revised: We acknowledge that all authors of this work have read and commit to adhering to the ICLR code of ethics.

Original: Our work has no ethical concern.
Revised: Our work has no ethical concern.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: For reproducing our results in the experiment section, codes and training/testing scripts are provided in the supplementary materials
Revised: For reproducing our results in the experiment section, codes and training/testing scripts are provided in the supplementary materials

Original: A I MPLEMENTATION DETAILS
Revised: A I MPLEMENTATION DETAILS

Original: Following Hu et al.
Revised: Following Hu et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021), we consider a BERT-base (Devlin et al., 2018) architecture for our captioning model.
Revised: (2021), we consider a BERT-base (Devlin et al., 2018) architecture for our captioning model.

Original: Given an image, the captioning model jointly takes the image region features and the predicted detection tags to generate the associated caption.
Revised: Given an image, the captioning model jointly takes the image region features and the predicted detection tags to generate the associated caption.

Original: We use the same region features as VinVL (Zhang et al., 2021), which are released on their project page.
Revised: We use the same region features as VinVL (Zhang et al., 2021), which are released on their project page.

Original: Since the object detection model Omni-detection used in previous works (Hu et al., 2020;
Revised: Since the object detection model Omni-detection used in previous works (Hu et al., 2020;

Original: Zhang et al., 2021) is not available, we replace it with a publicly available model of TSD (Song et al., 2020) to generate the object detection tag.
Revised: Zhang et al., 2021) is not available, we replace it with a publicly available model of TSD (Song et al., 2020) to generate the object detection tag.

Original: Reproducing our method.
Revised: Reproducing our method.

Original: We perform VIVO (Hu et al., 2020) pre-training for 100 epochs with a batch size of 1024 and a learning rate of 5 × 10 − 5 , which are exactly the same as the parameters stated in the VIVO paper.
Revised: We perform VIVO (Hu et al., 2020) pre-training for 100 epochs with a batch size of 1024 and a learning rate of 5 × 10 − 5 , which are exactly the same as the parameters stated in the VIVO paper.

Original: After that, we propose to train our model following the training process described in Algorithm 1 to learn to caption novel objects with linguistic ﬂuency.
Revised: After that, we propose to train our model following the training process described in Algorithm 1 to learn to caption novel objects with linguistic ﬂuency.

Original: We train our model for 20 epochs with an effective batch size of 512 (256 caption-labeled images and 256 uncaptioned images) and a learning rate of 1 . 5 × 10 − 5 .
Revised: We train our model for 20 epochs with an effective batch size of 512 (256 caption-labeled images and 256 uncaptioned images) and a learning rate of 1 . 5 × 10 − 5 .

Original: Then, to learn novel object captions with ﬁdelity and adequacy, we train our model as decsribed in Algorithm 2.
Revised: Then, to learn novel object captions with ﬁdelity and adequacy, we train our model as decsribed in Algorithm 2.

Original: Speciﬁcally, we train our model forepochs with an effective batch size of 128 (64 caption-labeled images and 64 uncaptioned images) and a learning rate of 2 .
Revised: Speciﬁcally, we train our model forepochs with an effective batch size of 128 (64 caption-labeled images and 64 uncaptioned images) and a learning rate of 2 .

Original: 5 × 10 − 6 .
Revised: 5 × 10 − 6 .

Original: We use 8 V100 GPUs to perform the above training algorithms.
Revised: We use 8 V100 GPUs to perform the above training algorithms.

Original: Codes can be found in the supplementary materials.
Revised: Codes can be found in the supplementary materials.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Reproducing baseline methods.
Revised: Reproducing baseline methods.

Original: For VinVL (Zhang et al., 2021)
Revised: For VinVL (Zhang et al., 2021)

Original: [Table]
Revised: [Table]

Original: we leverage the released model on their project page and directly inference on the nocaps dataset.
Revised: we leverage the released model on their project page and directly inference on the nocaps dataset.

Original: However, for VinVL+VIVO (Zhang et al., 2021)
Revised: However, for VinVL+VIVO (Zhang et al., 2021)

Original: [Table]
Revised: [Table]

Original: since the pre-trained model is not publicily available
Revised: since the pre-trained model is not publicily available

Original: [Table]
Revised: [Table]

Original: we reproduce this method using the image region features and object detection tags generated by models mentioned in the beginning of this section to train this model.
Revised: we reproduce this method using the image region features and object detection tags generated by models mentioned in the beginning of this section to train this model.

Original: Speciﬁcally, the model is trained for 160K iterations (aboutepochs) with a batch size of 1024 and a learning rate of 5 × 10 − 5 , and ﬁne-tuned for 30 epochs with a batch size of 256 and a learning rate of 5 × 10 − 5 using the cross-entropy loss.
Revised: Speciﬁcally, the model is trained for 160K iterations (aboutepochs) with a batch size of 1024 and a learning rate of 5 × 10 − 5 , and ﬁne-tuned for 30 epochs with a batch size of 256 and a learning rate of 5 × 10 − 5 using the cross-entropy loss.

Original: Last, we perform the SCST optimization (Rennie et al., 2017) with a learning rate of 2 × 10 − 6 for 5 epochs to obtain the ﬁnal model.
Revised: Last, we perform the SCST optimization (Rennie et al., 2017) with a learning rate of 2 × 10 − 6 for 5 epochs to obtain the ﬁnal model.

Original: The numbers reported in Table 1 are derived using this version of model.
Revised: The numbers reported in Table 1 are derived using this version of model.

Original: B A DDITIONAL EXPERIMENTS B. E XPERIMENTS ON THE COCO C APTION DATASET
Revised: B A DDITIONAL EXPERIMENTS B. E XPERIMENTS ON THE COCO C APTION DATASET

Original: To validate that our method generalize well on the task of describing the seen objects, we conduct experiments on the COCO Caption test set and report the numbers in Table 5.
Revised: To validate that our method generalize well on the task of describing the seen objects, we conduct experiments on the COCO Caption test set and report the numbers in Table 5.

Original: The training data for
Revised: The training data for

Original: VinVL (Zhang et al., 2021) is image caption pairs from the COCO (Lin et al., 2014) dataset.
Revised: VinVL (Zhang et al., 2021) is image caption pairs from the COCO (Lin et al., 2014) dataset.

Original: While for VinVL + VIVO and our method, we additionally leverage the uncaptioned image from the Open
Revised: While for VinVL + VIVO and our method, we additionally leverage the uncaptioned image from the Open

Original: Images (Kuznetsova et al., 2020) dataset as extra data.
Revised: Images (Kuznetsova et al., 2020) dataset as extra data.

Original: One can see that our method outperforms the other competitive approaches on different metrics which veriﬁes the effectiveness of our approach.
Revised: One can see that our method outperforms the other competitive approaches on different metrics which veriﬁes the effectiveness of our approach.

Original: B. D ETAILED ABLATION ANALYSIS
Revised: B. D ETAILED ABLATION ANALYSIS

Original: Table 6 lists the performances and compares contributions of the imposed objectives in our VLAF2.
Revised: Table 6 lists the performances and compares contributions of the imposed objectives in our VLAF2.

Original: The baseline model in Table 6 is only trained on the COCO Caption dataset using the sequence-tosequence objective.
Revised: The baseline model in Table 6 is only trained on the COCO Caption dataset using the sequence-tosequence objective.

Original: To conﬁrm our introduction of exploiting BERT to learn the associated wordings of novel images, we apply L BERT to the baseline model, and report the results in the second row of Table 6.
Revised: To conﬁrm our introduction of exploiting BERT to learn the associated wordings of novel images, we apply L BERT to the baseline model, and report the results in the second row of Table 6.

Original: The CIDEr scores improve signiﬁcantly after adopting reinforce algorithm (Williams, 1992) and using CIDEr scores of the generated captions as reward, and the results are in the third row.
Revised: The CIDEr scores improve signiﬁcantly after adopting reinforce algorithm (Williams, 1992) and using CIDEr scores of the generated captions as reward, and the results are in the third row.

Original: One can see that the SPICE scores largely increase but the CIDEr scores slightly decrease after the deployment of CLIP.
Revised: One can see that the SPICE scores largely increase but the CIDEr scores slightly decrease after the deployment of CLIP.

Original: We hypothesize that the captioning model properly captures the visual content in images, but it describes the scene with poor linguistic ﬂuency.
Revised: We hypothesize that the captioning model properly captures the visual content in images, but it describes the scene with poor linguistic ﬂuency.

Original: As the discussion in
Revised: As the discussion in

Original: Sec. 3.2, we attribute the performance drop to the degenerate solution of increasing the association between the captions and the corresponding images.
Revised: Sec. 3.2, we attribute the performance drop to the degenerate solution of increasing the association between the captions and the corresponding images.

Original: Note that we further consider the repetition penalty to regularize the captioning model.
Revised: Note that we further consider the repetition penalty to regularize the captioning model.

Original: The results are shown in the last row of Table 6.
Revised: The results are shown in the last row of Table 6.

Original: One can see that this regularization slightly improves the SPICE scores but signiﬁcantly increase the CIDEr scores.
Revised: One can see that this regularization slightly improves the SPICE scores but signiﬁcantly increase the CIDEr scores.

Original: By comparing the performances listed in Table 6, we see that the full version of our VLAF achieved the best performance in terms of CIDEr and SPICE.
Revised: By comparing the performances listed in Table 6, we see that the full version of our VLAF achieved the best performance in terms of CIDEr and SPICE.

Original: Thus, the design of our VLAF2 can be successfully veriﬁed. 
Revised: Thus, the design of our VLAF2 can be successfully veriﬁed. [Table] [Table]

Revised: B. E XPERIMENTS ON THE NOCAPS (XD) BENCHMARK

Revised: To investigate the limits of performance on nocaps without any restraints on the training datasets

Original: [Table]
Revised: [Table]

Revised: we conduct experiments on the nocaps (XD) benchmark to verify the effectiveness of our method when more image-caption pairs are considered.

Original: VinVL
Revised: Speciﬁcally, we additionally consider Conceptual Captions (CC) (Sharma et al., 2018) as labeled training samples X l in our learning framework and perform joint-training to see if extra image-caption pairs beneﬁt the model on novel object captioning. The results are shown in Table 7. Note that both VIVO, SimVLM and our method adopt

Revised: BERT-based architecture as the backbones for captioning.

Original: Ours
Revised: As for the training set, since VIVO (Hu et al., 2020) did not specify the dataset details for their evaluation for the nocaps (XD), we compared our method to SimVLM, which applied a much larger web-scale dataset (1.8B image-text pairs) than the CC dataset (3.1M pairs). Yet, our method still performs favorably against SOTAs on the nocaps (XD) protocol and benchmark, verifying the effectiveness of our method even if more image-caption pairs are considered. In addition, to quantitatively show that the performance gain in Table 7 is not simply contributed by the additional data we considered, we ablate our model on the nocaps validation set and show the results in Table 8. We observed a similar performance trend as we reported in Table 6, where L BERT slightly improves the CIDEr scores, and r CLIP siginiﬁcantly boost SPICE but slightly deteriorates the CIDEr scores.

Original: VinVL
Revised: One can see that the regularization r rep slightly improves the SPICE scores but signiﬁcantly increase the CIDEr. By comparing the performances listed in Table 6 and Table 8, we see that our design of distilling BERT to enhance ﬂuency (in terms of CIDEr) and the uses of CLIP to encourage captions with sufﬁcient ﬁdelity and adequacy (in terms of SPICE) still function properly when more diverse image-caption pairs are considered, verifying the design of our VLAF2. B. H UMAN STUDY

Original: Ours
Revised: To conduct human study, we randomly picked 60 images from the nocaps validation set, and compared the captions generated by our method to those generated by the SOTA of VinVL+VIVO (Zhang et al., 2021), and the human-annotated captions provided by the nocaps dataset. Following the evaluation protocols used in the COCO Captioning Challenge 2015 (Lin et al., 2014), we designed 4 different metrics and asked individuals to evaluate captions from these aspects. The following are the four metrics we used in the experiment: M1:

Revised: Is the caption generated

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: of the captions on a scale 1-5 (lack of ﬂuency-very ﬂuent): Whether the caption use phrases/words that human generally would use to describe the scene, i.e., the caption is linguistically natural and ﬂuent.

Revised: Speciﬁcally, M2, M3, M4 correspond to the ﬁdelity, adequacy, and ﬂuency, respectively, which are the particular objectives desired to be achieved.

Revised: We asked 24 people two answer 6 different questionnaires, and each questionnaire contains 10 captions from each method (i.e., ours, sota, and human caption presented in a random order).

Original:  B. M ORE QUALITATIVE RESULTS
Revised: We report the results in Table 9. We see that our method surpassed the SOTA by clear margins, while our performances were comparable to those the human ones across different metrics. This further supports the design of our model for NOC with sufﬁcient ﬂuency, ﬁdelity, and adequacy. B. M ORE QUALITATIVE RESULTS

Original: Qualitative comparison on ﬂuency, ﬁdelity and adequacy.
Revised: Qualitative comparison on ﬂuency, ﬁdelity and adequacy.

Original: In this part, we provide more qualitative results on the nocaps validation/test set, and the results are shown in Fig.
Revised: In this part, we provide more qualitative results on the nocaps validation/test set, and the results are shown in Fig.

Original: 5 and 6.
Revised: 5 and 6.

Original: Note that wordings that are less accurate or incorrectly describe the associated visual content are marked in bold.
Revised: Note that wordings that are less accurate or incorrectly describe the associated visual content are marked in bold.

Original: And, our wording improvements are highlighted in red.
Revised: And, our wording improvements are highlighted in red.

Original: Take results in the bottom row of Fig.
Revised: Take results in the bottom row of Fig.

Original: 5 for example.
Revised: 5 for example.

Original: For the column of ﬂuency, our model particularly described the turtle being “crawling on some rocks” instead of “sitting on the top of a beach” .
Revised: For the column of ﬂuency, our model particularly described the turtle being “crawling on some rocks” instead of “sitting on the top of a beach” .

Original: For ﬁdelity, our model predicted the background  as “race track” instead of “street” from the prediction of VinVL model.
Revised: For ﬁdelity, our model predicted the background preferably as “race track” instead of “street” from the prediction of VinVL model.

Original: As for the column of adequacy, though both captions described a young men running, our model successfully captures more details in the image (i.e., “there are number on their shirts” ).
Revised: As for the column of adequacy, though both captions described a young men running, our model successfully captures more details in the image (i.e., “there are number on their shirts” ).

Original: For more qualitative results, please refer to Fig.
Revised: For more qualitative results, please refer to Fig.

Original: Qualitative results of some failure cases.
Revised: Qualitative results of some failure cases.

Original: In this part, we demonstrate some failure cases of our
Revised: In this part, we demonstrate some failure cases of our

Original: VLAF2 model.
Revised: VLAF2 model.

Original: We empirically observe that the failure cases mainly come from the wrong/missing detection tags predicted by the pre-trained object detectors.
Revised: We empirically observe that the failure cases mainly come from the wrong/missing detection tags predicted by the pre-trained object detectors.

Original: To be more speciﬁc, the captioning model largely relies on the detection tags as clues to correctly describe novel objects.
Revised: To be more speciﬁc, the captioning model largely relies on the detection tags as clues to correctly describe novel objects.

Original: Take the result in the left-side of Fig.
Revised: Take the result in the left-side of Fig.

Original: 7 for example, the detection model falsely recognizes the raccoon as a squirrel, and this detection result consequently damages the caption prediction.
Revised: 7 for example, the detection model falsely recognizes the raccoon as a squirrel, and this detection result consequently damages the caption prediction.

Original: Therefore, how to jointly improve the detection model and captioning model is still a open question, and we leave this problem for future research.
Revised: Therefore, how to jointly improve the detection model and captioning model is still a open question, and we leave this problem for future research.

Original: For more failure cases, please refer to Fig.
Revised: For more failure cases, please refer to Fig.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]