ECOL - R : Encouraging Copying in Novel Object Captioning with Reinforcement Learning (2021)

Abstract: Novel Object Captioning is a zero-shot Image Captioning task requiring describing objects not seen in the training captions, but for which information is available from external object detectors. The key challenge is to select and describe all salient detected novel objects in the input images. In this paper, we focus on this challenge and propose the ECOL-R model (Encouraging Copying of Object Labels with Reinforced Learning), a copy-augmented transformer model that is encouraged to accurately describe the novel object labels. This is achieved via a specialised reward function in the SCST reinforcement learning framework (Rennie et al., 2017) that encourages novel object mentions while maintaining the caption quality. We further restrict the SCST training to the images where detected objects are mentioned in reference captions to train the ECOL-R model. We additionally improve our copy mechanism via Abstract Labels, which transfer knowledge from known to novel object types, and a Morphological Selector, which determines the appropriate inflected forms of novel object labels. The resulting model sets new state-of-the-art on the nocaps (Agrawal et al., 2019) and held-out COCO (Hendricks et al., 2016) benchmarks.

## introduction novel object captioning is a zero-shot image captioning task where the captions should mention novel objects (i.e., not seen in the training captions), but for which information is available from external object detectors. to produce high-quality captions, the captioning models should select and describe all salient detected objects and avoid mentioning minor or irrelevant details in the input images. as shown in figure 1 , caption a is the best caption among the three because a mentions all salient objects in the images without any unnecessary details while b mentions bread which is just a figure 1 : caption a is the ground-truth caption for the image. compared with b and c, a is the best caption because it mentions all salient objects (i.e, hamburger, french fries and drinks). we use abstract labels, that is hypernyms of the objects' detected object labels in the object representations, transferring knowledge from the objects seen in the training captions to novel objects. our copy mechanism also selects appropriate inflected forms of object labels (i.e., hamburgers vs. hamburger). we use an advanced copy mechanism, similar to the one in @xcite , to effectively integrate novel objects. we follow the setup in @xcite and use two object detectors: one providing rich object visual features and another providing task specific (including novel) object labels as copy candidates. our preliminary experiments show that the copy mechanism is infrequently triggered and unable to mention many salient objects in the input images. we propose the ecol-r model (encouraging copying of object labels with reinforced learning), a copy-augmented transformer model trained in the self-critical sequence training (scst) framework @xcite . scst with a cider reward @xcite is a standard approach for training the captioning models @xcite , but this paper will show that it does not sufficiently encourage the model to use copy operations. we design a new reward function that provides a reward for each copy operation proportional to the caption quality. we further restrict the scst training to the images that contain at least one word in the ground truth captions that corresponds to one of the detected object labels. with these innovations, the ecol-r model outperforms a scst baseline and a strong inference encouragement baseline by a large margin. our copy mechanism and caption generator incorporate two enhancements to better choose and incorporate novel objects: a) abstract labels which correspond to hypernyms of the object labels and facilitate knowledge transfer between objects appearing in training captions and novel objects; b) a morphological selector which determines the correct inflected form of the copied task specific object labels which is similar in purpose to that proposed in @xcite . we evaluate the ecol-r model on the novel object captioning benchmark nocaps @xcite and held-out @xcite . the ecol-r model achieves a new state of the art on both benchmarks and generalizes well to in-domain images. ## related work popular image captioning models include lstmbased @xcite and transformerbased decoders @xcite . the visual encoders are often neural object detectors @xcite producing region-of-interest (roi) vectors. to train the model to copy novel object labels, the neural baby talk model (nbt) @xcite and follow-up work @xcite @xcite use copy mechanisms @xcite . the copying candidates are labels of salient objects produced by external object detectors. in this paper, we follow previous work by using the visual genome object detector from @xcite as the visual feature extractor and a task specific object detector to provide object labels for copying. these models are typically trained with the cross-entropy loss (ce). this creates a mismatch between the training and testing environments because the evaluation metrics are non-differentiable text-based measures @xcite . self-critical sequence training (scst) @xcite was proposed to address this issue by directly optimizing the inference output using caption-level rewards, such as cider-d @xcite . there are two existing novel object captioning benchmarks: a) the held-out coco benchmark @xcite , constructed by excluding images containing one of eight selected object classes from the standard coco 2014 benchmark, and b) nocaps @xcite , which uses the coco 2017 benchmark for training and provides new validation and test images from the open images dataset with over 400 novel objects. both benchmarks are object-centric and there is no reliable benchmarks that systematically evaluate the quality of generated actions or attributes. ## model figure 2 provides an overview of the ecol-r model. we refer to the ecol-r model without scst training as ecol. we describe this model in sec. 3.1 and our novel reinforced copy encouragement training in sec. 3.2. ## experiments we conduct experiments on the nocaps @xcite and the held-out coco (hendricks et al., 2016) benchmark. we set the layer and embedding size to @xmath0, 2014). we train our models 15 epochs with batch size 100 for ce loss and 15 epochs with batch size 10 for scst loss. ## conclusion and future work this paper proposes the ecol-r model that includes a training scheme to encourage copying novel object labels using reinforced learning. our experiments show that the ecol-r model successfully integrates novel object information and achieves state-of-the-art performance on two novel object caption benchmarks. in the future, we plan to extend our scst reward function to other metrics such as spice @xcite and @xcite .