Paper:
Partially-Supervised Image Captioning
Image captioning models are becoming increasingly successful at describing the content of images in restricted domains. However, if these models are to function in the wild -for example, as assistants for people with impaired vision -a much larger number and variety of visual concepts must be understood. To address this problem, we teach image captioning models new visual concepts from labeled images and object detection datasets. Since image labels and object classes can be interpreted as partial captions, we formulate this problem as learning from partiallyspecified sequence data. We then propose a novel algorithm for training sequence models, such as recurrent neural networks, on partially-specified sequences which we represent using finite state automata. In the context of image captioning, our method lifts the restriction that previously required image captioning models to be trained on paired image-sentence corpora only, or otherwise required specialized model architectures to take advantage of alternative data modalities. Applying our approach to an existing neural captioning model, we achieve state of the art results on the novel object captioning task using the COCO dataset. We further show that we can train a captioning model to describe new visual concepts from the Open Images dataset while maintaining competitive COCO evaluation scores.

Review:
The paper works on the problem of captioning novel object in images without paired image-caption training data, assuming object labels but no captions are available for the objects.  The paper proposes to use constrained beam search [19] as a subroutine at training time to generate a caption with includes the object labels for an image, relying on a finite state machine to constrain the beam search with multiple labels. This allows to generate new training data, and an EM algorithm is used to generate training data and learn a better captioning model  The approach is more general in the sense that it could not just handle object labels but other forms of incomplete captions, e.g. phrases, however, this is not evaluated in the paper.    Strength: - Novel and interesting direction to approach novel image object captioning - Evaluation which includes ablations and comparison to state-of-the art, including very recent work from CVPR 2018 [23]. - Outperforms all prior work (one small exception is F1 Measure, where [23]+CBS is higher)  Weaknesses: 1. While authors discuss [19], it is not obvious in section 3.1. (and line 57) that [19] uses same Fine state automaton (or is there any difference?). 2. It would have been interesting to compare to the same out of coco test as in [20,21] with F1 score for better comparability. 3. It would be interesting to see (qualitatively) what kind of captions the CBS + fine state machine gives. I guess this is quantitatively shown in row 6, Table 1? Why is the F1 score in this case not significantly higher although gt labels are used? 4. Figure 1(b): It would be great if the authors would clarify the use of the example in Figure 1(b), in general and specifically if this is also used for captioning. As well as which sequences this automaton would allow, anything which starts with “the” but does not contain “score”? a. More state-machine examples would have been interesting in supplemental, also jointly with image + gt labels examples + generated caption. 5. Minor: line 40: “This reflects the observation that, in general, object detection datasets are easier to scale”. I don’t agree that there is much evidence for this. I think it is rather an artifact on what the community focuses on. The authors might want to consider reducing the strength of this statement. However, I do strongly agree that image captioning should be able to exploit detection data as done in this work.  Conclusion: The paper presents an interesting approach for novel object image captioning with strong experimental validation which I advocate to accept.   ----  After going through the author response and reviews: + Overall the authors addressed mine and other reviewers questions and promised to add further comparisons in the final. - unfortunately, the authors did not provide the additional results already in the author response. Both for mine and the concern of not comparable networks for LRCN+CBS, brought up by R2, which I agree to.  In conclusion I recommend acceptance, in expectation that the authors will add the promised results to the final version. I also suggest to include comparable results for LRCN+CBS or a fair ablation of the author's model with the same image network.