Paper:
Dialog-based Interactive Image Retrieval
Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demonstrated in a footwear retrieval application. Experiments on both simulated and real-world data show that 1) our proposed learning framework achieves better accuracy than other supervised and reinforcement learning baselines and 2) user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results, and a more natural and expressive communication interface.
: In the context of interactive image retrieval, the agent incorporates the user's feedback to iteratively refine retrieval results. Unlike existing work which are based on relevance feedback or relative attribute feedback, our approach allows the user to provide feedback in natural language.
on the specific form of user interaction largely constrain the information that a user can convey to benefit the retrieval process.
In this work, we propose a new approach to interactive visual content retrieval by introducing a novel form of user feedback based on natural language. This enables users to directly express, in natural language, the most prominent conceptual differences between the preferred search object and the already retrieved content, which permits a more natural human-computer interaction. We formulate the task as a reinforcement learning (RL) problem, where the system directly optimizes the rank of the target object, which is a non-differentiable objective.
We apply this RL based interactive retrieval framework to the task of image retrieval, which we call dialog-based interactive image retrieval to emphasize its capability in aggregating history information compared to existing single turn approaches [13,14,15,16]. In particular, a novel end-to-end dialog manager architecture is proposed, which takes natural language responses as user input, and delivers retrieved images as output. To mitigate the cumbersome and costly process of collecting and annotating human-machine dialogs as the system learns, we utilize a model-based RL approach by training a user simulator based on a corpus of human-written relative descriptions. Specifically, to emulate a single dialog turn, where the user provides feedback regarding a candidate image relative to what the user has in mind, the user simulator generates a relative caption describing the differences between the candidate image and the user's desired image. 1 Whereas there is a lot of prior work in image captioning [17,18,19], we explore the problem of relative image captioning, a general approach to more expressive and natural communication of relative preferences to machines, and to use it as part of a user simulator to train a dialog system. The efficacy of our approach is evaluated in a real-world application scenario of interactive footwear retrieval. Experimental results with both real and simulated users show that the proposed reinforcement learning framework achieves better retrieval performance than existing techniques. Particularly, we observe that feedback based on natural language is more effective than feedback based on predefined relative attributes by a large margin. Furthermore, the proposed RL training framework of directly optimizing the rank of the target image shows promising results and outperforms the supervised learning approach which is based on the triplet loss objective. The main contributions of this work are as follows:
• A new vision/NLP task and machine learning problem setting for interactive visual content search, where the dialog agent learns to interact with a human user over the course of several dialog turns, and the user gives feedback in natural language.
• A novel end-to-end deep dialog manager architecture, which addresses the above problem setting in the context of image retrieval. The network is trained based on an efficient policy optimization strategy, employing triplet loss and model-based policy improvement [20].
• The introduction of a computer vision task, relative image captioning, where the generated captions describe the salient visual differences between two images, which is distinct from and complementary to context-aware discriminative image captioning, where the absolute attributes of one image that discriminate it from another are described [21].
• The contribution of a new dataset, which supports further research on the task of relative image captioning. 2

Review:
This paper proposes a novel approach to interactive visual content retrieval.  It formulates the task as a reinforcement learning problem, and rewards the dialog system for improving the rank of the target image during each dialog turn.  To alleviate the cost of collecting human-machine conversations as the dialog system learns, the dialog system is trained with a user simulator, which is itself trained to describe the differences between target and retrieved images.  Experiments show the advantage of the proposed RL-based dialog approach over alternative baselines.   [strengths]  1. To my knowledge, this is the first dialog-based interactive image retrieval system where the system and user communicate using natural language.  The proposed framework and its components are appropriate for the problem.  2. The idea of using a user simulator for relative image captioning is interesting.  This helps avoid the costly alternative which would be to collect human-machine dialogs as the system learns.  3. A new dataset for the task of relative image captioning is introduced.  However, the usefulness of the dataset may be limited (see below, weaknesses  point 2).  [weaknesses]  1. The paper's conclusion that "user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results" is too strong given the weakness of the pre-specified attributes baseline (Sec. 5.2). To me, this is the biggest weakness of the paper. Specifically, if my understanding is correct, that baseline uses a 10-D attribute vector, which is computed using hand-crafted features like GIST and color histograms [13].  Thus, it is not clear whether the improvement is due to natural language feedback (as the paper claims) or if it's due to better features.  This aspect needs to be studied in more detail.   2. For the relative image captioning dataset to be truly useful, a more systematic and automatic way of evaluation should be presented that goes beyond user studies (as presented in Appendix C). This is important so that new methods that try to tackle this problem can be properly compared.    3.  It'd be helpful if the paper could provide some typical failure cases so that the reader can have a better understanding of the system.   [summary]  Overall, due to the novelty of the problem and sound approach, my initial rating is to accept the paper. However, there are also some weaknesses (as mentioned above) that I would like to see addressed in the rebuttal.  [post-rebuttal final comments] Overall, the rebuttal has addressed to some degree the weaknesses stated in my initial review. I am, however, not completely satisfied with the authors' response regarding my point that the paper's claim "user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results" is too strong. Any comparison with hand-crafted feature based approaches needs to be stated clearly as so, as it is difficult to draw other conclusions out of such a comparison; i.e., if the baseline were to use a deep representation, it may perform just as well as the proposed approach. This needs to be empirically verified. Although this weakness is not enough for me to lower my initial rating, I feel that the paper either needs to (1) perform the appropriate experiment (i.e. the baseline should use a 10-D attribute vector derived from a deep neural network) or (2) clearly state that the baselines' features are hand-crafted and thus much weaker and may be the key cause for its lower performance. The rebuttal states that the revised paper will have something along the lines of (2), but I think it's important to explicitly state that the baseline is using "hand-crafted features".