Paper:
Learning to Specialize with Knowledge Distillation for Visual Question Answering
Visual Question Answering (VQA) is a notoriously challenging problem because it involves various heterogeneous tasks defined by questions within a unified framework. Learning specialized models for individual types of tasks is intuitively attracting but surprisingly difficult; it is not straightforward to outperform naïve independent ensemble approach. We present a principled algorithm to learn specialized models with knowledge distillation under a multiple choice learning (MCL) framework, where training examples are assigned dynamically to a subset of models for updating network parameters. The assigned and non-assigned models are learned to predict ground-truth answers and imitate their own base models before specialization, respectively. Our approach alleviates the limitation of data deficiency in existing MCL frameworks, and allows each model to learn its own specialized expertise without forgetting general knowledge. The proposed framework is model-agnostic and applicable to any tasks other than VQA, e.g., image classification with a large number of labels but few per-class examples, which is known to be difficult under existing MCL schemes. Our experimental results indeed demonstrate that our method outperforms other baselines for VQA and image classification.

Review:
# Response to authors’ response # I appreciate the authors' effort to include (1) VQA2 (2) an MLP baseline (3) the result at k = 4. Overall, I like the paper’s idea and am satisfied with the response, although the improvement from the independent random ensemble (IE) to the proposed method (MCL-KD) becomes less significant at (1) and (2). The performance of a single model on VQA2 seems to be lower than that reported in [14]. The authors should check on this.  Reviewer 2 and I have several similar concerns---the paper needs to work on real datasets and more algorithms to show its robustness. I strongly suggest the authors to include one more dataset and at least one more algorithm. Also, I would like to see analysis on models learned for VQA2---is each model specialized to a different question type? The paper could bring fruitful thinking and impact to the Visual QA community. I will maintain my score for paper acceptance.  My specific response is in [] after each comment. =========================================================================   # Summary # This paper aims to learn an ensemble of specialized models for Visual QA---each model ideally will be dedicated to different question (or image) styles. The authors propose an objective with distillation on top of the “multiple choice learning MCL framework” to prevent (1) data deficiency on learning each specialized model and (2) forgetting general knowledge. The contribution is complementary and compatible to different Visual QA models (most of the Visual QA papers focus on developing new models), and thus could potentially benefit the whole community.   # Strengths # S1. As mentioned above, the proposed objective is model-agnostic---any existing Visual QA model can potentially be applied to be the specialized model. This is particularly important in the literature of Visual QA---there have been too many papers (at least 30) on proposing different models but the performance gap is very limited and it is hard to tell what are the essential insights / concepts / components for Visual QA. The authors should emphasize this.  S2. The paper is clearly written. The idea is clean and inspiring. The performance improvement compared to the existing MCL and CMCL objectives is significant and consistent.  # Main weaknesses (comments) # W1. The paper only evaluates on a synthetic dataset (i.e., CLEVR). It will be important to evaluate on real datasets like VQA2 or Visual Genome to demonstrate the applicability. It will be interesting to see if each specialized model will be dedicated to different questions types (e.g., questions begin with why, what, how). [The authors do provide results on VQA2, but the improvement is largely reduced. Moreover, the single model performance on VQA2 is lower than reported in [14]. What is the “k” here? In the future, the authors can use the datasets by K. Kafle and C. Kanan, An Analysis of Visual Question Answering Algorithms, ICCV 2017, which has less bias issues.]  W2. To support the strength, it will be important to evaluate on different Visual QA models beyond stacked attention network. A simple MLP baseline is a good starting point. See the following paper as an example to show its applicability. [The authors add the experiments. Although on MLP the gain becomes smaller, it still shows improvement. I would like to see more on this in the final version. Again, what is the “k” here?]  Hexiang Hu, Wei-Lun Chao, and Fei Sha, Learning answer embeddings for visual question answering, CVPR 2018  It will be great if the authors can also discuss other papers that can potentially benefit the whole community such as   Kushal Kafle and Christopher Kanan, Answer-Type Prediction for Visual Question Answering, CVPR 2016  W3. Besides the MCL framework, the authors should discuss other ways to learn an ensemble of models (e.g., mixture of experts) and the possibility (or difficulties) to apply them. [The authors promise to add the discussions.]  # Minor weaknesses (comments) # W4. Eq. (6) seems not mention the case where k > 1. Should it be argmin for top k? Eq. (4) and (5) might be simplified by removing Z and \nu---\nu(Z(x)) is actually P(y|x).  W5. From Table 1, it seems that the larger the k is, the higher the performance is. It would be important to evaluate on k = 4 to see if the performance will still increase or decrease to demonstrate the need of specialized models. [The authors show that the performance degrades when k= 4. Thanks.]  W6. What is M for the image classification experiment?  # Further comments # how to bridge the gap between the oracle and top 1 performance seems to be a promising direction for future work. I’m thinking if changing the decision rule in Eq. (8) by learning an attention p(m|x) on how confidence each model can help. Specifically, if p(m|x) can give higher weight to the model that gives the correct prediction, we can achieve the oracle performance.