Paper:
A Simple Cache Model for Image Recognition
Training large-scale image recognition models is computationally expensive. This raises the question of whether there might be simple ways to improve the test performance of an already trained model without having to re-train or fine-tune it with new data. Here, we show that, surprisingly, this is indeed possible. The key observation we make is that the layers of a deep network close to the output layer contain independent, easily extractable class-relevant information that is not contained in the output layer itself. We propose to extract this extra class-relevant information using a simple key-value cache memory to improve the classification performance of the model at test time. Our cache memory is directly inspired by a similar cache model previously proposed for language modeling . This cache component does not require any training or fine-tuning; it can be applied to any pre-trained model and, by properly setting only two hyper-parameters, leads to significant improvements in its classification performance. Improvements are observed across several architectures and datasets. In the cache component, using features extracted from layers close to the output (but not from the output layer itself) as keys leads to the largest improvements. Concatenating features from multiple layers to form keys can further improve performance over using single-layer features as keys. The cache component also has a regularizing effect, a simple consequence of which is that it substantially increases the robustness of models against adversarial attacks.

Review:
This paper presents a cache model to be used in image recognition tasks. The authors argue that class specific information can be retrieved from earlier layers of the network to improve the accuracy of an already trained model, without having to re-train of finetune. This is achieved by extracting and caching the activations of some layers along with the class at training time. At test time a similarity measure is used to calculate how far/close the input is compared to information stored in memory. Experiments show that performance is improved in CIFAR 10/100 and ImageNet. In addition the authors show that a cache model increases resistance in adversarial attacks and show that experimentally that the cache model is more stable at larger regions around the training data.  The paper is well written and contains a thorough experimental design. I would like to see a more conventional organisation of the paper. It feels off to not read previous research and jump straight at the results.  In general I liked reading the paper. The authors hypothesize that using a cache model similar to work of Grave et al. can benefit image recognition tasks as well. The motivation is clearly explained and the contributions are easy to extract.  I liked the idea that using a cache model is like putting a prior on data so that the model prefers inputs similar to the training set. How would that work in a real world case where data is not that perfect though as in benchmark datasets? I feel that favouring similar data to training set might not be usable in datasets in the wild where some of the assumptions on data break.  I do have some concerns over computation. Very deep networks will require searching in a combinatorial fashion for number of layers, which layers, what should lambda and theta be set to etc. I think a discussion on this should be present and ways to go around it. Perhaps this can be something that is learnt concurrently to the training procedure.  Table 2 shows p_adv. This was tuned to the baseline model and tested at cache3 and cache-only using that setting. I would like to see what happens when p_adv is tuned on the other models too. The main issue I have is that by adding a cache model, essentially you have changed the model and how decisions are made. So resnet32-cache3 is a different model than vanilla resnet32. Looking at how adversarial attacks transfer with resnet20 for instance, though there is an impact it's not nearly as catastrophic as in resnet32. So I am left feeling that maybe the performance of cache models is because it is more like transfer and can't judge with certainty if the adversarial behaviour is indeed there. Perhaps tuning p_adv for cache3 and cache-only will provide more information on how large a perturbation needs to be to break the model.  Regarding references, there's a lot of arxiv work when most have been published to peer reviewed journals and conference. There should be a second pass on that to fix it.  * The rebuttal has addressed most of my concerns and cleared up a few other places.