Paper:
Deep networks face challenges of ensuring their robustness against inputs that cannot be effectively represented by information learned from training data. We attribute this vulnerability to the limitations inherent to activation-based representation. To complement the learned information from activation-based representation, we propose utilizing a gradient-based representation that explicitly focuses on missing information. In addition, we propose a directional constraint on the gradients as an objective during training to improve the characterization of missing information. To validate the effectiveness of the proposed approach, we compare the anomaly detection performance of gradient-based and activationbased representations. We show that the gradient-based representation outperforms the activation-based representation by 0.093 in CIFAR-10 and 0.361 in CURE-TSR datasets in terms of AUROC averaged over all classes. Also, we propose an anomaly detection algorithm that uses the gradient-based representation, denoted as GradCon, and validate its performance on three benchmarking datasets. The proposed method outperforms the majority of the state-of-the-art algorithms in CIFAR-10, MNIST, and fMNIST datasets with an average AUROC of 0.664, 0.973, and 0.934, respectively.

Review:
[Summary]
The paper proposed to use gradient-based presentation in deep neural networks to capture missing information unavailable in the activation-based network representation. It is claimed that the missing information that could not be encoded during learning from training set can be revealed by taking the gradient of an example with respect to the model parameters. Based on this idea, a new learning algorithm is proposed to combine both the conventional loss for activation-based presentation and gradient-based regularization. As an illustration, the method is evaluated on anomaly detection benchmarks with gradient-based representation as part of the anomaly inference. 
[Comments]
I’m not sure if I fully understand the claimed contribution and how it is justified either theoretically and empirically. It seems to me that the paper claims the the current activation-based network does not fully encode information from training set during learning, and taking the gradient over a novel example (either training or test) with respect to (w.r.t,) model parameters can encode the missing information. I don't quite follow a few things, and had a difficult time in interpreting the novelty here.
- What is the missing information (a model should learn but fails to capture from training set) exactly? The example in the introduction uses digit 0 for training and digit 6 for testing and claims the vertical edge in 6 is missing. But isn’t that expected if the model is only trained with digit 0s only? How would one expect the model to learn the vertical edge if there is none in the training data?       
- More critically, the idea of taking the gradient of a model w.r.t.its parameters over an example to encode geometric relationship of the example in the data manifold is by no means novel at the conceptual level. There have been at least a bunch of similar classical methods derived from the perspective of information geometry. For instance, the Fisher kernel and vector  have been well studied for more than a decade, with both theoretical treatment (e.g., “Exploiting Generative Models in Discriminative Classifiers” in NIPS 1998) and application across multiple areas (e.g., “Fisher Kernels on Visual Vocabularies for Image Categorization” in CVPR 2007). I don’t see any novelty here by doing this to a new model (deep neural networks), and it is frustrating to see classical methods are being “invented” over and over again by decorating them with fashionable wrappers without any reference to the origin in the literature.  
- Besides, to use gradients to capture missing information, the paper proposes to append a regularization term by enforcing the gradient at particular learning iteration to be close to the mean of those of previous iterations (via cosine similarity) as in (3). This seems not quite different from many existing popular inertia-based strategies like gradients with momentum, Adam, etc. It needs to be elucidated more how the proposed optimizer compares to these benchmark methods.        
In terms of implementation, I also found some details missing. For instance, the objective function of (4) requires evaluation of second order derivatives (since the second term in the cosSIM involves dL/dPhi(x, Phi)). How to efficiently compute this term is not clear to me yet.   It is also mentioned that only decoder of the AE is used for constraint gradient, The reason for this is also not explained.
For evaluation, only anomaly detection examples are provided. It would be great if more general tasks like image classification can be studied. After all, the proposed method is claimed to be a fairly general framework (the introduction actually uses image classification as the example), and evaluation on the most popular benchmarks provides the most convincing justification. Even on the anomaly detection tasks, the proposed approach does not seem to have solid advantages over baselines. I’m not sure how significant are the differences of 0.007 between GradCon and OCGAN (table 2 and 3), and those less than 1% in table 4.
Overall the paper does not seem to have a clear and well-defined motivation, contribution is also vague and trivial compared to literature, plus very convincing empirical justification. Thus I do not think it is ready for publication at ICLR.