Original: Reinforcement Based Learning on Classiﬁcation Task Could Yield Better Generalization and Adversarial Accuracy
Revised: Reinforcement Based Learning on Classiﬁcation Task Could Yield Better Generalization and Adversarial Accuracy

Original: Author(s)

Original: Abstract
Revised: Abstract

Original: Deep Learning has become interestingly popular inthe ﬁeld of computer vision,mostly attaining near or above human-level performance in various vision tasks.
Revised: Deep Learning has become interestingly popular in computer vision, mostly attaining near or above human-level performance in various vision tasks.

Original: But recent work has also demonstrated that these deep neural networks are veryvulnerable to adversarial examples (adversarial examples - inputs to a model whichare naturally similar to original data but fools the model in classifying it into awrong class).
Revised: But recent work has also demonstrated that these deep neural networks are very vulnerable to adversarial examples (adversarial examples - inputs to a model which are naturally similar to original data but fools the model in classifying it into a wrong class).

Original: Humans are very robust against such perturbations, one possiblereason could be that humans do not learn to classify  on the basis error between"target label" and "predicted label" but rather they learn on the basis of reinforcements which is based on what predictions they make.
Revised: Humans are very robust against such perturbations; one possible reason could be that humans do not learn to classify based on an error between "target label" and "predicted label" but possibly due to reinforcements that they receive on their predictions.

Original: In this work, we proposed anovel method to train deep learning models on an image classiﬁcation task.
Revised: In this work, we proposed a novel method to train deep learning models on an image classiﬁcation task.

Original: Weused a reward-based optimization function, similar to the vanilla policy gradientmethod in reinforcement learning to train our model instead of conventional crossentropy loss.
Revised: We used a reward-based optimization function, similar to the vanilla policy gradient method used in reinforcement learning, to train our model instead of conventional cross-entropy loss.

Original: An empirical evaluation on  cifar10 dataset showed that our methodoutperforms the same model architecture trained using cross-entropy loss function(on adversarial training).
Revised: An empirical evaluation on the cifar10 dataset showed that our method learns a more robust classiﬁer than the same model architecture trained using cross-entropy loss function (on adversarial training).

Original: At the same time, our method generalizes better to thetraining data with the difference in test accuracy and train accuracy < 2% for mostof the timeas compared to  cross-entropy one, whose difference most of the timeremains > 2% .
Revised: At the same time, our method shows a better generalization with the difference in test accuracy and train accuracy < 2% for most of the time compared to the cross-entropy one, whose difference most of the time remains > 2% .

Original:  Introduction
Revised:  Introduction

Original: There’s a tremendous increase in using deep learning models for various perceptual tasks in computervision.
Revised: There’s a tremendous increase in using deep learning models for various perceptual tasks in computer vision.

Original: Thousands of new works are being published every year on attaining better accuracy ondifferent datasets [8].
Revised: Thousands of new works are being published every year on attaining better accuracy on different datasets [14].

Original: But these deep neural networks are very vulnerable to adversarial examples(adversarial examples - inputs to a model which are naturally similar to original data but fools themodel in classifying it into a wrong class)  which raises great concern about whether the modelshould be used for real-time application purpose or not [6].
Revised: But these deep neural networks are very vulnerable to adversarial examples (adversarial examples - inputs to a model which are naturally similar to original data but fools the model in classifying it into a wrong class) [10], which raises a concern about whether the model should be used for real-time application purpose or not.

Original: While the past few years have seenintense research in training robust models against adversarial attacks most of them have focused onusing various adversarial training approaches, unlabelled data, or revisiting misclassiﬁed examples[9, 2, 12].
Revised: While the past few years have seen intense research in training robust models against adversarial attacks, most of them have focused on using various adversarial training approaches, unlabelled data, or revisiting misclassiﬁed examples [15, 5, 21].

Original: On the other hand humans are very robust against such perturbations.
Revised: On the other hand, humans are very robust against such perturbations.

Original: Previously, Engstromet al.
Revised: Previously, Engstrom et al.

Original: [5] and Ilyas et al.
Revised: [8] and Ilyas et al.

Original: [7] shownthat how adversarial perturbations corresponds to non-robust butpredictive features and that adversarial robustness can indeed help deep neural networks in learningperceptually similar features. 
Revised: [12] shown how adversarial perturbations correspond to non-robust but predictive features and that adversarial robustness can indeed help deep neural networks in learning perceptually similar features. This gives us a hint to look at the problem of adversarial robustness by introducing a more human-like learning approach.

Original: So we tried to ﬁnd a different approach to train deep neural networkson the basis how possibly humans learn to classify images.
Revised: So we tried to ﬁnd a different approach to train deep neural networks based on how possibly humans learn to classify images.

Original: Humans do not learn to classify  on thebasis error between "target label" and "predicted label" but instead they predicts a label and thensome other people give them feedback  which results into a positive or negative reward for the learnerand onthe basis of this reinforcements they learn to classify different stimuli.
Revised: Humans do not learn to classify based on an absolute error between "target label" and "predicted label". Instead, they predict a label and then get feedback from another person or a source, which results in a positive or negative reward for the learner, and based on this reward; they learn to classify. Past research has also shown that if only one [Figure]

Revised: Randomly

Original: In this work, we focused on introducing a new kind of objective function.
Revised: [Figure] based on [Figure] probability

Revised: Predicted class

Revised: [Figure]

Revised: a_t

Revised: [Figure]

Revised: Calculate reward

Revised: [Figure]

Revised: [Figure]

Revised: base

Revised: [Figure]

Revised: y_

Revised: [Figure]

Revised: Loss:

Revised: [Figure]

Revised: 1*log(P(a_t|s_t))*R_t

Revised: [Figure]

Revised: type of reward is given, i.e., either a positive or negative reward, learning follows a simple rule-based approach.

Revised: But if both negative and positive rewards are given, learning follows an integration-based approach [3, 2, 18].

Original:  We used a reward-basedoptimization function, similar to the vanilla policy gradient method in reinforcement learning to trainour model instead of conventional cross-entropy loss.
Revised: In this work, we focused on introducing a new kind of objective function. We used a reward-based optimization function, similar to the vanilla policy gradient method in reinforcement learning, to train our model instead of conventional cross-entropy loss.

Original: The formulationof this method is fairly simpleand similar to the vanilla policy gradient [14].
Revised: Our formulation is fairly simple and similar to the vanilla policy gradient [23].

Original: We design the reward environment, which is as simpleas giving positive rewards for correct classiﬁcation and negative rewards for the wrong classiﬁcation.
Revised: We design the reward environment, which is as simple as giving positive rewards for correct classiﬁcation and negative rewards for the wrong classiﬁcation.

Original: And in the end, we train the model to maximize reward using a policy gradient.
Revised: And in the end, we train the model to maximize reward using a policy gradient.

Original: Here our policyis simply the softmax probability distribution to classify the given input image among the variousclasses.
Revised: Here our policy is simply the softmax probability distribution to classify the given input image among the various classes.

Original: We trained a very minimal CNN architecture against FGSM attacks [6] using our methodand with cross-entropy loss.
Revised: We trained a very minimal CNN architecture against FGSM attacks [10] using our method and crossentropy loss.

Original: We observed that even though the model trained using cross-entropyachieved higher accuracy on the test set as compared to our method, the accuracy against adversarialexamples was higher for our method.
Revised: We observed that even though the model trained using cross-entropy achieved higher accuracy on the test set than our approach, the accuracy against adversarial examples (generated using test data) was higher for our method.

Original: One more very interesting resultthat came out was that ourmethod generalizes much better than the model trained using cross-entropy loss i.e. our training andvalidation accuracy remains almost close to each other.
Revised: One more exciting result was that our method generalizes much better than the model trained using cross-entropy loss, i.e., our training and validation accuracy remains almost close to each other.

Original:  Method
Revised:  Method

Original: Some of the previous work has focused on Reinforcement Based learning in classiﬁcation tasks butnone of them evaluated its adversarial robustness, as well as most of them, are  complex formulation[13].
Revised: A combination of deep learning and reinforcement learning (RL) is widely being used in state-based decision-making tasks [16, 9]. But there are very few works that have focused on reinforcementbased learning in classiﬁcation tasks [22]. As per our knowledge, no evaluation is done on its adversarial robustness, and generalization. Also, previous implementations of RL for classiﬁcation are more complex than ours.

Original: In this work, we propose a fairly simple implementation.
Revised: We propose a fairly simple implementation.

Original: We will also release a simple
Revised: We designed a simple

Original: RL environment for the classiﬁcation task which will act similar to other RL environments.
Revised: RL environment for the classiﬁcation task, which will act similar to other RL environments used to evaluate reinforcement learning algorithms.

Original: Thisenvironment can be used to run various RL algorithms to train on the classiﬁcation tasks.
Revised: This environment can be used to run various RL algorithms to train on any classiﬁcation task.

Original: In thiswork, we used the basic formulation of the Vanilla Policy Gradient (VPG) method to test our model.
Revised: In this work, we used the basic formulation of the Vanilla Policy Gradient (VPG) method to test our method [23].

Original: The state s t is the input image, action a t is the predicted class, and reward R t depends upon a t andactual class label y t .
Revised: In our formulation, the state s t is the input image, action a t is the predicted class, and reward R t depends upon a t and actual class label y t .

Original: If a t = y t , we give a positive reward (+1) on the other hand if a t (cid:54) =
Revised: If a t = y t , we give a positive reward (+1) on the other hand if a t (cid:54) =

Original: y t , we givea negative reward (-1).
Revised: y t , we give a negative reward (-1).

Original: Finally, we train the model using VPG loss (refer to Eq. 1).
Revised: Finally, we train the model using VPG loss (refer to Eq. 1).

Original: Here t is theexample image in the given training batch of size B .
Revised: Here t is the example image in the given training batch of size B .

Original: [Equation]
Revised: [Equation]

Original:  Note that during training we predict the class label i.e. take action by using random selection basedon the softmax probability. This is extensively used in RL to bring randomness during training.Thecomplete procedure is shown in Figure 1.
Revised: The complete ﬂowchart is shown in Figure 1. Note that during training, the model takes action by using random selection based on the softmax probability, which is extensively used in RL to bring randomness during training.

Original: One direct beneﬁt of this implementation is that we arepenalizing/rewarding our network based on what predictions it makes.
Revised: One direct beneﬁt of this implementation is that we are [Figure] penalizing/rewarding our network based on what predictions it makes.

Original: So if the network predicts awrong class, it will be penalized based on the gradients calculated using that speciﬁc wrong classwhich in comparison to cross-entropy depends only on the gradients calculated using the correctlabel.
Revised: So if the network predicts a wrong class, it will be penalized based on the gradients calculated using that speciﬁc wrong class, which in comparison to cross-entropy depends only on the gradients calculated using the correct label.

Original: Another advantage of using this method is that we can develop different strategies for assigningrewards based on the prediction that the networks makes.
Revised: Another advantage of using this method is that we can develop different strategies for assigning rewards based on the networks’ prediction.

Original: For example giving higher negative rewards to those examples on which networks makes repeated wrong prediction couldprobably help in fastenthe learning.
Revised: For example, giving higher negative rewards to those examples on which the network makes repeated wrong predictions could help improve the learning.

Original:  Experiment
Revised:  Experiment

Original: We trained a very minimal CNN architecture on cifar10. The  architecture used was shown in Figure 2. 
Revised: The CNN architecture used for the study is shown in Figure 2. The architecture follows a similar heuristic of convolution layers used in VGG16 architecture [19], but the network’s depth is relatively less.

Revised: One more important thing to note in the architecture we used is the "layer normalization" layer just before the softmax activation, which we experimentally found to avoid obfuscated gradient issue effectively [4].

Original: The model was trained on adversarial images generated using FGSM, these adversarial images weregenerated at the beginning of each training step on the trained model.
Revised: We ﬁrst performed the evaluation without any speciﬁc adversarial training approach. We found that the RL method indeed showed better FGSM adversarial accuracy than the CE method (refer to Appendices, Figure 5).

Original: We trained both the modeli.e.
Revised: But it was easy to attack the trained model via AutoAttack (an ensemble of diverse parameter-free attacks [7]). We then performed adversarial training by generating adversarial images using FGSM attack (at (cid:15) = 8 / 255 ). The adversarial images were generated at the beginning of each training step (refer to Equation 1 in Appendices for adversarial training algorithm).

Original:  cross-entropy (CE) one and our  method (RL) one for 220 epochs with RMSprop optimiser atlearning_rate=0.0001 and decay=1e-6 (This choice of hyperparameter was taken from tensorﬂowexample model on cifar10).
Revised: We trained the model on those adversarial images using both the methods, i.e., cross-entropy (CE) one and our proposed method (RL) one for 220 epochs with RMSprop optimizer at learning _ rate = 0 . 0001 and decay = 1 e − 6 (The choice of hyperparameter was taken from TensorFlow example model on cifar10).

Original: We initially made checkpoints of our model at intervals of 20 epochs andreporting training, testing, and adversarial accuracy (on test data) at those intervals.
Revised: We initially made checkpoints of our model at intervals of 20 epochs and reporting training, testing, and adversarial accuracy (on test data) at those intervals (4).

Original: We ran a secondinstance in which we recorded the training history till 150 epochs have included the correspondingplot in our supplementary pdf.

Original: Wethen picked the checkpoint at which the respective methodperformed best on FGSM attack.
Revised: We picked the checkpoint at which the respective method performed best on adversarial images generated using test data.

Original: Which was RL model at 220th epoch and CE model at 60thepoch.
Revised: For the RL model, it was the 220th epoch. And for the CE model, it was the 60th epoch.

Original: We tested the robustness of both of the model using AutoAttack (an ensemble of diverseparameter-free attacks [4]).
Revised: Finally, the robustness of both of the models was tested using AutoAttack. To test the CE method’s performance at near the 60th epoch, after which the CE model’s performance starts to decrease and for the sake of generalization, we retrained the model from scratch in which we recorded the training history till 150 epochs. We found similar results (Appendices,

Revised: Figure 6).

Revised: [Figure]

Revised: with time.

Revised: Adversarial accuracy corresponds to

Revised: FGSM attacks with (cid:15) = 8 / 255 on test cifar test data

Revised: [Figure]

Original:  Results
Revised:  Results

Original: The training performance for both the model is shown in Figure 3.
Revised: The training performance for both the model is shown in Figure 3.

Original: The model trained using crossentropy loss is abbreviated as ‘CE’ while the model trained using our reinforcement-based methodis abbreviated as ‘RL’.
Revised: The model trained using crossentropy loss is abbreviated as ‘CE’ while the model trained using our reinforcement-based method is abbreviated as ‘RL.’

Original: The CE model reaches it maximum adversarial accuracy at 60th epochwith adversarial accuracy on FGSM attack 32.86% and reduces afterwards whereas the RL modelperformance improved over successive epochs with adversarial accuracy on FGSM attack 37.66%.
Revised: The CE model reaches its maximum adversarial accuracy at 60th epoch with adversarial accuracy on FGSM attack 32.86% and reduces afterward. In contrast, the RL model performance improved over successive epochs with adversarial accuracy on FGSM attack 37.66% at the end of 220th epoch.

Original: From the Figure 3, we can also see that the RL model generalises much better than the CE model withits training and testing curve to be very close to each other.
Revised: We can also see that the RL model generalizes much better than the CE model, with its training and testing curve to be very close to each other (Figure 3).

Original: We also tested the robustness accuracyfor both the models i.e. CE and RL one against differentlevel of perturbations  added to the originalimage.
Revised: We also tested the robustness accuracy for both the models, i.e., CE and RL one, against different perturbations level added to the original image.

Original: Results are shown in 
Revised: Results are shown in Figure 4.

Original: [Table] [Figure]
Revised: RL model always performed better than the CE model, which was trained till 220 epochs. We also tested the CE model, which was trained till 60th epochs; we found that for eps ≤ 3 / 255 , the CE model performed better than the RL model, but its performance got worse on reaching eps = 8 / 255 .

Original: We further tested the robustness of bothof the model using AutoAttack (contains ensemble of variousadversarial attacks - consisting of apgd-ce, apgd-t, fab-t, and square attack [4, 1, 3]) to conﬁrm therobustness against several other adversarial attacks.
Revised: We further tested the robustness of both the model using AutoAttack (contains ensemble of various adversarial attacks - consisting of apgd-ce, apgd-t, fab-t, and square attack [7, 1, 6]) to conﬁrm the robustness against several other adversarial attacks.

Original: We found that our model still performs muchbetter than the CE model (Table 1).
Revised: We found that our model still performs much better than the CE model (Table 1).

Original: Please note that all the reported accuracy are on cifar10’s testdata.
Revised: Please note that all the reported accuracy are on cifar10’s test data. [Table]

Original: Discussion
Revised: There are some concerns regarding the issue of label leaking with the FGSM method for adversarial training [13]. Therefore we also evaluated the performance CE case with adversarial training using PGD-Linf attack (eps = 8/255, step_size=2/255, and num_step=5)[15]. We show that even if we train the network using cross-entropy loss on adversarial images generated using PGD-Linf attack, the RL method’s performance remains better than the CE method.

Revised: The best accuracy against AutoAttack for

Revised: CE method trained using PGD-Linf attack on test data was 27.76%.

Revised: Moreover, some recent work suggests that the FGSM based method can be used for adversarial training with proper early stopping [24].

Revised:  Discussion

Original: In this work, we proposed a fairly simple method to introduce reinforcement based learning forclassiﬁcation tasks.
Revised: In this work, we proposed a fairly simple method to introduce reinforcement-based learning for classiﬁcation tasks.

Original: Our method shows an initial sign of improvements for better generalisation aswell as more robustness to adversarial attacks when compared with same model architecture trainedon cross-entropy loss.
Revised: Our method shows an initial sign of improvements for better generalization and more robustness to adversarial attacks than the same model architecture trained on cross-entropy loss.

Original: Even though our present model does not beat the SOTA results. If we considersome recent work involving faster training for  robust model against adversarial attack our modelperformance is quite comparable and outperforms [11] which shows 29.35% accuracy on cifar10against AutoAttack [4].
Revised: Even though our present model does not beat the SOTA results if we consider some recent work involving faster training for a robust model against adversarial attack. Our model performance is quite comparable and outperforms [20] which shows 29.35% accuracy on cifar10 against AutoAttack [7].

Original: But considering that this is a priliminary result, we still need to evaluate ourmethod on a larger scale with  more complex modellike which are known to perform best on cifar10and better parameter tuning.
Revised: But considering that this is a preliminary result we still need to evaluate our method on a larger scale with a more complex model known to perform best on cifar10 and better parameter tuning.

Original: On the other hand, our RL formulation for classiﬁcation task could probably beneﬁt other computervision problems such as datasets having imbalanced class in which we cansimply change themagnitude of positive or negative reward for different classes to remove discrepancy due to imbalancedata.
Revised: On the other hand, our RL formulation for classiﬁcation task could probably beneﬁt other computer vision problems such as datasets having imbalanced class in which we can change the magnitude of positive or negative reward for different classes to remove discrepancy due to imbalanced data.

Original: For future research, it would also be interesting to evaluate how the model performs if wechange themethod for reward assignment.
Revised: For future research, it would also be interesting to evaluate how the model performs if we change the reward assignment method.

Original: It would also be interesting to test model trained usingthis method on predicting human similarity judgements as done by Peterson et al.
Revised: It would also be interesting to test model trained using RL method on predicting human similarity judgments (Peterson et al.

Original: And ﬁnally, we hope that this result will bring on interesting discussion on  intersection of cognitivescience and artiﬁcial intelligence interms of understanding how humans might be learning to classifydifferent objects.
Revised: Finally we hope that this result will bring on exciting discussions on the intersection of cognitive sciences and artiﬁcial intelligence in understanding how humans might be learning to classify different objects A. Adversarial Training Algorithm

Revised: The pseudo-code shown below does not show the FGSM attack algorithm [10] and RMSprop [11].

Revised: Both are very widely known, and their algorithms can be referred from the original work.

Original: A. Results Without Any Adversarial Training Against FGSM Attacks
Revised: for t = 1 , 2 , . . . T do x shuffle = shuffle _ dataset ( x ) ; for b = 1 , 2 , . . . N/B do x b , y b = x shuffle [( b − 1) ∗ B : b ∗ B ] , y [( b − 1) ∗ B : b ∗ B ] ; x adv = FGSM _ attack ( x b , y b , (cid:15) ) ; // perform FGSM attack //

Revised: Calculate ∇ θ using RMSprop optimiser ∇ θ = RMSprop _ optimiser ( loss ( f θ ( x adv ) , y b ) , θ ) ; θ = θ − ∇ θ ; // update θ end end

Original: We also performed the evaluation without any adversarial training against FGSM attack for which RLmodel indeed showed better FGSM adversarial accuracy as compared to CE model.
Revised: We also performed the evaluation without any adversarial training against FGSM attack for which the RL model indeed showed better FGSM adversarial accuracy than the CE model.

Original: But as statedbefore this was very easily attacked by AutoAttack.
Revised: But as stated before, AutoAttack very easily attacked this.

Original: B. Adversarial Accuracy vs Epochs on Another Training Instance (with adversarial training)
Revised: C. Adversarial Accuracy vs Epochs on Another Training Instance (with adversarial training)

Original: We initially made checkpoints of our model at intervals of 20 epochs.
Revised: We initially made checkpoints of our model at intervals of 20 epochs.

Original: To test the  performancefor themodel at near the 60th epoch after which the CE model performance starts to decrease and for thesake of generalisation, We ran a second training instance in which we recorded the training historytill 150 epochs.
Revised: To test the model’s performance at near the 60th epoch, after which the CE model’s performance starts to decrease and for the sake of generalization, we retrained the model from scratch in which we recorded the training history tillepochs.

Original: It again resulted into similar results.
Revised: It again resulted in similar results.

Original: Results are shown in Figure [-
-]
Revised: Results are shown in Figure [-
-]6.

Original: C. Gradient Examples  Calculated on the Both Model
Revised: D. Examples of Gradient Calculated on Models Trained Using Both Methods

Original: To look further into the kind of features learned by both type of  objective function.
Revised: To look further into the kind of features learned by both types of the objective function.

Original: We checked thegradients calculated during  FGSM attack on both the models.
Revised: We checked the gradients calculated during the FGSM attack on both the models.

Original: While for many of the examples wedid not ﬁnd any useful differences between the two method, some of the examples were worth addinghere in which we did ﬁnd  difference among the two model.
Revised: While for many of the examples, we did not ﬁnd any useful differences between the two methods, some of the examples were worth adding here in which we did ﬁnd the difference between the two methods. A closer look into various examples is needed to make any clear distinction.