Original: Reinforcement Explanation LearningAnonymous Author(s) AfﬁliationAddress email
Revised: Reinforcement Explanation Learning

Original: Abstract
Revised: Abstract

Original: Deep Learning has become overly complicated and has enjoyed stellar successin solving several classical problems like image classiﬁcation, object detection,etc.
Revised: Deep Learning has become overly complicated and has enjoyed stellar success in solving several classical problems like image classiﬁcation, object detection, etc.

Original: Several methods for explaining these decisions have been proposed.
Revised: Several methods for explaining these decisions have been proposed.

Original: Blackbox methods to generate saliency maps are particularly interesting due to thefact that they do not utilize the internals of the model to explain the decision.
Revised: Blackbox methods to generate saliency maps are particularly interesting due to the fact that they do not utilize the internals of the model to explain the decision.

Original: Major black box methods perturb the input and observe the changes in the output.
Revised: Most black-box methods perturb the input and observe the changes in the output.

Original: We formulate saliency map generation as a sequential search problem andleverage upon Reinforcement Learning (RL) to accumulate evidence from input images that most strongly support decisions made by a classiﬁer.
Revised: We formulate saliency map generation as a sequential search problem and leverage upon Reinforcement Learning (RL) to accumulate evidence from input images that most strongly support decisions made by a classiﬁer.

Original: Such astrategy encourages to search intelligently for the perturbations that will lead tohigh-quality explanations.
Revised: Such a strategy encourages to search intelligently for the perturbations that will lead to high-quality explanations.

Original: While successful black box explanation approachesneed to rely on heavy computations and suffer from small sample approximation, the deterministic policy learned by our method makes it a lot more efﬁcient during the inference.
Revised: While successful black box explanation approaches need to rely on heavy computations and suffer from small sample approximation, the deterministic policy learned by our method makes it a lot more efﬁcient during the inference.

Original: Experiments on three benchmark datasets demonstrate the superiority of the proposed approach in inference time over state-ofthe-arts without hurting the performance.
Revised: Experiments on three benchmark datasets demonstrate the superiority of the proposed approach in inference time over state-of-the-arts without hurting the performance.

Original: The anonymized code [-can be found athttps://anonymous.4open.science/r/RExL-88F
-]
Revised: Our code [-can be found athttps://anonymous.4open.science/r/RExL-88F
-]is made public at https://cvir.github.io/projects/rexl.html

Original:  Introduction
Revised:  Introduction

Original: Deep Learning methods are enjoying enormous success in computer vision, natural language androbotics research.
Revised: Deep Learning methods are enjoying enormous success in computer vision, natural language and robotics research.

Original: Over the years, the models have become overly complex and large with millionsof parameters.
Revised: Over the years, the models have become overly complex and large with millions of parameters.

Original: For example, GPT- 3 [4] has about 175 billion parameters.
Revised: For example, GPT- 3 [4] has about 175 billion parameters.

Original: Yet, it remains largelyunclear how the system comes to a decision, how certain the model is about its decision, if and whenit can be trusted or when it has to be corrected.
Revised: Yet, it remains largely unclear how the system comes to a decision, how certain the model is about its decision, if and when it can be trusted or when it has to be corrected.

Original: Explainable AI (XAI) is being extensively researchedupon [10, 11, 24, 31, 35, 42, 43, 44] to get a better insight into AI made decisions to not only improvethe models but also to instill accountability in them.
Revised: Explainable AI (XAI) is being extensively researched upon [10, 11, 24, 31, 35, 42, 43, 44] to get a better insight into AI made decisions to not only improve the models but also to instill accountability in them.

Original: The explanations could beneﬁt users in manyways such as improving safety and fairness when relying on AI decisions, especially in safety criticalapplications e.g. , healthcare [7, 23], autonomous driving [32] or criminal justice [16].
Revised: The explanations could beneﬁt users in many ways such as improving safety and fairness when relying on AI decisions, especially in safety critical applications e.g. , healthcare [7, 23], autonomous driving [32] or criminal justice [16].

Original: XAI Strategies for deep visual models offer model speciﬁc [1, 28] or saliency map based explanations [44, 31, 24, 27, 11].
Revised: XAI Strategies for deep visual models offer model speciﬁc [1, 28] or saliency map based explanations [11, 24, 27, 31, 44].

Original: This work subscribes to the later where saliency maps highlight relevantregions in an input e.g. , image, video or natural language texts.
Revised: This work subscribes to the later where saliency maps highlight relevant regions in an input e.g. , image, video or natural language texts.

Original: Saliency maps are obtained eitherfollowing a black box or a white/grey box technique.
Revised: Saliency maps are obtained either following a black box or a white/grey box technique.

Original: White/grey box approaches [44, 31, 43] eitherassume access to the internals of the base model (the model whose predictions are explained) ormodify the base model which can cost the base model its performance.
Revised: White/grey box approaches [31, 43, 44] either assume access to the internals of the base model (the model whose predictions are explained) or modify the base model which can cost the base model its performance [44].

Original: This makes them unsuitableto be generalised over all architectures.
Revised: This makes them unsuitable to be generalized over all architectures.

Original: Black box methods [24, 11, 10, 42] on the contrary, do not touch the internals of the base model.
Revised: Black box methods [10, 11, 24, 42] on the contrary, do not touch the internals of the base model.

Original: Instead, the input image pixels are perturbed and passed through the base model.
Revised: Instead, the input image pixels are perturbed and passed through the base model.

Original: The output predictionresulting in from the perturbed image differentiates between the important and non-important regions as perturbing important or relevant regions affect the output score more than the case when non- important image regions are perturbed.
Revised: The output prediction resulting in from the perturbed image differentiates between the important and non-important regions as perturbing important or relevant regions affects the output score more than the case when nonimportant image regions are perturbed.

Original: A popular technique to achieve this is to view saliency mapsas weighted masks.
Revised: A popular technique to achieve this is to view saliency maps as weighted masks.

Original: The perturbations to the input are performed by masking portions of the imageusing these weighted masks.
Revised: The perturbations to the input are performed by masking portions of the image using these weighted masks.

Original: Different methods differ in the way these masks are constructed.
Revised: Different methods differ in the way these masks are constructed.

Original: Fonget al.
Revised: Fong et al.

Original: [11, 10] solve an optimization objective to construct optimal masks while RISE [24] useslarge number of random masks for an image.
Revised: [10, 11] solve an optimization objective to construct optimal masks while RISE [24] uses large number of random masks for an image.

Original: A good explanation is associated with a low deletionscore [24] which measures the drop in probability of a class as pixels are gradually removed accordingto the saliency map given importance.
Revised: A good explanation is associated with a low deletion score [24] which measures the drop in probability of a class as pixels are gradually removed according to the saliency map given importance.

Original: In this paper, we address explainability as a sequential search problem where a sequence of imagepatches or regions are investigated in order to get the mask giving the saliency map.
Revised: In this paper we address explainability as a sequential search problem where a sequence of image patches or regions are investigated in order to get the mask giving the saliency map.

Original: Speciﬁcally, anagent starting with an unperturbed image, learns to choose the best image region to perturb giventhe choices made earlier.
Revised: Speciﬁcally, an agent starting with an unperturbed image, learns to choose the best image region to perturb given the choices made earlier.

Original: The agent chooses the sequence of regions from the currently unperturbedportions of the image so that perturbing the region maximally drops the class probability which inturn, causes a low deletion score. 
Revised: The agent chooses the sequence of regions from the currently unperturbed portions of the image so that perturbing the region maximally drops the class probability which in turn, causes a low deletion score. An approximation of the exhaustive search is performed by

Original: An approximate exhaustive search is performed by RISE [24] andis shown to achieve good deletion score albeit being prohibitively expensive both in time and memoryrequirements.
Revised: RISE [24] and is shown to achieve good deletion score albeit being prohibitively expensive both in time and memory requirements.

Original: Our approach is fundamentally different as RISE does not make any conscious effortto get low deletion score whereas we learn to intelligently search for the mask with a goal to achieveas low deletion score as quickly possible.
Revised: Our approach is fundamentally different as RISE does not make any conscious effort to get low deletion score whereas we learn to intelligently search for the mask with a goal to achieve as low deletion score as quickly possible.

Original: The goal oriented learning offered by the proposed methodnot only makes the inference fast but also makes it deterministic.
Revised: The goal oriented learning offered by the proposed method not only makes the inference fast but also makes it deterministic.

Original: The core of the problem is to ﬁnd a policy that assesses the image perturbed by the present mask anddecides to perturb a new relevant image region.
Revised: The core of the problem is to ﬁnd a policy that assesses the image perturbed by the present mask and decides to perturb a new relevant image region.

Original: Figure (1) illustrates the decision making processof such an agent.
Revised: Figure (1) illustrates the decision making process of such an agent.

Original: At ﬁrst, the unperturbed image is passed through the base classiﬁer to get theprobability of the class present in the image (an ‘aeroplane’ in Figure (1))
Revised: At ﬁrst, the unperturbed image is passed through the base classiﬁer to get the probability of the class present in the image (an ‘aeroplane’ in Figure (1))

Original: The agent then starts bychoosing a patch from the image and replacing it with uniform random noise.
Revised: The agent then starts by choosing a patch from the image and replacing it with uniform random noise.

Original: The perturbed image isassessed by the base model to get the change in probability of the same class.
Revised: The perturbed image is assessed by the base model to get the change in probability of the same class.

Original: Subsequently, the agentdecides to pick the most important regions so that perturbing it decreases the probability of the classfurther until the agent decides to stop exploring or the maximum number of patches is perturbed.
Revised: Subsequently, the agent decides to pick the most important region so that perturbing it decreases the probability of the class further until the agent decides to stop exploring or the maximum number of patches is perturbed.

Original: Asthe perturbed regions contribute to a drop of class probability, these regions are pivotal evidencesfor the classiﬁer to classify the image.
Revised: As the perturbed regions contribute to a drop of class probability, these regions are pivotal evidences for the classiﬁer to classify the image.

Original: The signiﬁcance of the perturbed regions behind the baseclassiﬁer’s decision is manifested by the drop in probability.
Revised: The signiﬁcance of the perturbed regions behind the base classiﬁer’s decision is manifested by the drop in probability.

Original: Thus at every step, the chosen region gets an weight proportional to the drop of the class probability.
Revised: Thus at every step, the chosen region gets an weight proportional to the drop of the class probability.

Original: This weighted regions provide adiscrete weighted mask that aftersuitable smoothening provides the saliency map explaining thedecision of the classiﬁer.
Revised: This weighted regions provide a discrete weighted mask that after smoothening provides the saliency map explaining the decision of the classiﬁer.

Original: As region perturbation is sequential and its effect is cumulative on the basemodel, we introduce a cumulating factor λ ∈ (0 , 1) that partially credits the regions perturbed earlierbut may have resulted in a signiﬁcant drop in class probability later.
Revised: As region perturbation is sequential and its effect is cumulative on the base model, we introduce a cumulating factor λ ∈ (0 , 1) that partially credits the regions perturbed earlier but instead of immediate drop, may have resulted in a signiﬁcant drop in class probability later.

Original: Learning such a dynamic agent capable to performing sequential actions towards a long-term goal byactively interacting with the environment in absence of any supervision is performed byReinforcement
Revised: Learning such a dynamic agent capable of performing sequential actions towards a long-term goal by actively interacting with the environment in absence of any supervision is performed by

Original:  Learning (RL).
Revised: Reinforcement Learning (RL).

Original: Motivated by its recent success [37, 41] coupled with its ease of use, we use ACER [40]to learn an RL agent giving the ﬁnal saliency map.
Revised: Motivated by its recent success [37, 41] coupled with its ease of use, we use ACER [40] to learn an RL agent giving the ﬁnal saliency map.

Original: Our approach is equally capable of learning theoptimal perturbing sequence on a single image, a class of images or a whole dataset of images.
Revised: Our approach is equally capable of learning the optimal perturbing sequence on a single image, a class of images or a whole dataset of images.

Original: Thismakes our approach ﬂexible in terms of scalability and generalization.
Revised: This makes our approach ﬂexible in terms of scalability and generalization.

Original: To the best of our knowledge,we are the ﬁrst to propose a RL based strategy for explaining AI made decisions and we term ourapproach as R einforcement Ex planation L earning (RExL).
Revised: To the best of our knowledge, we are the ﬁrst to propose a RL based strategy for explaining AI made decisions and we term our approach as R einforcement Ex planation L earning (RExL).

Original: We perform experiments on three popular benchmarks in Computer Vision, ImageNet [8], PASCAL
Revised: We perform experiments on three popular benchmarks in Computer Vision, ImageNet [8], PASCAL

Original: VOC [9] and MSCOCO [17].
Revised: VOC [9] and MSCOCO [17].

Original: For each of these datasets, we use VGG 16 [34] and ResNet 50 [14] asthe base models i.e. the classiﬁers who’s decisions are to be explained.
Revised: For each of these datasets, we use VGG 16 [34] and ResNet 50 [14] as the base models i.e. the classiﬁers who’s decisions are to be explained.

Original: We compare our results withrecent black box and white box methods.
Revised: We compare our results with recent black box and white box methods.

Original: We show that RExL does not hurt the performance andprovides competitive results.
Revised: We show that RExL does not hurt the performance and provides competitive results.

Original: Infact it performs better than the baselines in a few cases.
Revised: In fact it performs better than the baselines in a few cases.

Original: Finally wealso show the speedup given by RExL compared to other the black box techniques.
Revised: Finally we also show the speedup given by RExL compared to other the black box techniques.

Original:  Related Work
Revised:  Related Work

Original: There have been a lot of work in Explainable AI [33, 36, 3, 24, 10, 31] recently.
Revised: There have been a lot of work in Explainable AI [33, 36, 3, 24, 10, 31] recently.

Original: These have useddifferent techniques to explain the decisions taken by Deep Learning models.
Revised: These have used different techniques to explain the decisions taken by Deep Learning models.

Original: Some of them [3, 6]leverage the use of Information Theory to maximize the mutual information between the compressedrepresentation and the decision.
Revised: Some of them [3, 6] leverage the use of Information Theory to maximize the mutual information between the compressed representation and the decision.

Original: [3], in addition, also reduces the mutual information between theexplanation and the input to ensure that the explanation chooses the minimal subset of input features.
Revised: [3], in addition, also reduces the mutual information between the explanation and the input to ensure that the explanation chooses the minimal subset of input features.

Original: These involve complex objective functions which are difﬁcult to use.
Revised: These involve complex objective functions which are difﬁcult to use.

Original: Backpropagation Based Methods generate the important measure of a pixel by backpropagating theoutput of a deep neural network back to the input space using gradients or their variants.
Revised: Backpropagation Based Methods generate the important measure of a pixel by backpropagating the output of a deep neural network back to the input space using gradients or their variants.

Original: Some ofthem [2, 33] use the derivatives of a class score with respect to the image as an importance score.
Revised: Some of them [2, 33] use the derivatives of a class score with respect to the image as an importance score.

Original: Guided Backprop [36] and DeConvNet [42] modify the backpropogation scheme for ReLU while
Revised: Guided Backprop [ 36] and DeConvNet [42] modify the backpropogation scheme for ReLU while

Original: Excitation Backprop [43] ensures that the sum of the attribution signal is unity.
Revised: Excitation Backprop [43] ensures that the sum of the attribution signal is unity.

Original: Integrated gradients[25] additionally accumulate gradients along a path from base image to input image.
Revised: Integrated gradients [25] additionally accumulate gradients along a path from base image to input image.

Original: Activationbased methods like CAM [44] (and its variants like Grad-CAM [31] and NormGrad [27]) use alinear combination of activation values across the convolutional layers.
Revised: Activation based methods like CAM [44] (and its variants like Grad-CAM [31] and NormGrad [27]) use a linear combination of activation values across the convolutional layers.

Original: Grad-CAM and NormGraduses gradients information to weigh the maps.
Revised: Grad-CAM and NormGrad uses gradients information to weigh the maps.

Original: These methods can generate saliency maps for anyactivation layer in the network but it is found that later layers, i.e. layers closer to the output generatemore informative saliency maps.
Revised: These methods can generate saliency maps for any activation layer in the network but it is found that later layers, i.e. layers closer to the output generate more informative saliency maps.

Original: These techniques achieve interpretability by making changes to a white-box model or utilising theinternals of it.
Revised: These techniques achieve interpretability by making changes to a white-box model or utilising the internals of it.

Original: Thus, they cannot always be generalised over all model architectures.
Revised: Thus, they cannot always be generalised over all model architectures.

Original: This demandsfor a need of black box techniques that can be used for any base model.
Revised: This demands for a need of black box techniques that can be used for any base model.

Original: LIME [28] is a black boxapproach which draws random samples around each instance to be explained and ﬁts an approximatelocal linear decision model in the vicinity of the input.
Revised: LIME [28] is a black box approach which draws random samples around each instance to be explained and ﬁts an approximate local linear decision model in the vicinity of the input.

Original: For complex non-linear classiﬁers, LIMEfails to achieve good performance.
Revised: For complex non-linear classiﬁers, LIME fails to achieve good performance.

Original: Its dependence on superpixels leads to inferior saliency maps.
Revised: Its dependence on superpixels leads to inferior saliency maps.

Original: Perturbation based methods perturb the inputs (blur some pixels, or add some noise etc) and measurethe response of the model to these.
Revised: Perturbation based methods perturb the inputs (blur some pixels, or add some noise etc) and measure the response of the model to these.

Original: [39] is such a method that also ﬁne grains gradients to avoidadversarial examples.
Revised: [39] is such a method that also ﬁne grains gradients to avoid adversarial examples.

Original: RISE [24] is a perturbation based method that generates a large number ofrandom masks and applies them on the image.
Revised: RISE [24] is a perturbation based method that generates a large number of random masks and applies them on the image.

Original: It then generates the saliency map as the weightedsum of these masks.
Revised: It then generates the saliency map as the weighted sum of these masks.

Original: The weights are the class scores predicted by the model after the mask is applied.
Revised: The weights are the class scores predicted by the model after the mask is applied.

Original: It has been successful in generating decent saliency maps but it uses large number of masks (around5000 ) which makes it computationally expensive and time taking.
Revised: It has been successful in generating decent saliency maps but it uses large number of masks (around 5000 ) which makes it computationally expensive and time taking.

Original: Meaningful perturbations [11] and
Revised: Meaningful perturbations [11] and

Original: Extremal perturbations [10] solve optimization objective on single images to generate explanationsbut these involve several iterations of every images and take time.
Revised: Extremal perturbations [10] solve optimization objective on single images to generate explanations but these involve several iterations of every images and take time.

Original: Reinforcement Learning is increasingly been adopted in standard computer vision tasks like Object
Revised: Reinforcement Learning is increasingly been adopted in standard computer vision tasks like Object

Original: Detection [38, 5, 19], Visual Question Answering [18], Action Recognition [12] etc.
Revised: Detection [38, 5, 19], Visual Question Answering [18], Action Recognition [12] etc.

Original: TRPO [29],
Revised: TRPO [29],

Original: PPO [30] and DQN [21] are the most commonly used RL algorithms.
Revised: PPO [30] and DQN [21] are the most commonly used RL algorithms.

Original: TRPO and PPO are on-policywhile DQN is off-policy.
Revised: TRPO and PPO are on-policy while DQN is off-policy.

Original: There have been lots of developments on Actor Critic Methods [20, 40, 13].
Revised: There have been lots of developments on Actor Critic Methods [20, 40, 13].

Original: These include both off-policy algoritms [13] and on-policy methods [20].
Revised: These include both off-policy algoritms [13] and on-policy methods [20].

Original: There has been some workdone [26] where RL agents are used to perturb the inputs to study model behaviour.
Revised: There has been some work done [26] where RL agents are used to perturb the inputs to study model behaviour.

Original: In most cases,the agent chooses a particular perturbation on the input and then gets rewards from the outputs of adownstream model for the perturbed input.
Revised: In most cases, the agent chooses a particular perturbation on the input and then gets rewards from the outputs of a downstream model for the perturbed input.

Original: We follow a perturbation based explanation techniquethat uses RL to learn an optimal mask and generate accurate saliency maps quickly.
Revised: We follow a perturbation based explanation technique that uses RL to learn an optimal mask and generate accurate saliency maps quickly.

Original:  Proposed Approach
Revised:  Proposed Approach

Original: We now describe the proposed method which leverages Reinforcement Learning for getting saliencybased explanation of a convnet classiﬁer where at any time, most convincing evidences are collecteddepending on what evidences have already been collected.
Revised: We now describe the proposed method which leverages Reinforcement Learning for getting saliency based explanation of a convnet classiﬁer where at any time, most convincing evidences are collected depending on what evidences have already been collected.

Original: 3.1 Problem Deﬁnition
Revised: 3.1 Problem Deﬁnition

Original: Formally, we deal with a dataset D of images where each image I can contain one or more objects.
Revised: Formally, we deal with a dataset D of images where each image I can contain one or more objects.

Original: The base model or the model we want to explain provides the probabilities p c , ∀ c ∈ { 1 , 2 , · · · , C }indicating the presence of at least one object of class c among C classes.
Revised: The base model or the model we want to explain provides the probabilities p c , ∀ c ∈ { 1 , 2 , · · · , C } indicating the presence of at least one object of class c among C classes.

Original: For each object in the image,the explanation generation task is to output importance values for each pixel (or sub-images/patches)of the image.
Revised: For each object in the image, the explanation generation task is to output importance values for each pixel (or sub-images/patches) of the image.

Original: An explanation is good if gradually removing pixels from the image in order ofdecreasing importance score confuses the base model.
Revised: An explanation is good if gradually removing pixels from the image in order of decreasing importance score confuses the base model.

Original: This will be manifested by a sharp drop of theprobability of the object compared to the case when all pixels were present.
Revised: This will be manifested by a sharp drop of the probability of the object compared to the case when all pixels were present.

Original: We cast explanation generation as a Reinforcement Learning (RL) problem in a Markov Decision
Revised: We cast explanation generation as a Reinforcement Learning (RL) problem in a Markov Decision

Original: Process (MDP) setting.
Revised: Process (MDP) setting.

Original: An MDP has a set of actions A , a set of states S and a reward function
Revised: An MDP has a set of actions A , a set of states S and a reward function

Original: R : S ×
Revised: R : S ×

Original: A → R .
Revised: A → R .

Original: At any time instant t , the agent selects an action a t ∈ A , executes it and as aresult, the state changes from s t to s t +1 .
Revised: At any time instant t , the agent selects an action a t ∈ A , executes it and as a result, the state changes from s t to s t +1 .

Original: From the environment, the agent receives 1) a scalar reward
Revised: From the environment, the agent receives 1) a scalar reward

Original: R ( s t , a t ) and 2) the next state in the form of an observation.
Revised: R ( s t , a t ) and 2) the next state in the form of an observation.

Original: The action to be taken by the agent isgoverned by a policy π : S
Revised: The action to be taken by the agent is governed by a policy π : S

Original: → A that takes in environment states at any point of time and outputs aprobability distribution over a set of actions.
Revised: → A that takes in environment states at any point of time and outputs a probability distribution over a set of actions.

Original: The goal of an RL agent is to learn an optimum policyπ ∗ such that the sum of rewards through an episode is maximized.
Revised: The goal of an RL agent is to learn an optimum policy π ∗ such that the sum of rewards through an episode is maximized.

Original: Next we outline the actions, statesand rewards of the proposed RL setup.
Revised: Next we outline the actions, states and rewards of the proposed RL setup.

Original: States : For an image of size H × W , the number of possible masks naturally, is 2 H × W as each pixelcan be either masked or not.
Revised: States : For an image of size H × W , the number of possible masks naturally, is 2 H × W as each pixel can be either masked or not.

Original: The state at any time instant t is the masked image where masked pixelsare replaced with random noise sampled uniformly in the range of the pixel intensity values.
Revised: The state at any time instant t is the masked image where masked pixels are replaced with random noise sampled uniformly in the range of the pixel intensity values.

Original: For areasonably sized image, the state-space and thus the search-space of size 2 H × W can be huge.
Revised: For a reasonably sized image, the state-space and thus the search-space of size 2 H × W can be huge.

Original: Thuswe relax the state-space size by discretizing the image into a K × K grid, ( K being 7 or 14 ).
Revised: Thus we relax the state-space size by discretizing the image into a K × K grid, ( K being 7 or 14 ).

Original: Actions : Denoting the probability given by the base model for the image to belong to class c ∈{ 1 , 2 , · · · , C } as p c , the actions taken by the agent aims to reduce p c to a minimum.
Revised: Actions : Denoting the probability given by the base model for the image to belong to class c ∈ { 1 , 2 , · · · , C } as p c , the actions taken by the agent aims to reduce p c to a minimum.

Original: The agentchooses regions to construct the mask sequentially so that the base model provides a low probabilityor a low score for the masked image to belong to that class.
Revised: The agent chooses regions to construct the mask sequentially so that the base model provides a low probability or a low score for the masked image to belong to that class.

Original: It should be noted that theoretically,the agent can choose an already masked region also.
Revised: It should be noted that theoretically, the agent can choose an already masked region also.

Original: However, a well trained agent would ﬁnd noincentive in performing such an action as the probability value will not reduce much (if at all, itreduces) as a result of masking an already masked region.
Revised: However, a well trained agent would ﬁnd no incentive in performing such an action as the probability value will not reduce much (if at all, it reduces) as a result of masking an already masked region.

Original: For an image divided as K × K grid,the number of possible actions is naturally K 2 .
Revised: For an image divided as K × K grid, the number of possible actions is naturally K 2 .

Original: There seems to be an incentive in terms of trainingcomputation to stop the search early by limiting the agent to take only a preset number of actions.
Revised: There seems to be an incentive in terms of training computation to stop the search early by limiting the agent to take only a preset number of actions.

Original: However, this may restrict the exploration ability of the agent.
Revised: However, this may restrict the exploration ability of the agent.

Original: Thus, to strike a balance, we allow theagent to take as many actions as there are number of grids i.e. , K 2 .
Revised: Thus, to strike a balance, we allow the agent to take as many actions as there are number of grids i.e. , K 2 .

Original: Rewards : The reward function quantiﬁes the worth of the agent’s effort for successfully maskingimportant image regions.
Revised: Rewards : The reward function quantiﬁes the worth of the agent’s effort for successfully masking important image regions.

Original: A successful masking operation in turn, provides a good saliency mapexplaining the decision behind predicting the presence of a class in the image.
Revised: A successful masking operation in turn, provides a good saliency map explaining the decision behind predicting the presence of a class in the image.

Original: This is manifested bythe drop in probability of a class present in it.
Revised: This is manifested by the drop in probability of a class present in it.

Original: As a natural choice, thus, the reward is taken to be thenegative of the probability given by the base model for the class when the perturbed image is passedthrough the base model.
Revised: As a natural choice, thus, the reward is taken to be the negative of the probability given by the base model for the class when the perturbed image is passed through the base model.

Original: It is given by,[Equation]
Revised: It is given by,

Original: where s t is the state and a t is the action executed at timestep t .

Original: p ( t ) c is the probability of the imageto belong to class c , when the masked image at timestep t is passed through the base model.

Original: Tomaximize the discounted return G = E (cid:2) (cid:80) t γ t R ( s t , a t ) (cid:3) , the agent will have to choose the regionssuch that the classiﬁcation score of the class drops as quickly as possible thereby minimizing thedeletion score.

Original: The discount factor γ ∈ (0 , 1] determines the present value of future rewards.

Original: Asγ → 0 , the agent tries to maximize only the immediate rewards, while a value close to 1 means theagent becomes farsighted.

Original: 3.2 Solution Strategy

Original: The goal of the agent is to choose relevant regions of the input image to mask such that the probabilityor the score of the masked image drops as quickly as possible during the interactions with theenvironment until termination.

Original: The core of the problem is to ﬁnd the policy function π mappingstates to actions that guides the agent’s decision making process.

Original: We use a recently proposed Actor

Original: Critic with Experience Replay (ACER) [40] which is an improvement to the traditional actor-criticalgorithm that is more sample efﬁcient.

Original: The policy function of the agent is represented by a multi-layer neural network with learnableparameters θ .

Original: An episode in an RL setting is composed of all the state action pairs from start tothe terminal action and is denoted as τ =

Original: ( s 1 , a 1 , s 2 , a 2 , · · · s T , a T ) where T is the number of stepstaken before the termination of the episode.

Original: The agent learns the optimal policy by maximizing theexpected return J ( θ ) over the possible trajectories.

Original: [Equation]

Original: R ( s t , a t ) is abbreviated as r t +1 above to reduce clutter.

Original: The objective function in Equation (2) canbe directly optimized using standard optimization techniques like SGD but suffers from high varianceespecially under small samples resulting in very fragile convergence.

Original: With the help of the state andaction value functions, the Actor Critic algorithms [15] overcome this limitation.

Original: While the statevalue function V π ( s ) gives the expected return given the agent is at state s and follows the policy π ,the action value function Q π ( s, a ) is the expected return given that the agent performs an action a atstate s and follows the policy π thereafter.

Original: The modiﬁed objective in terms of the advantage A π isgiven by,

Original: [Equation]
Revised: [Equation]

Original: The basic objective function (Equation (3)) optimized by the Actor-Critic methods is inherently onpolicy.

Original: ACER’s objective contains the both off-policy and on-policy components.

Original: For the off-policycomponent, it uses Retrace algorithm [22] to estimate the action value function and further usesimportance sampling.

Original: The on-policy part is very similar to the basic objective shown in Equation (3).

Original: An illustration of our proposed approach RExL is provided in Figure (2).

Original: We provide a saliency mapbased explanation to the classiﬁcation decision made by the base model on an image.

Original: The image is given as the input.

Original: The RL agent of RExL consists of a pretrained feature extarctor and a multi-layerperceptron (MLP).

Original: The feature extractor remains ﬁxed during the training and the learnable MLP

Original: isthe policy network to predict the action.

Original: At every step, the agent observes the perturbed image whichconstitutes it’s state and predicts the next action.

Original: The action is a speciﬁc region in the image to mask.

Original: Once the image is perturbed or the action is executed, the base model gives the classiﬁcation score forthe class to be explained.

Original: This conﬁdence score is used as reward during training (ref.

Original: Equation (1))and also to generate the explanations as saliency maps.

Original: The perturbed image becomes the next stateand the loop continues until termination.

Original: RExL gives the ﬂexibility to train a single agent to explain all images of a dataset ( dataset speciﬁc ),multiple agents each one expertly trained to explain a single class ( class speciﬁc ) or extremelyﬁne-grained agents that are experts in explaining a single image ( image speciﬁc ).

Original: We train the datasetspeciﬁc agents by using the whole training partition of the dataset in which the base model wastrained.

Original: A class speciﬁc agent is trained on a single class of training images.

Original: Finally, an image speciﬁcagent is trained only on the image to be explained.

Original: The explainability of image speciﬁc agents arevery good but this is not scalable.

Original: Although dataset speciﬁc agent is scalable and performs good forimages containing mostly single objects, the performance may degrade for a complex dataset withimages having multiple instances from possibly multiple classes.

Original: As our experiments show, the besttrade-off between performance and scalability can be seen for class speciﬁc agents.

Original: 3.3 Inference

Original: After a model is trained, it is used to generate saliency maps.

Original: Given an image, the agent sequentiallydeletes a block of the grid from it and incrementally builds the saliency map.

Original: Let block b t of an imageis perturbed as a result of the action a t taken by the agent at time t and the change in classiﬁcationconﬁdence from the previous state is δ t .

Original: The saliency weight assigned to block b t is T (cid:80) i =

Original: t λ i − t δ t ,where λ ∈ [0 , 1] is the cumulating factor that gives the due credit to previous deletions that may haveresulted in a greater drop in probability at a later stage.

Original: A λ value equal to 0 is the trivial case whereonly the grid that is deleted most recent gets all the credit.

Original: Finally, the map is normalized so that thesaliency weights sum to 1 .

Original: Experiments

Original: [-Datasets and Evaluation Metrics : We evaluate the performance of our proposed method on threebenchmark datasets namely ImageNet [8], PASCAL VOC 
-]
Revised: [-Datasets and Evaluation Metrics : We evaluate the performance of our proposed method on threebenchmark datasets namely ImageNet [8], PASCAL VOC 
-]

Original: [9] and MSCOCO 2014 [17].

Original: Following the standard practice, we have reported the results from the val , test and val splits respectively.

Original: For all the three datasets, we use pretrained VGG 16 [34] and ResNet 50 [14] as base models.

Original: For

Original: ImageNet, we used the trained models provided by Pytorch Models Zoo 1 .

Original: For PASCAL VOC and

Original: MSCOCO, we used the same multilabel models as RISE and made public by [43].

Original: We follow [24] and use the causal deletion and insertion metrics to compare the performance withdifferent approaches [24, 10, 27, 31].While GradCAM [31] and NormGrad [27] are white boxtechniques that make use of the intermediate activations and gradients of the network, RISE [24] and

Original: Extremal Perturbations [10] are black box techniques that only study the input-output relationship ofthe model.

Original: RISE is a sampling based method and samples thousands of random masks and weighsthem based on their performance.

Original: Extremal Perturbations on the other hand is a learning basedmethod which solves an optimization objective for each image to obtain the corresponding mask.

Original: Deletion metric removes pixels from the image gradually according to the importance weight givenby the saliency map and measures the classiﬁcation score.

Original: The deletion score is the area under thecurve (AUC) of these classiﬁcation scores with percentage of pixels removed.

Original: Similarly the insertionscore evaluates the performance inversely i.e. , uncovering highly blurred image regions graduallyaccording to the importance weights and getting the AUC.

Original: While a low deletion score is indicative ofbetter explanation, in case of insertion score the opposite is true.

Original: Though, RExL follows the deletionphilosophy while training, the insertion score for it is also competitive.

Original: We posit that an agent trainedwith an insertion strategy ( i.e. , starting with a blank canvas and learning to choose relevant regions touncover) would be equally useful and we keep it as a possible future work.

Original: While our primary aim

Original: [Table]

Original: [Table]

Original: was to improve on the inference time of black box model, we ﬁrst show that RExL does not hurt theperformance, infact in some cases, it performs better than even white box methods.

Original: Thereafter weprovide a runtime analysis showing the signiﬁcant speedup given by us compared to similar blackbox XAI strategies.

Original: Implementation Details : The policy feature extractor is ResNet 50 pretrained on the dataset the RLagent being trained on.

Original: The learnable MLP consists of two hidden layers with 256 and 128 unitsrespectively.

Original: We ﬁx the value of K as 7 and the length of the episode, thus, will be 7 2 = 49 .

Original: Wetrained ACER 1 for 2 × 10 6 steps when training class speciﬁc and 10 7 steps when training datasetspeciﬁc and used discount factor, γ = 1 throughout our experiments.

Original: Also for all the inferences, wehave used λ = 1 unless explicitly speciﬁed.

Original: As dataset speciﬁc setting is a single agent to explainall the classes in the dataset, information about the class to be explained was passed as an one-hotencoding along with the image during training.

Original: We used NVIDIA GeForce RTX 2080 Ti GPUs fortraining all our models.

Original: Next we discuss the experimental results for all the three variants of RExL i.e. ,dataset speciﬁc ( RExL-DS ), class speciﬁc ( RExL-CS ) and image speciﬁc ( RExL-IS ).

Original: Due to limitationsin space, we provide the comparison over deletion metric only.

Original: Additional results including insertionsscores, more implementation details and source codes are provided in the supplementary material.

Original: 4.1 Results and Discussions

Original: Table (1) shows a comparative evaluation of the deletion scores on the three benchmark datasetswith two different base models explaining the classes present in the images.

Original: RExL-DS performsbetter when explaining a stronger model like ResNet 50 .

Original: For PASCAL VOC 2007 and COCO 2014datasets, RExL-DS outperforms all the baselines (including the white box approaches like GradCAMand NormGrad) while for ImageNet, its performance is at par (slightly better than most of the otherbaselines).

Original: With VGG 16 , the performance is at par with the competing methods.

Original: The comparativelyhigher deletion scores of RExL-DS for VGG 16 is intuitive as this being an inferior base model thereward for a right choice of image region to delete may not always be right.

Original: [Table]

Original: As we go to class speciﬁc RExL agents (ref Table (2)),our approach becomes more competitive for VGG 16base model while further improvement is seen in caseof ResNet 50 making RExL-CS the new SOTA for PASCAL VOC.

Original: In case of Imagenet, due to the presence of alarge number of classes, we focused on explaining the top10 classes for which VGG16 and ResNet50 classiﬁers givebest performance in terms of top- 1 accuracy.

Original: The averagedeletion score of the RExL-CS agents trained on these 10classes for ResNet50 base model is as low as 0 . 111 while

Original: RISE being the closest with a value of 0 . 149 .

Original: Likewise,for VGG16, the proposed approach gives a value of 0 .

Original: 139with RISE being close with a deletion score of 0 . 149 .

Original: The improvement over RExL-DS shows theﬂexibility of our approach as well as the expertise imbibed by the class speciﬁc models compared toone single agent trained to explain all the classes in the dataset.

Original: Additional results on both Imagenetand PASCAL VOC are provided in the supplementary.

Original: Figure (3) shows a comparison between the saliency maps generated by different baseline methodsand the two variants of RExL. Targeted saliency and much less noise of the RExL maps make themqualitatively better compared to the others.

Original: While extremal perturbation [10] generates less noisymaps, its dependence on manually set area of relevant regions can harm if the object of interest doesnot conform to this preset value (pronounced in the top row for VGG 16 and penultimate row for

Original: ResNet 50 ).

Original: This is especially detrimental for RISE where the probability of the masked regions in theimage is ﬁxed likewise.

Original: RExL, on the other hand, learns to mask regions that is important for makingthe base model decide the class without constraining the area to be prespeciﬁed.

Original: Improvement by ourmethod is also observed when the speciﬁcity of the agent increases and it becomes more expert i.e. ,when we go from dataset-speciﬁc to class-speciﬁc agents.

Original: This is especially prominent for the imageof the cycle in case of VGG 16 and for the image of the cow in case of ResNet 50 .

Original: Although RExL-IS is the extremely speciﬁc, time to train image speciﬁc models for all images isalso high.

Original: As a result, we don’t speciﬁcally evaluate RExL-IS on all images and provide quantitativeresults.

Original: Instead, we do qualitative comparison with existing approaches on a few images along with acomparison of the deletion scores.

Original: Due to space limitation we provide additional qualitative examplesincluding RExL-IS comparisons in the supplementary.

Original: 4.2 Comparison of Inference Time

Original: The major advantage of and motivation for RExL is the targeted fast search of relevant image regionswithout degrading the quality of the explanations.

Original: In this section we compare the inference times of

Original: [Table]

Original: [Table]

Original: RExL with that of the state-of-the-art black box explanation techniques e.g. , RISE and the extremalperturbation approaches.

Original: We do not compare the running time of RExL with white box techniqueslike GradCAM and NormGrad because they are much quicker but come at the cost of peeking intothe model.

Original: Our aim is to bring about considerable speedup in the inference time of a black boxexplanation technique without hurting the performance.

Original: For comparing the running times, we use a computer with one NVIDIA GeForce GTX 1080 Ti, ryzen2700 x CPU and 16 GB DDR 4 3000 MHz RAM.

Original: Table (3) shows the average running times over 100images.

Original: RExL provides almost 10 × speedup over both the approaches for VGG 16 base model.

Original: Theimprovement in speed is almost 5 × and 10 × respectively over RISE and extremal perturbation incase of ResNet 50 base model.

Original: RExL-CS is comparatively fast than RExL-DS as RExL-DS gets anadditional class encoding as input.

Original: The faster running time with comparable or better performanceshows the merit of our RL based explainable AI approach.

Original: 4.3 Ablation on Cumulating Factor

Original: To see the effect of the cumulating factor λ while assigning credit to the chosen image regions weexperimented with different values of it for RExL-DS on the PASCAL VOC dataset.

Original: Table (4) showsthe comparison.

Original: Though the values are close, one important observation is that giving the entirecredit at a certain instant to the region that is deleted most recently ( i.e. , λ =0 ) hurts the performance.

Original: As λ =1 provides the best performance, we went ahead in experimenting with this value for all thedatasets.

Original: Figure (4) shows sample saliency maps with different λ values applied on the same image.

Original: For a large object, naturally, a large portion of the image is responsible for the base model to get itsdecision and this is rightly captured when credit assignment is delayed instead of immediate.

Original: Conclusions

Original: We have presented a reinforcement learning based model for explaining deep classiﬁers.

Original: In contrastto methods that approximate an exhaustive space, we propose to formulate XAI as an intelligentsearch and direct an agent to efﬁciently choose and accumulate evidences responsible behind AImade decisions.

Original: Our model is dataset, category and image speciﬁc striking a perfect balance of modelspeciﬁcity and training burden.

Original: Ours is a plug and play black box approach that is applicable to anyclassiﬁer without requiring to modify the internals of them.

Original: We report encouraging results in threebenchmark datasets achieving 5 to 10 times speed-up over related black box XAI approaches.

Original: Introduction

Original: Deep Learning methods are enjoying enormous success in computer vision, natural language androbotics research.

Original: Over the years, the models have become overly complex and large with millionsof parameters.

Original: For example, GPT- 3 [4] has about 175 billion parameters.

Original: Yet, it remains largelyunclear how the system comes to a decision, how certain the model is about its decision if and whenit can be trusted or when it has to be corrected.

Original: Explainable AI (XAI) is being extensively researchedupon [10, 11, 24, 31, 35, 42, 43, 44] to get a better insight into AI made decisions to not only improvethe models but also to instill accountability in them.

Original: The explanations could beneﬁt users in manyways such as improving safety and fairness when relying on AI decisions, especially in safety criticalapplications e.g. , healthcare [7, 23], autonomous driving [32] or criminal justice [16].

Original: XAI Strategies for deep visual models offer model speciﬁc [1, 28] or saliency map based explanations [44, 31, 24, 27, 11].

Original: This work subscribes to the later where saliency maps highlight relevantregions in an input e.g. , image, video or natural language texts.

Original: Saliency maps are obtained eitherfollowing a black box or a white/grey box technique.

Original: White/grey box approaches [44, 31, 43] eitherassume access to the internals of the base model (the model whose predictions are explained) ormodify the base model which can cost the base model its performance.

Original: This makes them unsuitableto be generalised over all architectures.

Original: Black box methods [24, 11, 10, 42] on the contrary, do not touch the internals of the base model.

Original: Instead, the input image pixels are perturbed and passed through the base model.

Original: The output predictionresulting in from the perturbed image differentiates between the important and non-important regionsas perturbing important or relevant regions affect the output score more than the case when nonimportant image regions are perturbed.

Original: A popular technique to achieve this is to view saliency mapsas weighted masks.

Original: The perturbations to the input are performed by masking portions of the image using these weighted masks.

Original: Different methods differ in the way these masks are constructed.

Original: Fonget al.

Original: [11, 10] solve an optimization objective to construct optimal masks while RISE [24] useslarge number of random masks for an image.

Original: A good explanation is associated with a low deletionscore [24] which measures the drop in probability of a class as pixels are gradually removed accordingto the saliency map given importance.

Original: In this paper, we address explainability as a sequential search problem where a sequence of imagepatches or regions are investigated in order to get the mask giving the saliency map.

Original: Speciﬁcally, anagent starting with an unperturbed image, learns to choose the best image region to perturb giventhe choices made earlier.

Original: The agent chooses the sequence of regions from the currently unperturbedportions of the image so that perturbing the region maximally drops the class probability which inturn, causes a low deletion score.

Original: An approximate exhaustive search is performed by RISE [24] andis shown to achieve good deletion score albeit being prohibitively expensive both in time and memoryrequirements.

Original: Our approach is fundamentally different as RISE does not make any conscious effortto get low deletion score whereas we learn to intelligently search for the mask with a goal to achieveas low deletion score as quickly possible.

Original: The goal oriented learning offered by the proposed methodnot only makes the inference fast but also makes it deterministic.

Original: The core of the problem is to ﬁnd a policy that assesses the image perturbed by the present mask anddecides to perturb a new relevant image region.

Original: Figure (1) illustrates the decision making processof such an agent.

Original: At ﬁrst, the unperturbed image is passed through the base classiﬁer to get theprobability of the class present in the image (an ‘aeroplane’ in Figure (1))

Original: The agent then starts bychoosing a patch from the image and replacing it with uniform random noise.

Original: The perturbed image isassessed by the base model to get the change in probability of the same class.

Original: Subsequently, the agentdecides to pick the most important regions so that perturbing it decreases the probability of the classfurther until the agent decides to stop exploring or the maximum number of patches is perturbed.

Original: Asthe perturbed regions contribute to a drop of class probability, these regions are pivotal evidencesfor the classiﬁer to classify the image.

Original: The signiﬁcance of the perturbed regions behind the baseclassiﬁer’s decision is manifested by the drop in probability.

Original: Thus at every step, the chosen regiongets an weight proportional to the drop of the class probability.

Original: This weighted regions provide adiscrete weighted mask that after suitable smoothening provides the saliency map explaining the decision of the classiﬁer.

Original: As region perturbation is sequential and its effect is cumulative on the basemodel, we introduce a cumulating factor λ ∈ (0 , 1) that partially credits the regions perturbed earlierbut may have resulted in the subsequent removal of a later region giving rise to a signiﬁcant drop inclass probability.

Original: Learning such a dynamic agent capable to performing sequential actions towards a long-term goal byactively interacting with the environment in absence of any supervision is performed by Reinforcement

Original: Learning (RL).

Original: Motivated by its recent success [37, 41] coupled with its ease of use, we use ACER [40]to learn an RL agent giving the ﬁnal saliency map.

Original: Our approach is equally capable of learning theoptimal perturbing sequence on a single image, a class of images or a whole dataset of images.

Original: Thismakes our approach ﬂexible in terms of scalability and generalization.

Original: To the best of our knowledge,we are the ﬁrst to propose a RL based strategy for explaining AI made decisions and we term ourapproach as R einforcement

Original: Ex planation L earning (RExL).

Original: We perform experiments on three popular benchmarks in Computer Vision, ImageNet [8], PASCAL

Original: VOC [9] and MSCOCO [17].

Original: For each of these datasets, we use VGG 16 [34] and ResNet 50 [14] asthe base models i.e. the classiﬁers who’s decisions are to be explained.

Original: We compare our results withrecent black box and white box methods.

Original: We show that RExL does not hurt the performance andprovides competitive results.

Original: Infact it performs better than the baselines in a few cases.

Original: Finally wealso show the speedup given by RExL compared to other the black box techniques.

Original: Related Work

Original: There have been a lot of work in Explainable AI [33, 36, 3, 24, 10, 31] recently.

Original: These have useddifferent techniques to explain the decisions taken by Deep Learning models.

Original: Some of them [3, 6]leverage

Original: the use of Information Theory to maximize the mutual information between the compressedrepresentation and the decision.

Original: [3], in addition, also reduces the mutual information between theexplanation and the input to ensure that the explanation chooses the minimal subset of input features.

Original: These involve complex objective functions which are difﬁcult to use.

Original: Backpropagation Based Methods generate the important measure of a pixel by backpropagating theoutput of a deep neural network back to the input space using gradients or their variants.

Original: Some ofthem [2, 33] use the derivatives of a class score with respect to the image as an importance score.

Original: Guided Backprop [36] and DeConvNet [42] modify the backpropogation scheme for ReLU while

Original: Excitation Backprop [43] ensures that the sum of the attribution signal is unity.

Original: Integrated gradients[25] additionally accumulate gradients along a path from base image to input image.

Original: Activationbased methods like CAM [44] (and its variants like Grad-CAM [31] and NormGrad [27]) use alinear combination of activation values across the convolutional layers.

Original: Grad-CAM and NormGraduses gradients information to weigh the maps.

Original: These methods can generate saliency maps for anyactivation layer in the network but it is found that later layers, i.e. layers closer to the output generatemore informative saliency maps.

Original: These techniques achieve interpretability by making changes to a white-box model or utilising theinternals of it.

Original: Thus, they cannot always be generalised over all model architectures.

Original: This demandsfor a need of black box techniques that can be used for any base model.

Original: LIME [28] is a black boxapproach which draws random samples around each instance to be explained and ﬁts an approximatelocal linear decision model in the vicinity of the input.

Original: For complex non-linear classiﬁers, LIMEfails to achieve good performance.

Original: Its dependence on superpixels leads to inferior saliency maps.

Original: Perturbation based methods perturb the inputs (blur some pixels, or add some noise etc) and measurethe response of the model to these.

Original: [39] is such a method that also ﬁne grains gradients to avoidadversarial examples.

Original: RISE [24] is a perturbation based method that generates a large number ofrandom masks and applies them on the image.

Original: It then generates the saliency map as the weightedsum of these masks.

Original: The weights are the class scores predicted by the model after the mask is applied.

Original: It has been successful in generating decent saliency maps but it uses large number of masks (around5000 ) which makes it computationally expensive and time taking.

Original: Meaningful perturbations [11] and

Original: Extremal perturbations [10] solve optimization objective on single images to generate explanationsbut these involve several iterations of every images and take time.

Original: Reinforcement Learning is increasingly been adopted in standard computer vision tasks like Object

Original: Detection [38, 5, 19], Visual Question Answering [18], Action Recognition [12] etc.

Original: TRPO [29],

Original: PPO [30] and DQN [21] are the most commonly used RL algorithms.

Original: TRPO and PPO are on-policy while DQN is off-policy.

Original: There have been lots of developments on Actor Critic Methods [20, 40, 13].

Original: These include both off-policy algoritms [13] and on-policy methods [20].

Original: There has been some workdone [26] where RL agents are used to perturb the inputs to study model behaviour.

Original: In most cases,the agent chooses a particular perturbation on the input and then gets rewards from the outputs of adownstream model for the perturbed input.

Original: We follow a perturbation based explanation techniquethat uses RL to learn an optimal mask and generate accurate saliency maps quickly.

Original: Proposed Approach

Original: We now describe the proposed method which leverages Reinforcement Learning for getting saliencybased explanation of a convnet classiﬁer where at any time, most convincing evidences are collecteddepending on what evidences have already been collected.

Original: 3.1 Problem Deﬁnition

Original: Formally, we deal with a dataset D of images where each image I can contain one or more objects.

Original: The base model or the model we want to explain provides the probabilities p c , ∀ c ∈ { 1 , 2 , · · · , C }indicating the presence of at least one object of class c among C classes.

Original: For each object in the image,the explanation generation task is to output importance values for each pixel (or sub-images/patches)of the image.

Original: An explanation is good if gradually removing pixels from the image in order ofdecreasing importance score confuses the base model.

Original: This will be manifested by a sharp drop of theprobability of the object compared to the case when all pixels were present.

Original: We cast explanation generation as a Reinforcement Learning (RL) problem in a Markov Decision

Original: Process (MDP) setting.

Original: An MDP has a set of actions A , a set of states S and a reward function R : S × A → R . At any time instant t , the agent selects an action a t ∈ A , executes it and as aresult, the state changes from s t to s t+1 .
Revised: where s t is the state and a t is the action executed at timestep t .

Original: From the environment, the agent receives 1) a scalar reward

Original: R ( s t , a t ) and 2) the next state in the form of an observation.

Original: The action to be taken by the agent isgoverned by a policy π : S

Original: → A that takes in environment states at any point of time and outputs aprobability distribution over a set of actions.

Original: The goal of an RL agent is to learn an optimum policyπ ∗ such that the sum of rewards through an episode is maximized.

Original: Next we outline the actions, statesand rewards of the proposed RL setup.

Original: States : For an image of size H × W , the number of possible masks naturally, is 2 H × W as each pixelcan be either masked or not.

Original: The state at any time instant t is the masked image where masked pixelsare replaced with random noise sampled uniformly in the range of the pixel intensity values.

Original: For areasonably sized image, the state-space and thus the search-space of size 2 H × W can be huge.

Original: Thuswe relax the state-space size by discretizing the image into a K × K grid, ( K being 7 or 14 ).

Original: Actions : Denoting the probability given by the base model for the image to belong to class c ∈{ 1 , 2 , · · · , C } as p c , the actions taken by the agent aims to reduce p c to a minimum.

Original: The agentchooses regions to construct the mask sequentially so that the base model provides a low probabilityor a low score for the masked image to belong to that class.

Original: It should be noted that theoretically,the agent can choose an already masked region also.

Original: However, a well trained agent would ﬁnd noincentive in performing such an action as the probability value will not reduce much (if at all, itreduces) as a result of masking an already masked region.

Original: For an image divided as K × K grid,the number of possible actions is naturally K 2 .

Original: There seems to be an incentive in terms of trainingcomputation to stop the search early by limiting the agent to take only a preset number of actions.

Original: However, this may restrict the exploration ability of the agent.

Original: Thus, to strike a balance, we allow theagent to take as many actions as there are number of grids i.e. , K 2 .

Original: Rewards : The reward function quantiﬁes the worth of the agent’s effort for successfully maskingimportant image regions.

Original: A successful masking operation in turn, provides a good saliency mapexplaining the decision behind predicting the presence of a class in the image. This is manifested bythe drop in probability ofa class present in it. As a natural choice, thus, the reward is taken tobe thenegative of the probability given by the base model for the class  when the perturbed image  is passedthrough the base model.It is given by,
Revised: p ( t ) c is the probability of the image to belong to class c , when the masked image at timestep t is passed through the base model.

Original: [Equation] ) where s t is the state and a t is the action executed at timestep t . p ( t ) c is the probability of the imageto belong to class c , when the masked image at timestep t is passed through the base model. To maximize the discounted return G = E (cid:2) (cid:80) t γ t R ( s t , a t ) (cid:3) , the agent will have to choose the regionssuch that the classiﬁcation score of the class drops as quickly as possible thereby minimizing thedeletion score.
Revised: To maximize the discounted return G = E (cid:2) (cid:80) t γ t R ( s t , a t ) (cid:3) , the agent will have to choose the regions such that the classiﬁcation score of the class drops as quickly as possible thereby minimizing the deletion score.

Original: The discount factor γ ∈ (0 , 1] determines the present value of future rewards.
Revised: The discount factor γ ∈ (0 , 1] determines the present value of future rewards.

Original: Asγ → 0 , the agent tries to maximize only the immediate rewards, while a value close to 1 means theagent becomes farsighted.
Revised: As γ → 0 , the agent tries to maximize only the immediate rewards, while a value close to 1 means the agent becomes farsighted.

Original: 3.2 Solution Strategy
Revised: 3.2 Solution Strategy

Original: The goal of the agent is to choose relevant regions of the input image to mask such that the probabilityor the score of the masked image drops as quickly as possible during the interactions with theenvironment until termination.
Revised: The goal of the agent is to choose relevant regions of the input image to mask such that the probability or the score of the masked image drops as quickly as possible during the interactions with the environment until termination.

Original: The core of the problem is to ﬁnd the policy function π mappingstates to actions that guides the agent’s decision making process.
Revised: The core of the problem is to ﬁnd the policy function π mapping states to actions that guides the agent’s decision making process.

Original: We use a recently proposed Actor
Revised: We use a recently proposed Actor

Original: Critic with Experience Replay (ACER) [40] which is an improvement to the traditional actor-criticalgorithm that is more sample efﬁcient.
Revised: Critic with Experience Replay (ACER) [40] which is an improvement to the traditional actor-critic algorithm that is more sample efﬁcient.

Original: The policy function of the agent is represented by a multi-layer neural network with learnableparameters θ .
Revised: The policy function of the agent is represented by a multi-layer neural network with learnable parameters θ .

Original: An episode in an RL setting is composed of all the state action pairs from start tothe terminal action and is denoted as τ =
Revised: An episode in an RL setting is composed of all the state action pairs from start to the terminal action and is denoted as τ =

Original: ( s 1 , a 1 , s 2 , a 2 , · · · s T , a T ) where T is the number of stepstaken before the termination of the episode.
Revised: ( s 1 , a 1 , s 2 , a 2 , · · · s T , a T ) where T is the number of steps taken before the termination of the episode.

Original: The agent learns the optimal policy by maximizing theexpected return J ( θ ) over the possible trajectories.
Revised: The agent learns the optimal policy by maximizing the expected return J ( θ ) over the possible trajectories.

Original: [Equation]
Revised: [Equation]

Original: R ( s t , a t ) is abbreviated as r t +1 above to reduce clutter.
Revised: R ( s t , a t ) is abbreviated as r t +1 above to reduce clutter.

Original: The objective function in Equation (2) canbe directly optimized using standard optimization techniques like SGD but suffers from high varianceespecially under small samples resulting in very fragile convergence.
Revised: The objective function in Equation (2) can be directly optimized using standard optimization techniques like SGD but suffers from high variance especially under small samples resulting in very fragile convergence.

Original: With the help of the state andaction value functions, the Actor Critic algorithms [15] overcome this limitation.
Revised: With the help of the state and action value functions, the Actor Critic algorithms [15] overcome this limitation.

Original: While the statevalue function V π ( s ) gives the expected return given the agent is at state s and follows the policy π ,the action value function Q π ( s, a ) is the expected return given that the agent performs an action a atstate s and follows the policy π thereafter.
Revised: While the state value function V π ( s ) gives the expected return given the agent is at state s and follows the policy π , the action value function Q π ( s, a ) is the expected return given that the agent performs an action a at state s and follows the policy π thereafter.

Original: The modiﬁed objective in terms of the advantage A π isgiven by,
Revised: The modiﬁed objective in terms of the advantage A π is given by,

Original: [Equation]
Revised: [Equation]

Original: where A π θ t =
Revised: where A π θ t =

Original: Q π θ ( s t , a t 
Revised: Q π θ ( s t , a t ) − V π θ ( s t ) [Equation]

Original: [Equation]
Revised: [Equation]

Original: − V π θ ( s t ) (3) The basic objective function (Equation (3)) optimized by the Actor-Critic methods is inherently onpolicy.
Revised: The basic objective function (Equation (3)) optimized by the Actor-Critic methods is inherently on policy.

Original: ACER’s objective contains the both off-policy and on-policy components.
Revised: ACER’s objective contains the both off-policy and on-policy components.

Original: For the off-policycomponent, it uses Retrace algorithm [22] to estimate the action value function and further usesimportance sampling.
Revised: For the off-policy component, it uses Retrace algorithm [22] to estimate the action value function and further uses importance sampling.

Original: The on-policy part is very similar to the basic objective shown in Equation (3).
Revised: The on-policy part is very similar to the basic objective shown in Equation (3).

Original: An illustration of our proposed approach RExL is provided in Figure (2).
Revised: An illustration of our proposed approach RExL is provided in Figure (2).

Original: We provide a saliency mapbased explanation to the classiﬁcation decision made by the base model on an image.
Revised: We provide a saliency map based explanation to the classiﬁcation decision made by the base model on an image.

Original: The image isgiven as the input.
Revised: The image is given as the input.

Original: The RL agent of RExL consists of a pretrained feature extarctor and a multi-layer perceptron (MLP).
Revised: The RL agent of RExL consists of a pretrained feature extarctor and a multi-layer perceptron (MLP).

Original: The feature extractor remains ﬁxed during the training and the learnable MLP isthe policy network to predict the action.
Revised: The feature extractor remains ﬁxed during the training and the learnable MLP is the policy network to predict the action.

Original: At every step, the agent observes the perturbed image whichconstitutes it’s state and predicts the next action.
Revised: At every step, the agent observes the perturbed image which constitutes it’s state and predicts the next action.

Original: The action is a speciﬁc region in the image to mask.
Revised: The action is a speciﬁc region in the image to mask.

Original: Once the image is perturbed or the action is executed, the base model gives the classiﬁcation score forthe class to be explained.
Revised: Once the image is perturbed or the action is executed, the base model gives the classiﬁcation score for the class to be explained.

Original: This conﬁdence score is used as reward during training (ref.
Revised: This conﬁdence score is used as reward during training (ref.

Original: Equation (1))and also to generate the explanations as saliency maps.
Revised: Equation (1)) and also to generate the explanations as saliency maps.

Original: The perturbed image becomes the next stateand the loop continues until termination.
Revised: The perturbed image becomes the next state and the loop continues until termination.

Original: RExL gives the ﬂexibility to train a single agent to explain all images of a dataset ( dataset speciﬁc ),multiple agents each one expertly trained to explain a single class ( class speciﬁc ) or extremelyﬁne-grained agents that are experts in explaining a single image ( image speciﬁc ).
Revised: RExL gives the ﬂexibility to train a single agent to explain all images of a dataset ( dataset speciﬁc ), multiple agents each one expertly trained to explain a single class ( class speciﬁc ) or extremely ﬁne-grained agents that are experts in explaining a single image ( image speciﬁc ).

Original: We train the datasetspeciﬁc agents by using the whole training partition of the dataset in which the base model wastrained.
Revised: We train the dataset speciﬁc agents by using the whole training partition of the dataset in which the base model was trained.

Original: A class speciﬁc agent is trained on a single class of training images.
Revised: A class speciﬁc agent is trained on a single class of training images.

Original: Finally, an image speciﬁcagent is trained only on the image to be explained.
Revised: Finally, an image speciﬁc agent is trained only on the image to be explained.

Original: The explainability of image speciﬁc agents arevery good but this is not scalable.
Revised: The explainability of image speciﬁc agents are very good but this is not scalable.

Original: Although dataset speciﬁc agent is scalable and performs good forimages containing mostly single objects, the performance may degrade for a complex dataset withimages having multiple instances from possibly multiple classes.
Revised: Although dataset speciﬁc agent is scalable and performs good for images containing mostly single objects, the performance may degrade for a complex dataset with images having multiple instances from possibly multiple classes.

Original: As our experiments show, the besttrade-off between performance and scalability can be seen for class speciﬁc agents.
Revised: As our experiments show, the best trade-off between performance and scalability can be seen for class speciﬁc agents.

Original: 3.3 Inference
Revised: 3.3 Inference

Original: After a model is trained, it is used to generate saliency maps.
Revised: After a model is trained, it is used to generate saliency maps.

Original: Given an image, the agent sequentiallydeletes a block of the grid from it and incrementally builds the saliency map.
Revised: Given an image, the agent sequentially deletes a block of the grid from it and incrementally builds the saliency map.

Original: Let block b t of an imageis perturbed as a result of the action a t taken by the agent at time t and the change in classiﬁcationconﬁdence from the previous state is δ t .
Revised: Let block b t of an image is perturbed as a result of the action a t taken by the agent at time t and the change in classiﬁcation conﬁdence from the previous state is δ t .

Original: The saliency weight assigned to block b t is T (cid:80) i =
Revised: The saliency weight assigned to block b t is T (cid:80) i =

Original: t λ i − t δ t ,where λ ∈ [0 , 1] is the cumulating factor that gives the due credit to previous deletions that may haveresulted in a greater drop in probability at a later stage.
Revised: t λ i − t δ t , where λ ∈ [0 , 1] is the cumulating factor that gives the due credit to previous deletions that may have resulted in a greater drop in probability at a later stage.

Original: A λ value equal to 0 is the trivial case whereonly the grid that is deleted most recent gets all the credit.
Revised: A λ value equal to 0 is the trivial case where only the grid that is deleted most recent gets all the credit.

Original: Finally, the map is normalized so that thesaliency weights sum to 1 .
Revised: Finally, the map is normalized so that the saliency weights sum to 1 .

Original:  Experiments
Revised:  Experiments

Original: Datasets and Evaluation Metrics : We evaluate the performance of our proposed method on threebenchmark datasets namely ImageNet [8], PASCAL VOC 

Revised: Datasets and Evaluation Metrics : We evaluate the performance of our proposed method on three benchmark datasets namely ImageNet [8], PASCAL VOC

Original: [9] and MSCOCO 2014 [17].
Revised: [9] and MSCOCO 2014 [17].

Original: Following the standard practice, we have reported the results from the val , test and val splits respectively.
Revised: Following the standard practice, we have reported the results from the val , test and val splits respectively.

Original: For all the three datasets, we use pretrained VGG 16 [34] and ResNet 50 [14] as base models.
Revised: For all the three datasets, we use pretrained VGG 16 [34] and ResNet 50 [14] as base models.

Original: For
Revised: For

Original: ImageNet, we used the trained models provided by Pytorch Models Zoo 1 .
Revised: ImageNet, we used the trained models provided by Pytorch Models Zoo 1 .

Original: For PASCAL VOC and
Revised: For PASCAL VOC and

Original: MSCOCO, we used the same multilabel models as RISE and made public by [43].
Revised: MSCOCO, we used the same multilabel models as RISE and made public by [43].

Original: We follow [24] and use the causal deletion and insertion metrics to compare the performance withdifferent approaches [24, 10, 27, 31].While GradCAM [31] and NormGrad [27] are white boxtechniques that make use of the intermediate activations and gradients of the network, RISE [24] and
Revised: We follow [24] and use the causal deletion and insertion metrics to compare the performance with different approaches [24, 10, 27, 31].While GradCAM [31] and NormGrad [27] are white box techniques that make use of the intermediate activations and gradients of the network, RISE [24] and

Original: Extremal Perturbations [10] are black box techniques that only study the input-output relationship ofthe model.
Revised: Extremal Perturbations [10] are black box techniques that only study the input-output relationship of the model.

Original: RISE is a sampling based method and samples thousands of random masks and weighsthem based on their performance.
Revised: RISE is a sampling based method and samples thousands of random masks and weighs them based on their performance.

Original: Extremal Perturbations on the other hand is a learning basedmethod which solves an optimization objective for each image to obtain the corresponding mask.
Revised: Extremal Perturbations on the other hand is a learning based method which solves an optimization objective for each image to obtain the corresponding mask.

Original: Deletion metric removes pixels from the image gradually according to the importance weight givenby the saliency map and measures the classiﬁcation score.
Revised: Deletion metric removes pixels from the image gradually according to the importance weight given by the saliency map and measures the classiﬁcation score.

Original: The deletion score is the area under thecurve (AUC) of these classiﬁcation scores with percentage of pixels removed.
Revised: The deletion score is the area under the curve (AUC) of these classiﬁcation scores with percentage of pixels removed.

Original: Similarly the insertionscore evaluates the performance inversely i.e. , uncovering highly blurred image regions graduallyaccording to the importance weights and getting the AUC.
Revised: Similarly the insertion score evaluates the performance inversely i.e. , uncovering highly blurred image regions gradually according to the importance weights and getting the AUC.

Original: While a low deletion score is indicative ofbetter explanation, in case of insertion score the opposite is true.
Revised: While a low deletion score is indicative of better explanation, in case of insertion score the opposite is true.

Original: Though, RExL follows the deletionphilosophy while training, the insertion score for it is also competitive.
Revised: Though, RExL follows the deletion philosophy while training, the insertion score for it is also competitive.

Original: We posit that an agent trainedwith an insertion strategy ( i.e. , starting with a blank canvas and learning to choose relevant regions touncover) would be equally useful and we keep it as a possible future work.
Revised: We posit that an agent trained with an insertion strategy ( i.e. , starting with a blank canvas and learning to choose relevant regions to uncover) would be equally useful and we keep it as a possible future work.

Original: While our primary aimwas to improve on the inference time of black box model, we ﬁrst show that RExL does not hurt
Revised: While our primary aim was to improve on the inference time of black box model, we ﬁrst show that RExL does not hurt

Original: the
Revised: the

Original: [Table][Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: performance, infact in some cases, it performs better than even white box methods.
Revised: performance, infact in some cases, it performs better than even white box methods.

Original: Thereafter weprovide a runtime analysis showing the signiﬁcant speedup given by us compared to similar blackbox XAI strategies.
Revised: Thereafter we provide a runtime analysis showing the signiﬁcant speedup given by us compared to similar black box XAI strategies.

Revised: Implementation Details : The policy feature extractor can be any good pretrained feature extractor and is independent from the base model.

Original: Implementation Details : The policy feature extractor is ResNet 50 pretrained on the dataset the RLagent being trained on.
Revised: We use ResNet 50 pretrained on the dataset the RL agent being trained on as the feature extractor for the policy.

Original: The learnable MLP consists of two hidden layers with 256 and 128 unitsrespectively.
Revised: The learnable MLP consists of two hidden layers with 256 and 128 units respectively.

Original: We ﬁx the value of K as 7 and the length of the episode, thus, will be 7 2 = 49 .
Revised: We ﬁx the value of K as 7 and the length of the episode, thus, will be 7 2 = 49 .

Original: Wetrained ACER 1 for 2 × 10 6 steps when training class speciﬁc and 10 7 steps when training datasetspeciﬁc and used discount factor, γ = 1 throughout our experiments.
Revised: We trained ACER 2 for 2 × 10 6 steps when training class speciﬁc and 10steps when training dataset speciﬁc and used discount factor, γ = 1 throughout our experiments.

Original: Also for all the inferences, wehave used λ = 1 unless explicitly speciﬁed.
Revised: Also for all the inferences, we have used λ = 1 unless explicitly speciﬁed.

Original: As dataset speciﬁc setting is a single agent to explainall the classes in the dataset, information about the class to be explained was passed as an one-hotencoding along with the image during training.
Revised: As dataset speciﬁc setting is a single agent to explain all the classes in the dataset, information about the class to be explained was passed as an one-hot encoding along with the image during training.

Original: We used NVIDIA GeForce RTX 2080 Ti GPUs fortraining all our models.
Revised: We used NVIDIA GeForce RTX 2080 Ti GPUs for training all our models.

Original: Next we discuss the experimental results for all the three variants of RExL i.e. ,dataset speciﬁc ( RExL-DS ), class speciﬁc ( RExL-CS ) and image speciﬁc ( RExL-IS ).
Revised: Next we discuss the experimental results for all the three variants of RExL i.e. , dataset speciﬁc ( RExL-DS ), class speciﬁc ( RExL-CS ) and image speciﬁc ( RExL-IS ).

Original: Due to limitationsin space, we provide the comparison over deletion metric only.
Revised: Due to limitations in space, we provide the comparison over deletion metric only.

Original: Additional results including insertionsscores, more implementation details and source codes are provided in the supplementary material.
Revised: Additional results including insertions scores, more implementation details and source codes are provided in the supplementary material.

Original: 4.1 Results and Discussions
Revised: 4.1 Results and Discussions

Original:  Table (1) shows a comparative evaluation of the deletion scores on the three benchmark datasetswith two different base models explaining the classes present in the images.
Revised: [Table] Table (1) shows a comparative evaluation of the deletion scores on the three benchmark datasets with two different base models explaining the classes present in the images.

Original: RExL-DS performsbetter when explaining a stronger model like ResNet 50 .
Revised: RExL-DS performs better when explaining a stronger model like ResNet 50 .

Original: For PASCAL VOC 2007 and COCO 2014datasets, RExL-DS outperforms all the baselines (including the white box approaches like GradCAMand NormGrad) while for ImageNet, its performance is at par (slightly better than most of the otherbaselines).
Revised: For PASCAL VOC 2007 and COCO 2014 datasets, RExL-DS outperforms all the baselines (including the white box approaches like GradCAM and NormGrad) while for ImageNet, its performance is at par (slightly better than most of the other baselines). With

Original: With VGG 16 , the performance is at par with the competing methods.
Revised: VGG 16 , the performance is at par with the competing methods.

Original: The comparativelyhigher deletion scores of RExL-DS for VGG 16 is intuitive as this being an inferior base model thereward for a right choice of image region to delete may not always be right.
Revised: The comparatively higher deletion scores of RExL-DS for VGG 16 is intuitive as this being an inferior base model the reward for a right choice of image region to delete may not always be right.

Original: [Table]
Revised: As we go to class speciﬁc RExL agents (ref Table (2)), our approach becomes more competitive for

Original: As we go to class speciﬁc RExL agents (ref Table (2)),our approach becomes more competitive for VGG 16base model while further improvement is seen in caseof ResNet 50 making RExL-CS the new SOTA for PASCAL VOC.
Revised: VGG 16 base model while further improvement is seen in case of ResNet 50 making RExL-CS the new SOTA for PASCAL VOC.

Original: In case of Imagenet, due to the presence of alarge number of classes, we focused on explaining the top10 classes for which VGG16 and ResNet50 classiﬁers givebest performance in terms of top- 1 accuracy.
Revised: In case of Imagenet, due to the presence of a large number of classes, we focused on explaining the top 10 classes for which VGG16 and ResNet50 classiﬁers give best performance in terms of top- 1 accuracy.

Original: The averagedeletion score of the RExL-CS agents trained on these 10classes for ResNet50 base model is as low as 0 .111 while
Revised: The average deletion score of the RExL-CS agents trained on these 10 classes for ResNet50 base model is as low as 0 .

Original:  RISE being the closest with a value of 0 . 149 .
Revised: 111 while RISE being the closest with a value of 0 . 149 .

Original: Likewise,for VGG16, the proposed approach gives a value of 0 .
Revised: Likewise, for VGG16, the proposed approach gives a value of 0 .

Original: 139with RISE being close with a deletion score of 0 .149 .
Revised: 139 with RISE being close with a deletion score of 0 .

Original: The improvement over RExL-DS shows theﬂexibility of our approach as well as the expertise imbibed by the class speciﬁc models compared toone single agent trained to explain all the classes in the dataset.
Revised: The improvement over RExL-DS shows the ﬂexibility of our approach as well as the expertise imbibed by the class speciﬁc models compared to one single agent trained to explain all the classes in the dataset.

Original: Additional results on both Imagenetand PASCAL VOC are provided in the supplementary.
Revised: Additional results on both Imagenet and PASCAL VOC are provided in the supplementary.

Original: Figure (3) shows a comparison between the saliency maps generated by different baseline methodsand the two variants of RExL. Targeted saliency and much less noise of the RExL maps make themqualitatively better compared to the others.
Revised: Figure (3) shows a comparison between the saliency maps generated by different baseline methods and the two variants of RExL. Targeted saliency and much less noise of the RExL maps make them qualitatively better compared to the others.

Original: While extremal perturbation [10] generates less noisymaps, its dependence on manually set area of relevant regions can harm if the object of interest doesnot conform to this preset value (pronounced in the top row for VGG 16 and penultimate row for
Revised: While extremal perturbation [10] generates less noisy maps, its dependence on manually set area of relevant regions can harm if the object of interest does not conform to this preset value (pronounced in the top row for VGG 16 and penultimate row for

Original: ResNet 50 ).
Revised: ResNet 50 ).

Original: This is especially detrimental for RISE where the probability of the masked regions in theimage is ﬁxed likewise.
Revised: This is especially detrimental for RISE where the probability of the masked regions in the image is ﬁxed likewise.

Original: RExL, on the other hand, learns to mask regions that is important for makingthe base model decide the class without constraining the area to be prespeciﬁed.
Revised: RExL, on the other hand, learns to mask regions that is important for making the base model decide the class without constraining the area to be prespeciﬁed.

Original: Improvement by ourmethod is also observed when the speciﬁcity of the agent increases and it becomes more expert i.e. ,when we go from dataset-speciﬁc to class-speciﬁc agents.
Revised: Improvement by our method is also observed when the speciﬁcity of the agent increases and it becomes more expert i.e. , when we go from dataset-speciﬁc to class-speciﬁc agents.

Original: This is especially prominent for the imageof the cycle in case of VGG 16 and for the image of the cow in case of ResNet 50 .
Revised: This is especially prominent for the image of the cycle in case of VGG 16 and for the image of the cow in case of ResNet 50 .

Original: Although RExL-IS is the extremely speciﬁc, time to train image speciﬁc models for all images isalso high.
Revised: Although RExL-IS is the extremely speciﬁc, time to train image speciﬁc models for all images is also high.

Original: As a result, we don’t speciﬁcally evaluate RExL-IS on all images and provide quantitativeresults.
Revised: As a result, we don’t speciﬁcally evaluate RExL-IS on all images and provide quantitative results.

Original: Instead, we do qualitative comparison with existing approaches on a few images along with acomparison of the deletion scores.
Revised: Instead, we do qualitative comparison with existing approaches on a few images along with a comparison of the deletion scores.

Original: Due to space limitation we provide additional qualitative examplesincluding RExL-IS comparisons in the supplementary.[Figure]
Revised: Due to space limitation we provide additional qualitative examples including RExL-IS comparisons in the supplementary.

Original: .2 Comparison of Inference Time
Revised: 4.2 Comparison of Inference Time

Original: The major advantage of and motivation for RExL is the targeted fast search of relevant image regionswithout degrading the quality of the explanations.
Revised: The major advantage of and motivation for RExL is the targeted fast search of relevant image regions without degrading the quality of the explanations.

Original: In this section we compare the inference times of
Revised: In this section we compare the inference times of

Original: [Table] [Table]
Revised: [Table] [Table]

Original: RExL with that of the state-of-the-art black box explanation techniques e.g. , RISE and the extremalperturbation approaches.
Revised: RExL with that of the state-of-the-art black box explanation techniques e.g. , RISE and the extremal perturbation approaches.

Original: We do not compare the running time of RExL with white box techniqueslike GradCAM and NormGrad because they are much quicker but come at the cost of peeking intothe model.
Revised: We do not compare the running time of RExL with white box techniques like GradCAM and NormGrad because they are much quicker but come at the cost of peeking into the model.

Original: Our aim is to bring about considerable speedup in the inference time of a black boxexplanation technique without hurting the performance.
Revised: Our aim is to bring about considerable speedup in the inference time of a black box explanation technique without hurting the performance.

Original: For comparing the running times, we use a computer with one NVIDIA GeForce GTX 1080 Ti, ryzen2700 x CPU and 16 GB DDR 4 3000 MHz RAM.
Revised: For comparing the running times, we use a computer with one NVIDIA GeForce GTX 1080 Ti, ryzen 2700 x CPU and 16 GB DDR 4 3000 MHz RAM.

Original: Table (3) shows the average running times over 100images.
Revised: Table (3) shows the average running times overimages.

Original: RExL provides almost 10 × speedup over both the approaches for VGG 16 base model.
Revised: RExL provides almost 10 × speedup over both the approaches for VGG 16 base model.

Original: Theimprovement in speed is almost 5 × and 10 × respectively over RISE and extremal perturbation incase of ResNet 50 base model.
Revised: The improvement in speed is almost 5 × and 10 × respectively over RISE and extremal perturbation in case of ResNet 50 base model.

Original: RExL-CS is comparatively fast than RExL-DS as RExL-DS gets anadditional class encoding as input.
Revised: RExL-CS is comparatively fast than RExL-DS as RExL-DS gets an additional class encoding as input.

Original: The faster running time with comparable or better performanceshows the merit of our RL based explainable AI approach.
Revised: The faster running time with comparable or better performance shows the merit of our RL based explainable AI approach.

Original: 4.3 Ablation on Cumulating Factor
Revised: 4.3 Ablation on Cumulating Factor

Original: To see the effect of the cumulating factor λ while assigning credit to the chosen image regions weexperimented with different values of it for RExL-DS on the PASCAL VOC dataset.
Revised: To see the effect of the cumulating factor λ while assigning credit to the chosen image regions we experimented with different values of it for RExL-DS on the PASCAL VOC dataset.

Original: Table (4) showsthe comparison.
Revised: Table (4) shows the comparison.

Original: Though the values are close, one important observation is that giving the entirecredit at a certain instant to the region that is deleted most recently ( i.e. , λ =0 ) hurts the performance.
Revised: Though the values are close, one important observation is that giving the entire credit at a certain instant to the region that is deleted most recently ( i.e. , λ =0 ) hurts the performance.

Original: As λ =1 provides the best performance, we went ahead in experimenting with this value for all thedatasets.
Revised: As λ =1 provides the best performance, we went ahead in experimenting with this value for all the datasets.

Original: Figure (4) shows sample saliency maps with different λ values applied on the same image.
Revised: Figure (4) shows sample saliency maps with different λ values applied on the same image.

Original: For a large object, naturally, a large portion of the image is responsible for the base model to get itsdecision and this is rightly captured when credit assignment is delayed instead of immediate.
Revised: For a large object, naturally, a large portion of the image is responsible for the base model to get its decision and this is rightly captured when credit assignment is delayed instead of immediate.

Original:  Conclusions
Revised:  Conclusions

Original: We have presented a reinforcement learning based model for explaining deep classiﬁers.
Revised: We have presented a reinforcement learning based model for explaining deep classiﬁers.

Original: In contrastto methods that approximate an exhaustive space, we propose to formulate XAI as an intelligentsearch and direct an agent to efﬁciently choose and accumulate evidences responsible behind AImade decisions.
Revised: In contrast to methods that approximate an exhaustive space, we propose to formulate XAI as an intelligent search and direct an agent to efﬁciently choose and accumulate evidences responsible behind AI made decisions.

Original: Our model is dataset, category and image speciﬁc striking a perfect balance of modelspeciﬁcity and training burden.
Revised: Our model is dataset, category and image speciﬁc striking a perfect balance of model speciﬁcity and training burden.

Original: Ours is a plug and play black box approach that is applicable to anyclassiﬁer without requiring to modify the internals of them.
Revised: Ours is a plug and play black box approach that is applicable to any classiﬁer without requiring to modify the internals of them.

Original: We report encouraging results in threebenchmark datasets achieving 5 to 10 times speed-up over related black box XAI approaches.
Revised: We report encouraging results in three benchmark datasets achieving 5 to 10 times speed-up over related black box XAI approaches.

Original:  Implementation Details
Revised:  Implementation Details

Original: In this section, we discuss all the details that are needed to reproduce our results as well as the scoresobtained for the baseline methods.
Revised: In this section, we discuss all the details that are needed to reproduce our results as well as the scores obtained for the baseline methods.

Original: These parameters can be categorized as follows:
Revised: These parameters can be categorized as follows:

Original: 1.1 Environment and Agent Parameters
Revised: 1.1 Environment and Agent Parameters

Original: The mask is discretized into a K × K grid as discussed in section 3 of the main paper.
Revised: The mask is discretized into a K × K grid as discussed in section 3 of the main paper.

Original: For ourexperiments we choose K = 7 .
Revised: For our experiments we choose K = 7 .

Original: Naturally, the number of steps in an episode as discussed willbe 49 .
Revised: Naturally, the number of steps in an episode as discussed will be 49 .

Original: For VOC 2007 and COCO 2014 , we used VGG 16 and ResNet 50 trained by [6].
Revised: For VOC 2007 and COCO 2014 , we used VGG 16 and ResNet 50 trained by [6].

Original: Thesemodels are trained on BGR images loaded with pixel values in [0 , 255] and then mean correctedto [104 . 01 , 116 . 67 , 122 . 68] respectively.
Revised: These models are trained on BGR images loaded with pixel values in [0 , 255] and then mean corrected to [104 . 01 , 116 . 67 , 122 . 68] respectively.

Original: For ImageNet, we use the pretrained models provided by
Revised: For ImageNet, we use the pretrained models provided by

Original: Pytorch model zoo 1 and used the standard normalization practices described there.
Revised: Pytorch model zoo 1 and used the standard normalization practices described there.

Original: The randomnoise that is used to mask the image is sampled from the same distribution as that of the image.
Revised: The random noise that is used to mask the image is sampled from the same distribution as that of the image.

Original: Forexample in case of VOC or COCO, the noise has a mean [104 . 01 , 116 .
Revised: For example in case of VOC or COCO, the noise has a mean [104 . 01 , 116 .

Original: The RL agent consists of two parts, a feature extractor which is ﬁxed during training and a learnable
Revised: The RL agent consists of two parts, a feature extractor which is ﬁxed during training and a learnable

Original: MLP.
Revised: MLP.

Original: The feature extractor used is a pretrained ResNet 50 .
Revised: The feature extractor used is a pretrained ResNet 50 .

Original: So, for VOC 2007 , we will be using the
Revised: So, for VOC 2007 , we will be using the

Original: ResNet 50 module that was pretrained on VOC 2007 by [6].
Revised: ResNet 50 module that was pretrained on VOC 2007 by [6].

Original: The learnable MLP consists of two hiddenlayers of 256 and 128 units each.
Revised: The learnable MLP consists of two hidden layers of 256 and 128 units each.

Original: The activation functions at these hidden units is ReLU.
Revised: The activation functions at these hidden units is ReLU.

Original: The outputlayer predicts the catergorical distribution over actions and hence contains 49( K 2 ) output neurons.
Revised: The output layer predicts the catergorical distribution over actions and hence contains 49( K 2 ) output neurons.

Original: [Table]
Revised: [Table]

Original: During inference, as reported in section 4.4, λ = 1 givesthe best performance so we report the results using λ = 1 .
Revised: During inference, as reported in section 4.4, λ = 1 gives the best performance so we report the results using λ = 1 .

Original: RExL agents are trained on the respective partitions ofthe train split and following standard practices [3, 2, 6]tested
Revised: RExL agents are trained on the respective partitions of the train split and following standard practices [3, 2, 1, 6] tested on the val, test and val split of ImageNet, VOC and

Original: COCO.
Revised: COCO.

Original: 1.2 RL training parameters
Revised: 1.2 RL training parameters

Original: We used ACER 2 to train the policy.
Revised: We used ACER 2 to train the policy.

Original: The parameters thatwe used during training are in Table (1).
Revised: The parameters that we used during training are in Table (1).

Original: We used thedefault hyperparameters as mentioned in the ofﬁcial documentation 3 for the rest of the parameters.
Revised: We used the default hyperparameters as mentioned in the ofﬁcial documentation 3 for the rest of the parameters.

Original: 1.3 Baselines
Revised: 1.3 Baselines

Original: For RISE we used the ofﬁcial implementation 4 with the hyperparameters discussed in [3].
Revised: For RISE we used the ofﬁcial implementation 4 with the hyperparameters discussed in [3].

Original: For
Revised: For

Original: GradCAM and NormGrad, we used the publically available implementation 5 .
Revised: GradCAM and NormGrad, we used the publically available implementation 5 .

Original: Finally for Extremal
Revised: Finally for Extremal

Original:  Perturbations, we used the implementation in the ofﬁcial repository 6 using contrastive rewards andarea parameter of 0 .
Revised: [Table] Perturbations, we used the implementation in the ofﬁcial repository 6 using contrastive rewards and area parameter of 0 .

Original: 12 while generating explanations.[Table]
Revised: 12 while generating explanations.

Original:  Insertion 
Revised:  Insertion Scores

Original: [Table]
Revised: [Table]

Original: RExL gets competitive scores for all the datasets even[Table] though the preocess of insertion is complementary to ourtraining process.
Revised: RExL gets competitive scores for all the datasets even though the preocess of insertion is complementary to our training process.

Original: A method that optimizes deletion scoreswill not necessary also give good insertion scores.
Revised: A method that optimizes deletion scores will not necessary also give good insertion scores.

Original: Fordeletion score to be low, the agent will learn the mask thatwill remove some features that increases the confusion inthe model for that class.
Revised: For deletion score to be low, the agent will learn the mask that will remove some features that increases the confusion in the model for that class.

Original: But inserting the same featuresmay not necessarily lead to a high probability for thatclass.
Revised: But inserting the same features may not necessarily lead to a high probability for that class.

Original: For example, in ImageNet, there are a lot of classespertaining to different kinds of dogs.
Revised: For example, in ImageNet, there are a lot of classes pertaining to different kinds of dogs.

Original: There will be somefeatures (like the some portion of the face) on removal ofwhich, the base model will get confused among all the other dogs.
Revised: There will be some features (like the some portion of the face) on removal of which, the base model will get confused among all the other dogs.

Original: But if we insert this feature on ablank image, the base model will not necessarily classify it correctly.
Revised: But if we insert this feature on a blank image, the base model will not necessarily classify it correctly.

Original: Inference
Revised: Inference

Original: VGG ResNet λ =0. λ = 0 .0.
Revised: [Table]

Original: λ

Original: = 0 .0. λ =0. Naturally this will be more pronounced where the classes area lot similar to each other.
Revised: Naturally this will be more pronounced where the classes are a lot similar to each other.

Original: As a result, (ref.
Revised: As a result, (ref.

Original: Table (2)) RExLdoes not do as well in terms of insertion scores (compared to)on ImageNet but does very well on VOC where the classes area lot different.
Revised: Table (2)) RExL does not do as well in terms of insertion scores (compared to) on ImageNet but does very well on VOC where the classes are a lot different.

Original: Similar to the deletion score, insertion scorefor RExL-CS is better than RExL-DS, as seen from Table (3).
Revised: Similar to the deletion score, insertion score for RExL-CS is better than RExL-DS, as seen from Table (3).

Original: RExL-CS results will be further analyzed in the next sections.
Revised: RExL-CS results will be further analyzed in the next sections.

Original: One interesting observation in Table (4) is about the generaltrend that shows insertion scores are better for lower values ofλ .
Revised: One interesting observation in Table (4) is about the general trend that shows insertion scores are better for lower values of λ .

Original: This is counterintuitive to the trend seen in case of deletionscores (Table 4 in the main paper).
Revised: This is counterintuitive to the trend seen in case of deletion scores (Table 4 in the main paper).

Original: For an object, the lowervalues of λ credit the later actions more while the larger λ value the earlier actions.
Revised: For an object, the lower values of λ credit the later actions more while the larger λ value the earlier actions.

Original: The results showthat the later actions are more important for insertion scores.
Revised: The results show that the later actions are more important for insertion scores.

Original: These are the actions that are probablyremoving the signiﬁcant portions of the object while the earlier actions aim to remove the featuresthat separate similar classes.
Revised: These are the actions that are probably removing the signiﬁcant portions of the object while the earlier actions aim to remove the features that separate similar classes.

Original: We posit that using an insertion-style training can lead to agents whichwill do better in terms of insertion scores too.
Revised: We posit that using an insertion-style training can lead to agents which will do better in terms of insertion scores too.

Original: This is a possible future work.
Revised: This is a possible future work.

Original:  RExL-CS on VOC2007 
Revised:  RExL-CS on VOC

Original: For VOC 2007 , we train RExL-CS on all the 20 classes.
Revised: For VOC 2007 , we train RExL-CS on all the 20 classes.

Original: Saliency maps generated by RExL-CS iscompared with the baseline methods for every class.
Revised: Saliency maps generated by RExL-CS is compared with the baseline methods for every class.

Original: For quantitative comparisons, we use both thedeletion scores and insertion scores.
Revised: For quantitative comparisons, we use both the deletion scores and insertion scores.

Original: The maps generated by RExL-CS and baselines for every classare also compared qualitatively.
Revised: The maps generated by RExL-CS and baselines for every class are also compared qualitatively.

Original: 3.1 VGG16 
Revised: 3.1 VGG

Original: Figure (1) and (2) show the comparison of class wise deletion and insertion scores among the differentmethods.
Revised: Figure (1) and (2) show the comparison of class wise deletion and insertion scores among the different methods.

Original: Moreover, Figure (3) and Figure (4) qualitatively compare the saliency maps generated by
Revised: Moreover, Figure (3) and Figure (4) qualitatively compare the saliency maps generated by

Original: RExL-CS with the baselines for every class.
Revised: RExL-CS with the baselines for every class.

Original: VGG 16 gives competitive results for both deletion andinsertion scores.
Revised: VGG 16 gives competitive results for both deletion and insertion scores.

Original: The saliency maps produced are also more accurate and concise.
Revised: The saliency maps produced are also more accurate and concise.

Original: 3.2 ResNet50 
Revised: 3.2 ResNet

Original: Similarly, RExL-CS agents are trained for all the 20 classes using ResNet 50 as the base model.
Revised: Similarly, RExL-CS agents are trained for all the 20 classes using ResNet 50 as the base model.

Original: Figure(5) and Figure (6) respectively show the comparison of deletion and insertion scores for every class.
Revised: Figure (5) and Figure (6) respectively show the comparison of deletion and insertion scores for every class.

Original: As seen in all the results, RExL-CS improves considerable as we use a stronger base model like
Revised: As seen in all the results, RExL-CS improves considerable as we use a stronger base model like

Original: ResNet 50 .
Revised: ResNet 50 .

Original: Finally, Figure (7) and Figure (8) contains the saliency maps generated by the differentmethods for qualitative comparison.
Revised: Finally, Figure (7) and Figure (8) contains the saliency maps generated by the different methods for qualitative comparison.

Original:  RExL-CS on ImageNet
Revised:  RExL-CS on ImageNet

Original: We also perform classwise comparison of the saliency maps generated by RExL-CS with the baselinemethods on ImageNet.
Revised: We also perform classwise comparison of the saliency maps generated by RExL-CS with the baseline methods on ImageNet.

Original: But instead of all the 1000 classes, we show results only on the top 10 classes(as per the classifcation accuracy).
Revised: But instead of all the 1000 classes, we show results only on the top 10 classes (as per the classifcation accuracy).

Original: Similar to VOC 2007 , we use deletion and insertion scores tocompare quantitatively while qualitatively we show the saliency maps produced for a random examplefrom each of the 10 classes.
Revised: Similar to VOC 2007 , we use deletion and insertion scores to compare quantitatively while qualitatively we show the saliency maps produced for a random example from each of the 10 classes.

Original: Again we perform the experiments for both the base models.
Revised: Again we perform the experiments for both the base models.

Original: [Figure] 
Revised: 4.1 VGG

Original: Figure (9) and (10) show the comparison of class wise deletion and insertion scores among thedifferent methods respectively.
Revised: Figure (9) and (10) show the comparison of class wise deletion and insertion scores among the different methods respectively.

Original: And, Figure (11) qualitatively compare the saliency maps generatedby RExL-CS with the baselines for every class.
Revised: And, Figure (11) qualitatively compare the saliency maps generated by RExL-CS with the baselines for every class.

Original: For VGG16, the scores are at par with the baselines,outperforming the baselines few times.
Revised: For VGG16, the scores are at par with the baselines, outperforming the baselines few times.

Original: 4.2 ResNet50 
Revised: 4.2 ResNet

Original: We perform similar comparisons using ResNet 50 as the base model.
Revised: We perform similar comparisons using ResNet 50 as the base model.

Original: Figure (12) and (13) compare theclass wise deletion and insertion scores among the different methods respectively.
Revised: Figure (12) and (13) compare the class wise deletion and insertion scores among the different methods respectively.

Original: And, Figure (14)qualitatively compares the saliency maps generated by RExL-CS with the baselines for every class.
Revised: And, Figure (14) qualitatively compares the saliency maps generated by RExL-CS with the baselines for every class.

Original: As expected, RExL performs better using ResNet 50 as the base model and this can be illustratedfrom both the scores and the saliency maps.
Revised: As expected, RExL performs better using ResNet 50 as the base model and this can be illustrated from both the scores and the saliency maps.

Original:  RExL-IS
Revised:  RExL-IS

Original: These agents are trained on single images and explain the decisions made by the base model on thatimage only.
Revised: These agents are trained on single images and explain the decisions made by the base model on that image only.

Original: These are hence not scalable for any practical datasets but if the base model needs to beexplained very accurately and for a small number of images, RExL-IS will provide the best results.
Revised: These are hence not scalable for any practical datasets but if the base model needs to be explained very accurately and for a small number of images, RExL-IS will provide the best results.

Original: Acomparison between the different baselines and RExL-IS can be seen in Figure (15).
Revised: A comparison between the different baselines and RExL-IS can be seen in Figure (15).

Original: These imagesare taken from VOC 2007 test split.[Figure]
Revised: These images are taken from VOC 2007 test split.