Spectral Robustness Analysis of Deep Imitation Learning
Abstract
Deep reinforcement learning algorithms enabled learning functioning policies in MDPs with complex state representations.
Following these advancements deep reinforcement learning polices have been deployed in many diverse settings.
However, a line of research argued that in certain settings building a reward function can be more complicated than learning it.
Hence, several studies proposed different methods to learn a reward function by observing trajectories of a functioning policy (i.e. inverse reinforcement learning).
Following this line of research several studies proposed to directly learn a functioning policy by solely observing trajectories of an expert (i.e. imitation learning).
In this paper, we propose a novel method to analyze the spectral robustness of deep neural policies.
We conduct several experiments in the Arcade Learning Environment, and demonstrate that simple vanilla trained deep reinforcement learning policies are more robust than deep imitation learning policies.
We believe that our method provides a comprehensive analysis on the policy robustness and can help in understanding the fundamental properties of different training techniques.
Introduction
The capabilities achieved via interacting with a given environment solely based on observations and receiving rewards upon taking actions in high-dimensional state observation MDPs gained substantial acceleration with the recent advancements in deep reinforcement learning research Mnih et al.
Currently, from robotics objectives to solving complex games, several different ﬁelds from pharmaceuticals to ﬁnance beneﬁted from the advancements achieved in deep neural policies.
While deep reinforcement learning progressed towards solving more complicated tasks, a line of research focused on the questions arising from learning functioning policies without the presence of a reward function.
In this line of research, initially some proposed to learn the reward function via observing the trajectories of a functioning policies (i e. inverse reinforcement learning).
Others proposed to learn a functioning policy via observing expert demonstrations (i e. imitation learning).
The adversarial non-robustness of deep neural networks has been extensively discussed in the machine learning community starting from the pioneering study of Goodfellow et al.
Following this, adversarial vulnerabilities have also been discussed in deep reinforcement learning policies Huang et al.
Korkmaz (2022a, 2021e).
While some of these studies focus on optimization of the adversarial directions Korkmaz (2020), in others these directions are used as a tool to highlight the vulnerability types and variations across different training techniques Korkmaz (2021e).
While the robustness of deep reinforcement learning policies has been extensively studied we focus on the robustness of the state-of-the-art deep inverse reinforcement learning and deep imitation learning policies and in our paper we want to answer the following questions:
• How does learning from expert demonstrations affect the robustness of deep imitation learning policies? • How can we analyze and quantify the robustness of deep imitation learning policies in the frequency spectrum?
Hence, to answer these questions in this paper we focus on analyzing the spectral properties of deep inverse reinforcement learning policies, and make the following contributions:
• We propose a novel method to analyze deep inverse reinforcement learning and deep imitation learning policy robustness in the frequency spectrum.
• We conduct experiments in the Arcade Learning Environment (ALE) and we compare the frequency vulnerabilities of the state-of-the-art imitation learning policy to the vanilla deep reinforcement learning algorithm for high dimensional state representation environments.
• Our method reveals the spectral contrast between the vanilla deep reinforcement learning policies and the state-of-the-art imitation learning policies.
Background and Related Work 2.1 Reinforcement Learning
A Markov Decision Process (MDP) is represented as a tuple (cid:104) S, A, P , r, γ, τ 0 (cid:105) of a set of states S , a set of actions A , transition probability distribution P ( s t +1 | s t , a t ) , and a reward function r : S ×
A → R , discount factor γ , and initial state distribution τ 0 .
The objective in reinforcement learning is to learn a policy that will maximize the expected discounted cumulative rewards obtained by the policy π : S → P ( A ) .
This objective can be achieved via Q -learning that essentially learns a Q function
Q : S ×
A → R that will assign values to each state-action ( s, a ) pair to reveal what would be the expected cumulative discounted rewards obtained if the action a is taken in state s .
The Q -function is learnt via iterative Bellman update
[Equation]
Upon the construction of the state-action value function the policy would execute the action that maximizes the state-action value function.
[Equation]
2.2 Deep Inverse Reinforcement Learning
For a given setting where the reward function is not present, inverse reinforcement learning was proposed to learn a reward function by observing the trajectories of a functioning policy.
The initial study that proposed inverse reinforcement learning achieves this objective via linear programming
Ng & Russell (2000).
[Equation]
While some studies focused on learning the reward function itself others focused on directly learning a policy from demonstrations Kostrikov et al.
Quite recently, Garg et al.
(2021) focused on learning a state-action value function via solely observing the trajectories of a functioning policy (inverse Q -learning).
Inverse Q -learning is the ﬁrst algorithm that can achieve the ability to learn policies in high dimensional state observations.
Most importantly, the authors of this study argue that once the state-action value function is learnt, the reward function can be reconstructed from this information.
Furthermore, note that the inverse Q -learning algorithm can learn a functioning policy and a reward function simultaneously; hence, throughout the paper the inverse Q -learning algorithm will be referred to as an imitation learning and inverse reinforcement learning algorithm interchangeably.
Input: Actions a ∈ A , states s ∈ S , policy π ( s, a ) , δ blocked frequencies, d dimension of the state
Output: Performance Drop for δ = 0 to d/ 2 do for s = s 0 to s T do
[Equation]
end for end for
Return: Impact I
2.3 Robustness in Reinforcement Learning
The adversarial vulnerabilities of deep reinforcement learning policies were initially discussed in
Huang et al.
This study essentially introduces fast gradient sign method produced adversarial perturbations Goodfellow et al.
(2015) in to the observation system of the deep reinforcement learning policy.
In this line of research some studies tried to further optimize adversarial directions Korkmaz (2020), while others focused on contrasting the differences between adversarial and natural directions Korkmaz (2023, 2021d).
Targeting the adversarial vulnerabilities of deep reinforcement learning policies, some studies proposed to train with an adversary to gain robustness to adversarial directions
Gleave et al.
; Pinto et al.
However, some recent studies discussed the problems with adversarial training starting from learning a different set of non-robust features Korkmaz (2021e) 1 , to losing generalization capacity Korkmaz (2023) and learning inaccurate and inconsistent state-action value functions Korkmaz (2021c).
Some further argued that there are underlying shared adversarial directions that are learnt by many different policies independent from the algorithm, and that are shared across MDPs Korkmaz (2022a).
Spectral Robustness Analysis of Deep Imitation Learning and Deep Inverse Reinforcement Learning
In this section we will describe the main proposal of the paper.
In particular, Spectral Robustness
Analysis (SRA) is based on systematically blocking frequencies and analyzing the effects of these frequencies on the deep reinforcement learning policy performance.
In particular, for a state s ∈ S the discrete Fourier transform of the state s is
[Equation]
Upon the blocking the δ -frequencies the discrete Fourier transform is inverted and the observation of the deep neural policy consists of s F
[Equation]
Algorithm 1 provides the pseudocode for the Spectral Robustness Analysis (SRA) algorithm. [Figure] [Equation] [Figure]
[Equation]
[Figure] [Equation] [Figure]  Experimental Analysis
Our experiments are conducted in the Arcade Learning Environment Bellemare et al.
The deep reinforcement learning policy is trained via double- Q learning Hasselt et al.
The experiments are conducted with 10 random runs.
We included standard error of the mean in all of the results presented in the paper.
The impact on the policy performance is measured by
[Equation]
[Equation]
Figure 1 reports the steps of the spectral robustness analysis (SRA) with variations of blocked frequencies.
Figure 2 provides results on the spectral robustness analysis of the deep reinforcement learning policy and the deep inverse reinforcement learning policy as we block the δ -frequencies that have been described in Algorithm 1.
The results reported in Figure 2 demonstrate that the vanilla trained deep reinforcement learning policies are more robust than the policies trained via deep inverse reinforcement learning.
In particular, there is a high increase in the sensitivities towards lower frequencies for the deep inverse reinforcement learning policy.
Conclusion
This paper aims to answer the following questions: (i) How can we analyze the deep neural policies that are trained in MDPs with complex state representations in the spectral domain?
, (ii) Is there a fundamental difference between learning via exploration vs learning via observing functioning policies in terms of their robustness?, and (iii) How can we unveil the fundamental robustness differences between deep reinforcement learning policies and deep inverse reinforcement learning policies?
To be able to address these questions we propose a novel method that provides a comprehensive analysis of the spectral robustness of deep neural policies.
We conduct extensive experiments in the Arcade
Learning Environment and demonstrate that the deep reinforcement learning policies are more robust than the deep inverse reinforcement learning policies.