D ON ’ T T HROW Y OUR O LD P OLICIES A WAY : K NOWLEDGE - BASED P OLICY R ECYCLING P ROTECTS A GAINST A DVERSARIAL A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work has shown that Deep Reinforcement Learning (DRL) is vulnerable to adversarial attacks, in which minor perturbations of input signals cause agents to behave inappropriately and unexpectedly.
Humans, on the other hand, appear robust to these particular sorts of input variations.
We posit that this part of robustness stems from accumulated knowledge about the world.
In this work, we propose to leverage prior knowledge to defend against adversarial attacks in RL settings using a framework we call Knowledge-based Policy Recycling (KPR).
Different from previous defense methods such as adversarial training and robust learning,
KPR incorporates domain knowledge over a set of auxiliary tasks policies and learns relations among them from interactions with the environment via a Graph
Neural Network (GNN).
KPR can use any relevant policy as an auxiliary policy and, importantly, does not assume access or information regarding the adversarial attack.
Empirically, KPR results in policies that are more robust to various adversarial attacks in Atari games and a simulated Robot Foodcourt environment.
I NTRODUCTION
Despite significant performance breakthroughs in recent years (Mnih et al., 2015; Silver et al., 2016;
Berner et al., 2019, e.g.,), Deep Reinforcement Learning (DRL) policies can be brittle.
Specifically, recent works have shown that DRL policies are vulnerable to adversarial attacks — adversarially manipulated inputs (e.g., images) of small magnitude can cause RL agents to take incorrect actions (Ilahi et al., 2022;
Chen et al., 2019; Behzadan & Munir, 2017; Oikarinen et al., 2021; Lee et al., 2021; Chan et al., 2020; Bai et al., 2018).
To counter such attacks, recent work has proposed a range of defense strategies including adversarial training (Oikarinen et al., 2021; Behzadan & Munir, 2018; Han et al., 2018), robust learning (Mandlekar et al., 2017; Smirnova et al., 2019; Pan et al., 2019), defensive distillation (Rusu et al., 2016), and adversarial detection (Gallego et al., 2019a;
Havens et al., 2018; Gallego et al., 2019a).
While these defense methods can be effective, each has its limitations; adversarial training and adversarial detection require specific knowledge about the attacker.
Robust learning adds noise during agent training, which can degrade performance (Tsipras et al., 2019; Yang et al., 2020).
Defensive distillation is typically unable to protect against diverse adversarial attacks (Carlini & Wagner, 2016; Soll et al., 2019).
In this work, we explore an alternative defense strategy that exploits existing knowledge encoded in auxiliary task policies and known relationships between the policies.
The key intuition underlying our approach is that existing task policies encode learnt low-level knowledge regarding the environment (e.g., possible observations, dynamics), whilst high-level specifications can provide guidance for transfer or generalization.
Our approach is to leverage known and learnt relations between different policies as structural priors for an ensemble of policies; our hypothesis is that while a single task policy can be attacked, perturbing inputs such that multiple policies are negatively affected in a consistent manner is more difficult.
Our framework, which we call Kno wledge-based P olicy F usion (KPR), is partially inspired by the use of domain knowledge to address vulnerabilities to adversarial attacks in supervised learning (Melacci et al., 2021; G¨urel et al., 2021;
Zhang et al., 2022).
In these works, domain knowledge is encoded as logical formulae over predicted labels and a set of features.
A soft satisfiability score between
[Table]
[Figure]
distributions (one for each graph).
In other words, each logical graph results in a different policy.
Finally, we combine/fuse these distributions (Section 3.3) to obtain a final action distribution.
In summary, KPR uses the specified logical formula graph as prior structure and then learns flexible relations over auxiliary tasks from interaction data.
the predictions and given logic formulae is added to the objective to encourage the predictions to comply with the logical formulae.
Even if part of the sample is corrupted by the adversary, the final output can be corrected by enforcing the domain knowledge rules.
KPR extends this line of research to RL settings.
Note that this extension is nontrivial; the auxiliary feature detectors used in supervised learning (G¨urel et al., 2021; Zhang et al., 2022) do not capture temporal features, and more importantly, the consistency between the predictions and actions is not directly computable since the optimal actions for each state are unknown.
We address these issues by using auxiliary task policies and their relationships.
In practice, these policies could be the intermediate by-products of curriculum/hierarchical learning or obtained via direct training with sub-goals.
By combining these policies with specified and learnt relations, we construct an ensemble of policies on the target task.
KPR uses graph neural networks (GNNs) as a backbone, which enables natural incorporation of graph-based domain knowledge while retaining the flexibility to learn from interaction data.
The ensemble is then fused in a simple parameter-less manner to obtain a new robust task policy (Figure 1).
From a practical perspective, KPR has a key advantage: it is both policy and attack agnostic.
Specifically, KPR can utilize any type of policy representation (e.g., neural networks, rule-based policies) as either the main task policy or an auxiliary task policy.
In addition, KPR doesn’t require knowledge of the specific attack or access to the adversarial environment.
Our empirical results show that KPR results in more robust policies across multiple attacks compared to baselines in a representative selection of Atari Games (Bellemare et al., 2013) and a high-dimensional Robot Food Court Environment (RoFoCo).
To summarize, this paper contributes K nowledge-based P olicy R ecycling (KPR), which leverages domain knowledge to defend against adversarial attacks in RL.
Different from prior defense methods in reinforcement learning, such as adversarial training and robust learning, KPR is able to incorporate domain knowledge as structural prior and then learn flexible relations from interaction data.
To the best of our knowledge, this is the first work to demonstrate that domain knowledge in the form of policies can be used to defend against adversarial attacks.
B ACKGROUND AND R ELATED W ORKS
Similar to supervised learning, recent work has shown that DRL is also susceptible to adversarial attacks (Ilahi et al., 2022;
Chen et al., 2019; Behzadan & Munir, 2017; Oikarinen et al., 2021).This area has drawn significant attention of late, and the following provides a brief overview; we refer readers desiring more detail to (Ilahi et al., 2022).
Broadly speaking, there are two basic attack types depending on the assumed adversary: (i) White-box attacks (Goodfellow et al., 2015; Carlini &
Wagner, 2017; Madry et al., 2018; Schwinn et al., 2021), where the adversary has perfect knowledge of the target model, which is the victim policy in the RL context, and (ii) Black-box attacks (Andriushchenko et al., 2020; Pomponi et al., 2022), where the adversary does not know the model nor any of its attributes.
Similar to previous works (Huang et al., 2017; Behzadan & Munir, 2017;
Pattanaik et al., 2018; Zhang et al., 2020; Sun et al., 2022), we assume that the attacker does not have the ability to change the environment directly but perturbs the state observations returned by the environment before the agent observes them.
Existing defense methods can be categorized into: (i)
Adversarial training (Oikarinen et al., 2021;
Behzadan & Munir, 2018; Han et al., 2018), where the RL agent is exposed to the adversarial environment during training (ii) Robust learning (Mandlekar et al., 2017; Smirnova et al., 2019; Pan et al., 2019), which is a training mechanism to ensure robustness against training-time adversarial attacks.
A common approach is to add noise to the parameter state while training; (iii) Adversarial detection (Gallego et al., 2019a; Havens et al., 2018; Gallego et al., 2019a) trains a separate model to detect adversarial input and contaminated input (e.g., image frames) are replaced with predicted/generated versions; (iv) Policy distillation (Czarnecki et al., 2019) focuses on transferring knowledge from one or multiple policies to the target policy in a student-teacher framework.
KPR is different from the above strategies.
Unlike adversarial training and variants of adversarial detection (Gallego et al., 2019b; Lin et al., 2017), we do not assume knowledge of the attack.
In contrast to robust learning, KPR does inject noise during training.
Although both policy distillation and KPR can fuse multiple policies, the methodology and application are different.
Policy distillation does not leverage the relation between input policies and thus does not explicitly encourage structural consistency, which limits its defense performance as found in (Carlini & Wagner, 2016; Soll et al., 2019) and in our experiments (Section 4).
KPR can be seen as a policy ensemble (Wiering & van
Hasselt, 2008), albeit one that leverages prior knowledge in its construction.
Prior works have suggested that ensemble methods offer some protection against adversarial attacks in supervised settings (Wiering & van Hasselt, 2008).
However, to our knowledge, policy ensembles have yet to be used as a defense strategy in RL.
K NOWLEDGE - BASED P OLICY R ECYCLING (KPR)
As in standard RL, we consider a discounted discrete-time Markov Decision Process (MDP) ( S , A , R, T , γ, d 0 ) , where S is a set of states, A is a set of discrete action, R : S ×
A → R is the reward function, and T : S ×
A → P ( s ) is the transition function, γ ∈ [0 , 1] is the discount factor, and d 0 ∈ P ( s ) is the distribution over the initial state.
At time step t , the agent is in the state s t ∼ T ( s t , a t ) and receives a reward R ( s t , a t ) .
The agent’s objective is to learn a policy π : S → P ( A ) that maximizes the expected cumulative rewards,
[Equation]
We consider a setting where the policy is trained in a benign environment and then deployed to a test environment that may be adversarial.
Our goal is to retain expected cumulative rewards in the presence of a white box attacker that aims to alter our agent’s actions by perturbing observations s ′ t =
s t + δ t , s.t. δ ∈ ∆ , where ∆ is a perturbation set, e.g., an ℓ 2 ball around s with ϵ radius, i.e. ℓ 2 ( δ t ) ≤ ϵ, ∀ t ≥ 0 .
We consider a setting where we do not have access to the attacker’s perturbation set nor how it is optimizing.
In this information impoverished setting, our defensive options are relatively limited.
One approach is to sample possible attackers (with various perturbation abilities) and train a policy to be robust against these attacks.
However, this approach is computationally expensive.
We posit the susceptibility of a policy to an attacker is due principally to overfitting on a specific task.
To alleviate this issue, we propose to leverage prior knowledge — comprising auxiliary tasks and relations between them — that can enable better robustness to noisy or perturbed observations.
Let us define our main task as T M (which is modeled by an MDP as above) and an associated main task policy π T M that maximizes returns.
We assume we possess a main task policy trained in the benign environment.
We also have access to n auxiliary task policies Π aux = { π T 1 , π T 2 , . . .
, π T n } for tasks Υ aux
= { T 1 , T 2 , . . .
, T n } .
Each task is modeled by an MDP tuple with the same elements as T M but with a different reward function.
We define the relation set I =
{K , f ( · ) } comprising explicitly specified relations between the task policies
K (e.g., from a domain expert) and learnt knowledge that will be represented implicitly by the function f ( · ) .
While prior knowledge usually enters into training as a regularization term in the loss, we incorporate this knowledge directly into the policy.
We seek to obtain an augmented policy (cid:98) π T M that maximizes returns and is conditioned upon Π aux and I .
The challenge is to ensure that this augmented policy has sufficient capacity to perform well on the task yet be robust to attacks.
Our approach is an ensemble method where we use I and Π aux to construct a new set of policies (cid:98) Π T M for the main task T M (Fig. 1).
We then combine these policies using a simple parameter-less fusion mechanism,
[Equation]
In the following subsections, we provide details on the above: we will first discuss how prior domain knowledge regarding existing tasks can be specified.
Next, we detail a Graph Neural Network (GNN) pooling network to process knowledge and state information to learn relations.
Finally, we discuss the simple voting mechanism to derive the final policy action distribution.
3. S PECIFYING R ELATIONS VIA L OGICAL G RAPHS
In our setting, domain knowledge comprises (i) the set of auxiliary tasks policies given, Π aux , which could be any type of policy (including non-differentiable or rule-based policies), and (ii) the logical relations among them.
We focus on propositional logic, which is well-defined and unambiguous compared to natural language, yet relatively easy for humans to derive and interpret.
A proposition p is a statement which is either True or False .
A formula F is a compound of propositions connected by logical connectives, e.g., ¬ , ∧∨ , = ⇒ .
A logical formula (and corresponding truth assignments) can be represented as a undirected graph G =
( V , E ) with nodes v i ∈ V , and edges ( v i , v j ) ∈ E .
Individual nodes are either propositions (leaf nodes) or logical operators ( ¬ , ∧ , ∨ ) , where subjects and objects are connected to their respective operators.
In our work, leaf nodes are auxiliary tasks, which are True if the task is successful and False otherwise, where success of the task could be defined as whether the accumulated return attains a certain threshold.
For example, the logical relation formula T M = ⇒ T 1 ∧ T 2 expresses that “if the the main task is successful, then both the auxiliary tasks T 1 and T 2 are successful”.
The features of the leaf nodes are the predicted action distributions conditioned on the state s t ; the features of the logical operators are fixed to randomly generated vectors.
To incorporate state information, we add a state feature node to every logical relation graph and connect it to every other node.
As state features, we use the latent vector z s obtained from a simple VAE (Kingma & Welling, 2014) trained to reconstruct the state s .
The set of logical relations forms the set K =
{ k i (Υ aux ) } mi =1 defined over the auxiliary tasks Υ aux where each k i is represented as a graph G k i .
Note that the logical relations need not to be complete nor error-free; KPR can tolerate a degree of misspecification as relations are also learnt from interaction data.
3. L EARNING R ELATIONS FROM I NTERACTION D ATA
[Table] Function.
To utilize the structural information contained in the logical relation graphs and enable learning from interaction data, we adopt Graph Neural Networks (GNNs) (Fey & Lenssen, 2019).
GNNs pass and aggregate the messages from the neighbors to encode the nodes in the graph with learnable weights.
To obtain a single action distribution from each logical relation graph, we add a graph pooling layer that maps the entire graph into a compact representation.
We adopt Graph Multiset Pooling (Baek et al., 2021), which satisfies permutation invariance, as our GNN pooling function f ( · ) .
The accumulated knowledge I consists of both the logical relations graphs and learnt weights of the GNN pooling function f ( · ) .
Graph Multiset Pooling network f ( · ) , consists of a two-layer message-passing module g ( · ) , a graph multi-head attention pooling module that condenses all nodes to representative nodes h l ( · ) , a multihead self-attention module q s ( · ) , and the second multi-head attention pooling module that condenses the entire graph to one vector
h 1 ( · ) .
Put together,
[Equation]
where G k i is the input logical graph constructed from the logical relation formula k i .
The message-passing function updates node representations by aggregating its neighbors’ messages.
In particular, the message-passing function we use is:
[Equation]
where x l v is the node representation of node v at l th step.
N ( v ) denotes a set of neighboring nodes of v , W is the weight matrix, and D ( v ) , D ( u ) denotes the degree of node v, u respectively.
An attention function q can be described as mapping a query and a set of key-value pairs to an output, where the query Q , keys K , values V , and output are all vectors.
The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.
[Equation]
The multi-head attention function q multi simply uses a multiple attention function with the linearly projected query, keys, and values matrices.
To explicitly use the graph structure in multi-head attention, the keys and values are generated using one layer of message-passing function instead of conventional linear projection.
[Equation]
where I ( · ) denotes the residual connection function,
q multi ( · ) is the multi-head attention function, Z is a parameterized seed matrix that is optimized end-to-end acting as the query matrix, and g K , g V represents the message-passing functions for key and value respectively.
Model Training.
Each f
( G k i ) can be interpreted as a policy in an ensemble; f takes a logical formula graph as input, and outputs an action distribution, i.e.,
p k i ( a | s ) =
f ( G k i ) .
The node feature for the auxiliary task node is the respective predicted action distribution and the auxiliary task identifier.
The node features for logical operators are fixed randomly generated vectors that share the same dimension with auxiliary task features.
To learn the relationships between tasks, we train each ensemble policy to predict the action with the highest probability under the main task policy.
More precisely, we train the graph neural network pooling function f with the following loss:
[Equation]
where τ = { ( s t , a t , R ( s t , a t ) } is a trajectory sampled using the main task policy π T M and z s t are the state features.
Note that KPR does not need to access the perturbed environment during training.
3. P OLICY VIA F USION OF A CTION D ISTRIBUTIONS
At this stage, we have m action distributions
p k
i ( a | s ) for each logical relation k i .
To obtain a final action distribution, we will need to combine or fuse these action distributions together.
We adopt a very simple voting-like mechanism that requires no training .
The fusion is performed via two main steps: (i) task policy filtering and (ii) action counting.
We first select the top-3 action distributions by selecting the most “confident” models as measured by negative entropy.
Intuitively, this filters away irrelevant action distributions given the current state.
Next, we form a new action distribution by vote counting.
Each of the three remaining policies casts a positive vote for their top scoring action (with largest p ( a j | s ) ) and a negative vote for their lowest scoring action (smallest p ( a j | s ) ).
For each action a j , we tally the number of positive votes o + j and negative votes o − j , and construct a new action distribution for the main task T M :
[Equation]
E XPERIMENTS
In this section, we focus on validating that incorporating knowledge offers protection against adversarial attacks in deep RL.
Specifically, we conduct experiments on a selection of Atari games and a high-dimensional Robot Food Court Environment (RoFoCo) 1 .
Compared Methods.
We compared our method with policy ensemble (Wiering & van Hasselt, 2008), policy distillation (Rusu et al., 2016), and adversarial training (Goodfellow et al., 2015) as strong baselines 2 .
For Policy Ensemble, we use five main task policies (trained with different random seeds) with majority voting.
Since we focus on attack-agnostic defense methods, we conduct adversarial training over a union of commonly-known attacks, including the attacks that we evaluate on; this method approximates an “attack agnostic” adversarial training.
Recall that the domain knowledge I comprises the relations among the auxiliary tasks and the main task.
To investigate the role I plays, we included a variant of our method, MLP Fusion, which replaces the GNN with a MLP whose input are the action distributions of the auxiliary policies.
Attacks.
In Atari games, we used four common white-box attacks, FGSM (Goodfellow et al., 2015;
Huang et al., 2017), PGD (Madry et al., 2018) and Jitter (Schwinn et al., 2021); and a black-box attack, Square (Andriushchenko et al., 2020).
In the Robot Food Court environment (RoFoCo), we selected FGSM and PGD as white-box attacks (since they caused the most significant performance degradation in the Atari games experiments) and Square as the black-box attack.
The attacks were implemented using the torchattacks package (Kim, 2020).
4. A TARI G AMES
Environment.
We evaluated KPR on three Atari games: Road Runner, River Raid, and Space Invader, which are representative tasks that can be naturally decomposed into auxiliary tasks, e.g., collect targets, shoot enemies, and avoid collisions.
For each game, the state is a stack of four consecutive frames, where each frame is pre-processed to size 84 × 84 .
Detailed environment settings can be found in Appendix B.2.
Main Task Policy.
We adopt PPO (Schulman et al., 2017; Huang et al., 2022) as the learning algorithm.
Each policy is trained with 10 million frames.
Alternative algorithms and policies such as
DQN (Mnih et al., 2015) and Rainbow DQN (Hessel et al., 2018) can be used without changing the overall proposed framework.
Auxiliary Tasks and Domain Knowledge.
Due to space restrictions, complete auxiliary tasks information and domain knowledge relations are in Appendix B.2.
Taking Road Runner as an example, the auxiliary tasks are: “T 1 : Collect the bird seeds on the road.”
and “T 2 : Avoid the cars.”.
Denote the main task, “Collect the bird seeds while avoiding colliding with cars on the road”, as π T M .
The logical relation is T M =
⇒ T {+
 ∧+} T 2 .
[Table]
Results and Discussion.
Our experimental results are summarized in Table 1, which shows test-time episode returns averaged over 70 episodes with standard errors in brackets.
In each game, the main policy solved the task but performance degraded when attacked, as expected.
In general, KPR improves robustness to all five attacks across the three environments.
There is some performance degradation but it is less severe compared to the unprotected main task policy.
This is most apparent for the strongest attacks, i.e., FGSM and PGD.
Interestingly, we see that adversarial training led to poorer policies in the benign environment and was also generally ineffective against FGSM and PGD.
We believe this is due to the attack-agnostic training scheme; in typical use, adversarial training requires knowledge of the attacker but in our experiments, different attackers were sampled and this led to significant noise that hampered training.
Policy Ensemble, MLP Fusion, and KPR are all ensemble methods, but differ in their construction.
Although not specifically designed for defense, the simple policy ensemble is surprisingly effective in the Road Runner domain.
Policy Ensemble and MLP fusion achieve comparable performance, but are poorer compared to KPR.
These results suggest that using existing prior knowledge in the form of policies and logical relations do result in more robust policies.
[Figure] [Figure]
4. R OBOT F OOD C OURT E NVIRONMENT (R O F O C O )
Environment.
This environment simulates a service robot in a food-court setting and was developed using Unity (Juliani et al., 2018).
The agent is a food serving robot.
A food stall number and a table number are given at the beginning of every episode.
The main task is to collect food from the instructed food stall and deliver it to the instructed table number.
After the customer finishes eating, the robot should pick up the used tray and deposit it at the tray collection point.
The task is divided into four stages as illustrated in Figure 2.
The agent receives a +10 reward for completing each intermediate stage and an additional +20 reward for completing the whole task.
The agent will get a − 0 .
1 penalty at every time step and an additional − 5 penalty if it tries to incorrectly perform pick up or put down actions on objects, such as trying to pick up food while the customer is eating.
The maximum number of steps is 1,000.
The available actions are: 1) Move forward; 2) Move backward; 3) Turn left 90 degrees; 4) Turn Right 90 degrees; 5) Pick up; 6) Put down; 7) Do nothing.
The observation at each time step is a 128 × 128 RGB image.
Main Task Policy.
As the observation space is significantly larger than the Atari games, exploring from scratch is computational resource consuming.
To obtain the main task policy, we first initialize the agent with the policy obtained using imitation learning (Hussein et al., 2017) on a set of expert demonstrations.
DQN (Mnih et al., 2015) is used to refine the policy via interactions further.
Auxiliary Tasks and Domain Knowledge.
Due to space constraints, we give a few examples of auxiliary tasks and domain knowledge relations here and provide the complete list in Appendix
B.3.
Three example auxiliary tasks are 1) T 1 : Navigate to the food stall; 2) T 2 : Pick up food from customer tables; and 3) T 3 : Pick up food from the food stall.
Denote the main task we described as
T M .
One example relation is T M =
⇒ T 1 ∧ ( ¬ T 2 ) ∧ T 3 .
Results and Discussion.
Test-time performance comparison is summarized in Table 2, which shows episode returns averaged over 50 test episodes.
KPR outperforms other baselines across attacks, which further supports the notion that prior policy knowledge encourages robustness in policies.
As before, KPR policies experienced less severe degradation against the various attacks (which are unknown at training-time).
As in the Atari games, we observed that the ensemble methods are able to provide comparably effective protection compared to policy distillation and adversarial training.
[Table]
✓ ✗
[Table]
=</latexit>
[Figure]
[Table]
As a qualitative comparison
[Table]
Figure 3 shows an example where the adversary successfully misled the robot into picking up food from other customers’ tables.
By leveraging auxiliary policies, such as navigating to the food stall and picking the food from the food stall, together with the relations between the tasks, KPR is able to correct the final action distribution prediction.
C ONCLUSION AND F UTURE W ORK
This paper proposes KPR, a novel approach to leverage domain knowledge to defend against adversarial attacks in reinforcement learning settings.
KPR incorporates domain knowledge from auxiliary policies and specified logical relations between tasks, then learns flexible relations from interaction data via graph neural networks.
The main advantage of KPR is that it is both policy and attack agnostic; any type of policy could be utilized, and no access nor information about the attack is required.
We demonstrated its efficacy empirically in both Atari games and the complex Robot Food
Court environment (RoFoCo).
A number of promising avenues exist for future research.
In this work, we mainly experimented with neural network policies and future work can look into other auxiliary and main policies (e.g., interpretable rule-based policies).
Next, KPR worked well in RoFoCo, which is a highly-complex environment, but it is necessary to test KPR (and other existing defense methods) in real-world environments.
Finally, we believe that KPR can also defend against alternative threat models, including observed adversaries that comply with environmental constraints (Gleave et al., 2020; Cao et al., 2022); these experiments would make for interesting next steps.
E THICS S TATEMENT
We propose a novel framework to defend against adversarial attacks in RL setting by leveraging prior knowledge.
This paper does not raise any ethical concerns.
Our study does not involve human subjects.
The Robot Foodcourt simulation environment we developed does not contain any sensitive information.
R EPRODUCIBILITY S TATEMENT
To ensure the reproducibility of our experimental results, we include detailed network architecture and hyper-parameters in the appendix and provide source code in the supplementary material
A T OY E XAMPLE : G OAL F INDING
To better illustrate how KPR works, we demonstrate it on a goal-finding grid-world environment as a proof of concept evaluation.
Environment.
The agent’s objective is to find a target goal while avoiding the obstacle in a 7 ×grid-world (Figure 4A.).
The agent receives a reward of +10 for reaching the target, a − 10 penalty for colliding with the obstacle, and a − 1 penalty at each time step.
The maximum episode length is 30 .
To mimic the redundant information in real-world scenarios, each object has a specific color, shape, and letter.
Auxiliary Tasks and Domain Knowledge.
We design auxiliary tasks that focus on the different aspects of the objects, such as “T 1 : Find the orange color (target object color)”, “T 2 : Avoid the diamond shape (obstacle object shape)”.
We denote the main task as T M .
A simple domain knowledge relation among them could be
{+T M = ⇒ T 
+}
∧ T 2 .
A complete list of auxiliary tasks and domain knowledge relations among them can be found in Appendix B.1.
Compared Methods.
We compared with policy ensemble (Wiering & van Hasselt, 2008) and MLP
Fusion which is a variant of KPR.
Instead of leveraging GNN based pooling function to incorporate logical relations among auxiliary tasks, MLP Fusion uses MLP to replace the GNN pooling function.
We aim to investigate the role of the logical relation domain knowledge component by comparing it with this variant.
We use simple Deep Q-Learning (DQN) (Mnih et al., 2015) to train our main task and auxiliary tasks policies.
Alternative algorithms and policies can be used without changing the overall proposed framework, e.g., improved DQN variants (Hasselt et al., 2016; Hessel et al., 2018) and alternative algorithms (Schaul et al., 2015; Bellemare et al., 2017; Christodoulou, 2019). Result and Discussion. We adopt FGSM attack (Goodfellow et al., 2015) to perturb the observation.
Since the observations are grid world states, the attacks here are generally stronger than the ones on pixel perturbations.
The performance is evaluated in terms of the episode return averaged over 1000 episodes.
The results are summarized in Figure 4B. C.. While the performance is similar in the Benign Environment, KPR outperforms others by a large margin in the perturbed environment. Note that KPR is not trained in the perturbed environment.
This demonstrated KPR’s ability to correct the inconsistency caused by adversarial attacks by leveraging domain knowledge over auxiliary policies. Furthermore, to investigate the qualitative correlation between the amount of domain knowledge and episode return, we study how different knowledge levels affect the test performance, as shown in Figure 4 D. E.. The “Low” knowledge level consists of 4 auxiliary policies and 2 formulae.
The “Moderate” knowledge level consists of 11 auxiliary tasks and 5 formulae.
And the “High” knowledge level consists of 17 auxiliary tasks and 12 formulae.
As shown in Figure 4 D. and E., as the knowledge level increases, the improved robustness to adversarial attacks also increases.
A. A UXILIARY T ASKS AND D OMAIN K NOWLEDGE
The objective for the agent is to find the goal while avoiding the obstacle in a 7 × 7 grid world (Figure 4A.).
The agent gets a +10 reward for reaching the target, a − 10 penalty for touching the obstacle, and a − 1 penalty every time step.
The maximum step number is 30 .
To mimic the redundant information in real-world scenarios, we use three features to represent a single object, color, shape, and the letter on it.
The auxiliary tasks are:
Find the orange star “T” object.
Find the orange object.
Find the star object.
Find the “T” object.
Avoid the blue diamond “O” object.
Avoid the blue object.
Avoid the diamond object.
Avoid the “O” object.
Avoid the orange star “T” object (negation task).
[Table]
A. B.
[Table]
(cid:46)(cid:81)(cid:82)(cid:90)(cid:79)(cid:72)(cid:71)(cid:74)(cid:72)(cid:3)(cid:47)(cid:72)(cid:89)(cid:72)(cid:79) Low
High Moderate
C.
D.
E.
E p
i s o d e R e t u r n
E p
i s o d e R e t u r n
Find the blue diamond “O” object (negation task).
Find the orange object and avoid the blue diamond “O” object.
Find the star object and avoid the blue diamond “O” object.
Find the “T” object and avoid the blue diamond “O” object.
Find the orange star “T” object and avoid the blue object.
Find the orange star “T” object and avoid the diamond object.
Find the orange star “T” object and avoid the “O” object.
Avoid the orange star “T” object and find the blue diamond “O” object.
There are many possible relations among them.
The logical relations we adopted are:
T M = ⇒ T 1 ∧ T2.
T M = ⇒ T 1 ∧ T3.
T M = ⇒ T 1 ∧ T4.
T M = ⇒ T 1 ∧ T5.
T M = ⇒ T 2 ∧ T6.
T M = ⇒ T 3 ∧ T7.
T M = ⇒ T 4 ∧ T8.
T M = ⇒ ¬ T9.
T M = ⇒ ( ¬ T 9 )
∧ ( ¬ T 10 ) 10.
T M = ⇒ ( T 2 ∨ T 3 ∨ T 4 )
∧ ( T 6 ∨ T 7 ∨ T 8 ) 11.
T M = ⇒ T 11 ∨ T 12 ∨ T12.
T M = ⇒ T 14 ∨ T 15 ∨ T
B A TARI G AMES E NVIRONMENT B. A UXILIARY T ASKS AND D OMAIN K NOWLEDGE B.1. R OAD R UNNER
The Road Runner runs endlessly to the left.
The player must pick up bird seeds on the street, avoid cars (Wikipedia, 2022b).
The auxiliary tasks are:
Collect bird seeds.
Avoid cars.
There are many possible relations among them.
The logical relations we adopted are:
{+T M = ⇒ T 
+}
∧ T
where T M denotes the main task.
B.1.
R IVER R AID
River Raid is a vertically scrolling shooter game.
The player flies a fighter get which can only move left and right.
It cannot maneuver up and down the screen, but it can accelerate and decelerate.
The player’s jet crashes if it collides with the riverbank or an enemy craft, or if the jet runs out of fuel.
The player scores points for shooting enemy tankers (30 points), helicopters (60 points), fuel depots (80 points), jets (100 points), and bridges (500 points).
The jet refuels when it flies over a fuel depot.
A bridge marks the end of a game level (Wikipedia, 2022a).
The auxiliary tasks are:
Shoot enemy tankers.
Shoot enemy helicopters.
Get fuel depots.
Shoot enemy jets.
Shoot enemy bridges.
Avoid collides with
riverbank or enemy craft.
There are many possible relations among them.
We sampled a subset:
{+T M = ⇒ T 
+}
∧ T 2 ∧ T 3 ∧ T 4 ∧ T 5 ∧ T2.
T M = ⇒ T 1 ∧ T3.
T M = ⇒ T 2 ∧ T4.
T M = ⇒ T 5 ∧ T5.
T M = ⇒ T 1 ∧ T6.
T M = ⇒ T 1 ∧ T7.
T M = ⇒ T 1 ∧ T8.
T M = ⇒ T 2 ∧ T9.
T M = ⇒ T 2 ∧ T10.
T M = ⇒ T 3 ∧ T11.
T [Equation] B S PACE I NVADERS
Space Invaders is a fixed shooter in which the player moves a laser cannon horizontally across the bottom of the screen and fires at aliens overhead.
The aliens begin as five rows of eleven that move left and right as a group, shifting downward each time they reach a screen edge.
The goal is to eliminate all of the aliens by shooting them.
While the player has three lives, the game ends immediately if the invaders reach the bottom of the screen.
The aliens attempt to destroy the player’s cannon by firing projectiles.
The laser cannon is partially protected by stationary defense bunkers which are gradually destroyed from the top by the aliens and, if the player fires when beneath one, the bottom (Wikipedia, 2022c).
The auxiliary tasks are:
Shoot the first row of aliens.
Shoot the second row of aliens.
Shoot the third row of aliens.
Shoot the forth row of aliens.
Shoot the fifth row of aliens.
Avoid being shot by aliens.
There are many possible relations among them.
We sampled a subset:
{+T M = ⇒ T 
+}
∧ T 2 ∧ T 3 ∧ T 4 ∧ T 5 ∧ T2.
T M = ⇒ T 1 ∧ T3.
T M = ⇒ T 2 ∧ T4.
T M = ⇒ T 5 ∧ T5.
T M = ⇒ T 1 ∧ T6.
T M = ⇒ T 1 ∧ T7.
T M = ⇒ T 1 ∧ T8.
T M = ⇒ T 2 ∧ T9.
T M = ⇒ T 2 ∧ T10.
T M = ⇒ T 3 ∧ T11.
T M = ⇒ T 3 ∧ T12.
{+T M = ⇒ T 
+}
∧ T
where T M denotes the main task.
C R OBOT F OOD C OURT (R O F O C O ) E NVIRONMENT
In order to evaluate the performance of KPR in a complex real-world like environment, we developed a high fidelity simulated Robot Food Court environment (RoFoCo) via Unity (Juliani et al., 2018), as shown in Figure 2.
The agent is a food serving robot.
A food stall number and a table number are instructed at the beginning of every trajectory.
The main task is to collect food from the correct food stall and deliver it to the correct table number, then after the customer finish eating, pick up the used tray and send it to the tray collection point.
The agent gets a +10 reward for completing each intermediate stage, and an additional +20 reward for completing the whole task.
The agent will get a − 0 .
1 penalty every time step and an additional − 5 penalty if it tries to perform pick up or put down actions on objects that are not affordable, such as trying to pick up something on humans. The maximum number of steps is 1000 . The available actions are: 1) Move forward; 2) Move backwards; 3) Turn left 90 degrees; 4) Turn Right 90 degrees; 5) Pick up; 6) Put down; 7) Do nothing. The observation is 128 × 128 RGB image.
C. A UXILIARY T ASKS AND D OMAIN K NOWLEDGE
The auxiliary tasks are:
Navigate to food stall number x . 2.
Navigate to customer table number x . 3.
Navigate to tray collection point. Pick up food from food stall number x . 5. Put down food on customer table number x . 6. Pick up used tray from customer table number x . 7.
Put down used tray to tray collection point. Pick up food from customer table number x (negation task). Put down used tray on customer table number x (negation task). Put down food on food stall number x (negation task).
Put down food on tray collection point (negation task). Put down used tray on food stall number x (negation task). There are many possible relations among them. The logical relations we adopted are:
T M = ⇒ T 1 ∧ T2.
T M = ⇒ T 2 ∧ T3.
T M = ⇒ T 2 ∧ T4.
T M = ⇒ T 3 ∧ T5.
{+T M = ⇒ T 
+}
∧ T 4 ∧ ( ¬ T 10 ) ∧ ( ¬ T 12 ) 6.
{+T M = ⇒ T 
 ∧ T 5 ∧ ( ¬ T 9 ) 7. T M = ⇒ T 
 ∧ T 6 ∧ ( ¬ T 8 ) 8.+}
{+T M = ⇒ T 
+}
∧ T 7 ∧ ( ¬ T 11 ) 9.
{+T M = ⇒ T 
+}
∧ T 4 ∧ ( ¬ T 8 ) 10.
{+T M = ⇒ T 
+}
{+∧ T 5 ∧ ( ¬ T 10 ) 11. T M = ⇒ T 
 ∧ T 7 ∧ ( ¬ T 9 ) 12. T M = ⇒+}
¬ T13.
T M = ⇒ ¬ T14.
T M = ⇒
¬ T15.
T M = ⇒
¬ T16.
T M = ⇒ ¬
T
where T M denotes the main task D A DVERSARIAL A TTACKS We evaluated against 3 white-box attacks, FGSM, PGD, Jitter, and 1 black-box attack, Square attack. We’ll briefly introduce their intuitions and mechanisms, as well as the hyper-parameters we use.
To ensure fair comparison, we utilize the same ϵ value, which controls the maximum perturbation value, for all 4 adversarial attacks.
According to different task characteristics, we use 0.002 for Road
Runner, 0.01 for River Raid, 0.002 for Space Invaders and 0.002 for Robot Foodcourt environment. Pixel values are scaled to [0 , 1] . D. F AST G RADIENT S IGN M ETHOD (FGSM) FGSM (Goodfellow et al., 2015) is a method to efficiently calculate the gradient of the cost function with respect to the input of the neural network.
The adversarial examples are generated using the following equation:
[Equation]
where θ denotes the parameters of a model, x is the input to the model, y is the target associated with x , J ( θ, x, y ) is the cost used to train the neural network, and sign is the component-wise signum operator.
The adversarial examples generated by FGSM exploit the “linearity” of deep network models in the higher dimensional spaces whereas such models were commonly thought to be highly non-linear at that time.
They hypothesized that the designs of deep neural networks that encourage linear behavior for computational gains also make them susceptible to cheap analytical perturbations, which is often referred as “linearity hypothesis”.
D. P ROJECTED G RADIENT D ESCENT (PGD)
PGD (Madry et al., 2018) improve the performance of FGSM by running a finer iterative optimizer for multiple iterations.
PGD performs FGSM with a smaller step size and projects the updated adversarial sample into the ϵ − L ∞ neighbor of the benign samples and a valid range.
Hence the adversarial perturbation size is smaller than ϵ .
The update procedure follows:
[Equation]
In order to make adversarial attacks more effective, Jitter (Schwinn et al., 2021) proposes a novel loss function to encourage logits scale invariance, diverse attack targets, and perturbation norm minimization. The final loss function can be described as follows: [Equation]
x ′ is misclassified
[Equation]
where y is the ground truth, z is the output logits after perturbation, and α controls the lowest and largest possible output values of the softmax function.
D. S QUARE A TTACK
Square attack (Andriushchenko et al., 2020) is based on a randomized search scheme which selects localized square-shaped updates at random positions so that at each iteration the perturbation is situated approximately at the boundary of the feasible set.
The objective is to solve the constrained optimization problem:
[Equation]
where f is the target network, x is the input, y is the ground truth, and S is the domain of the input.
E M ODEL A RCHITECTURE
Policy Network.
We adopt PPO (Schulman et al., 2017; Huang et al., 2022) as the learning algorithm.
We use a three-layer CNN with { 32 , 64 , 64 } hidden size, followed by a two linear layers withneurons for both actor and critic networks.
VAE.
The encoder is five-layer CNN with hidden dimension { 32 , 64 , 128 , 256 , 512 } , followed by a linear layer.
The decoder is symmetric to the encoder with an additional 2D transposed convolution layer (Zeiler et al., 2010) of hidden dimension 512 .
GNN Fusion Network.
Our GNN pooling network consists of 3 GCN convolution layers with hidden dimensions { 64 , 32 , 64 } , followed by the graph mutliset transformer as described in Section 3, andlinear layers of 64 hidden neurons.
MLP Fusion Network
The MLP Fusion Network consists of 2 linear layers of 64 hidden neurons.