Extended Abstract Track
Representing Repeated Structure in Reinforcement Learning Using Symmetric Motifs
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
Transition structures in reinforcement learning can contain repeated motifs and redundancies.
In this preliminary work, we suggest using the geometric decomposition of the adjacency matrix to form a mapping into an abstract state space.
Using the successor representation (SR) framework, we decouple symmetries in the transition structure from the reward structure, and form a natural structural hierarchy by using separate SRs for the global and local structures of a given task.
We demonstrate our method can achieve high policy evaluation accuracy while using representations that are significantly compressed.
1. Introduction
Knowledge of a task’s structure can enable more efficient policy learning in reinforcement learning (RL) Ravindran and Barto (2001).
For example, policies can be constructed that are temporally hierarchical Sutton et al. (1999) or an environment’s state space can be reduced by recognising its symmetries van der Pol et al. (2020b).
Unfortunately, discovering an environment’s latent structure is difficult.
Existing methods usually find reduced state-action spaces by learning a global mapping between the environment’s Markov decision process (MDP) and a more abstract MDP van der Pol et al. (2020a); Mavor-Parker et al.
In this paper we use a different strategy—we use results from graph theory and network analysis MacArthur et al. (2008); S´anchez-Garc´ıa (2020) to isolate repeated structural motifs in an MDP, allowing for the compression of its transition function into a more compact form.
Additionally, our approach is formulated using the Successor Representation (SR) Dayan (1993), meaning structural regularity can be decoupled from the reward function.
Empirically, we demonstrate effective compression of SRs formed over small world graphs Barab´asi and Albert (1999); Watts and Strogatz (1998), opening up future avenues of work using hierarchical reinforcement learning approaches within the SR. Extended Abstract Track
2. Related Work
To operate in large state spaces, knowledge of the structural regularities of an environment is essential.
Traditionally structural knowledge is obtained implicitly by learning an intermediate representation useful for value judgements—or explicitly by learning a compressed latent model of the environment Hafner et al.
Other approaches encode inductive biases into policies about structural regularities Ravindran and Barto (2001).
For example, van der Pol et al. (2020b) build knowledge of global symmetries into policy networks, while van der Pol et al. (2020a) learn the equivariances inherent in the dynamics of a given MDP.
The SR is a biologically plausible middle ground between model-based and model-free reinforcement learning that contains information about an environment’s structure—decoupled into its reward function and expected transition dynamics Dayan (1993); Stachenfeld et al.
However, as far as we are aware, we are first to build an abstraction on top of the SR to compress it into a more compact form, which we describe next.
3. Geometric Decomposition
For a given MDP, the transition function T ( s, a, s ′ ) implies an adjacency graph G =
( S , E ) where S is the set of states in the MDP, and E is the set of edges: pairs of states ( s, s ′ )
that are connected under some action.
Only deterministic MDPs are considered.
Structural regularities in this graph can be described using the notion of graph automorphisms .
An automorphism is a permutation of nodes that preserves the adjacency structure of said nodes.
The set of nodes being permuted is known as the orbit.
Within the overall adjacency graph of the MDP, there may exist smaller subgraphs that are symmetric and therefore can be described by some automorphism that permutes their nodes within the bounds of the local symmetry.
These automorphisms are often repeated across the structure of a graph— they are hence referred to as motifs (MacArthur et al., 2008).
From a given automorphism group, a graph can be represented as the union of the vertices not belonging to any symmetric subgraph, S 0 , and the the vertices of each motif i.e. S = S 0 ∪ S M 1 ∪ S M 2 . . .
∪ S M m where S M i is the subgraph of vertices that are symmetric under a specific automorphism.
This decomposition is the geometric decomposition of a graph (MacArthur et al., 2008).
The automorphism group for a given graph is not unique; we follow (S´anchez-Garc´ıa, 2020) and generate subgraphs that are non-overlapping, and therefore automorphisms that are support-disjoint.
We use saucy (Darga et al., 2012) to generate our automorphism groups, and in-line with (S´anchez-Garc´ıa, 2020) we restrict the motifs found to two orbits in size.
4. Structural Hierarchies with Local Symmetry and Successor Representations
Once an automorphism group is generated and the geometric decomposition of the graph is found, we generate a reduced graph with adjacency matrix ˆ A by representing every occurrence of a symmetric subgraph with a single node, whilst also keeping an index of which node represents which subgraph and the local transition structure of each symmetric subgraph A i . An illustrative diagram of this can be found in 1.
For the purposes of this work, we will only consider the task of policy evaluation under a random walk policy.
If one were to sample trajectories from this graph following a policy, or compute a policy through sample-based policy improvement, the graph-subgraph structure would need to be represented hierarchically using the options framework (Sutton et al., 1999), where nodes representing a symmetric subgraph would also lie in the initiating set of an option that would allow the agent to execute a policy specified for the subgraph.
A random walk SR is generated for the reduced graph ˆ A , and for each symmetric subgraph A i .
Let S A
i represent the state space of subgraph A i .
We generate expected random walk state transition dynamics using the analytic form of the SR: ψ π =
( I − γ A ) − 1 , where π is the policy, ψ is the SR and γ is the discount factor.
A policy over the MDP is defined for m subgraphs as π ( a | s )
=
argmax a ( ψ π
( R ( s ))— ψ is either the SR of the reduced graph, or a subgraph depending on the state being evaluated.
a represents actions and s represents graph states.
When evaluating states in S ˆ A , R ( s ) for states representing compressed subgraphs is the mean of the rewards contained in the subgraph.
There is an inherent bias in estimating the value function in this way, as the state occupancies obtained for S
ˆ A will not account for occupancies accumulated inside the subgraphs.
Future work will study deriving a bound for this value approximation error, but empirically this is not a significant malus.
4.1. Results
To test this approach, 100 Barab´asi-Albert (Barab´asi and Albert, 1999) and 100 Connected Watts-Strogatz (Watts and Strogatz, 1998) graphs were generated, each with 100 nodes.
Extended Abstract Track
The Barab´asi-Albert graphs were generated with m = 1 edges preferentially reattached, and the Watts-Strogatz graphs were generated with nodes initially attached to k = 2 neighbours and rerouted with p = 0 .
The Watts-Strogatz graphs were more densely connected than the Barab´asi-Albert graphs.
We used networkx (Hagberg et al., 2008) to generate these graphs.
To convert these to MDPs, an action space was chosen arbitrarily with size equal to the degree of the most connected node in a graph.
For each graph, the value function under a random walk policy was found for 100 reward functions, where the reward function r ( s, a, s ′ ) :
= r ( s ′ ) :
= N (0 , 1); that is, 100 reward functions for each of the 200 graphs.
The exact value functions can be recovered within symmetric motifs using their random walk SRs and the rewards associated with each state.
In the reduced graph, there will be a discrepancy between the value functions as the state occupancy for states in the core will not reflect transitions made inside motifs.
As such the difference between the optimal greedy policies obtained by the reduced random walk SR combined with a given reward vector, and the true optimal policies found using the SR defined over the original state space, was quantified as the total variation T V :=
1 |S ′ | (cid:80) s ∈S ′
|| π ( s ) − π ∗ ( s )
|| 1 .
The TV gives an indication of, on average, the proportion of actions taken under the inferred policy that deviate from the optimal policy.
The numbers quoted in Table 1 correspond to a deviation of around 4% and 0.1% for the Barab´asi-Albert and Watts-Strogatz graphs respectively.
The mean squared error between the derived value and optimal value is also shown—in practice this is dominated by a small number of graphs per node that are massively overestimated.
The compression is shown as a percentage size of the resulting SRs compared to an SR defined over the original graph.
For a graph with n states, the number of elements required for storage is equal to ( n − ( (cid:80) i | S A
i | − 1))
+ ( (cid:80) i | S A i | + 1) 2 .
An additional 1 is added to the second term to allow for a virtual state to be represented in the SRs of the subgraphs that allows for an agent to leave the subgraph and return to the main graph.
This is an upper bound and assumes that every subgraph is generated according to a unique motif.
[Table]
5. Future Work
An additional improvement that can be made is the use of the internal connectivities to adjust the SR of the ˆ A to reduced the value approximation error.
S´anchez-Garc´ıa (2020) use this information in the form of their Basic Symmetric Motifs to adjust pairwise network measures over compressed graphs.
The largest constraint is the need to know the full adjacency matrix—this is not tractable in reinforcement learning with high dimensional Extended Abstract Track
observations. Instead, we are interested in using the knowledge of symmetric graphs to allow agents to form expectations of local structures given their prior experience.