M ODULARITY IN R EINFORCEMENT L EARNING VIA A LGORITHMIC I NDEPENDENCE IN C REDIT A SSIGNMENT
A BSTRACT
Many transfer problems require re-using previously optimal decisions for solving new tasks, which suggests the need for learning algorithms that can modify the mechanisms for choosing certain actions independently of those for choosing others.
However, there is currently no formalism nor theory for how to achieve this kind of modular credit assignment.
To answer this question, we deﬁne modular credit assignment as a constraint on minimizing the algorithmic mutual information among feedback signals for different decisions.
We introduce what we call the modularity criterion for testing whether a learning algorithm satisﬁes this constraint by performing causal analysis on the algorithm itself.
We generalize the recently proposed societal decision-making framework as a more granular formalism than the Markov decision process to prove that for decision sequences that do not contain cycles, certain single-step temporal difference action-value methods meet this criterion while all policy-gradient methods do not.
Empirical evidence suggests that such action-value methods are more sample efﬁcient than policy-gradient methods on transfer problems that require only sparse changes to a sequence of previously optimal decisions.
I NTRODUCTION
Many transfer problems require re-using some previously optimal decisions while independently modifying others.
Existing work on modularity in machine learning has largely focused on enforcing the independence of mechanisms in the forward execution of a system (§C).
However, a main result we show is that static factorization of the mechanisms of a learner does not guarantee that those mechanisms remain independent conditioned on the training history after a credit assignment update: feedback signals produced by the credit assignment mechanism (CAM) must be independent as well.
Given the known beneﬁts of modular systems (Sussman, 2007), we would expect that when the environment changes, CAMs that produce independent feedback would enable previously optimal decisions to be re-used with less interference from other decisions, thus transfering more efﬁciently.
Our main contributions are: to formally deﬁne the constraint, which we call the modularity constraint , on the algorithmic independence among the feedback signals that modular credit assignment mechanisms must satisfy; to translate the incomputable modularity constraint into what we call the modularity criterion for practically designing learning algorithms with modular credit assignment; and to theoretically evaluate the major classes of reinforcement learning (RL) algorithms against the modularity criterion.
We show that the recently proposed cloned Vickrey society (Chang et al., 2020) satisﬁes this criterion in the most general setting, single-step temporal-difference methods satisfy it in the tabular setting, and policy gradient methods never satisfy it.
We present empirical evidence that suggests algorithms with modular credit assignment are more sample efﬁcient in transfer problems that require only sparse modiﬁcations to a sequence of previously optimal decisions.
These contributions required overcoming several theoretical challenges.
For one, we need a formalism for learning that is based on algorithmic information rather than standard Shannon informa- tion because the latter is not deﬁned for deterministic credit assignment operations.
To overcome the incomputability of algorithmic information, we draw upon algorithmic causality (Janzing &
Sch¨olkopf, 2010) to develop the algorithmic causal model of learning (ACL), which represents learning algorithms as causal graphs that can be inspected to verify the modularity constraint.
To construct ACL, we draw upon λ -calculus to reconcile the computational graph of execution, which treats learnable mechanisms as functions, with the computational graph of learning, which treats them as data.
Lastly, because the Markov decision process (MDP) is not granular enough for distinguishing feedback to the possible values an action variable could have taken, we generalize the recently-proposed societal decision-making framework (Chang et al., 2020) to a more granular computational graph for evaluating standard discrete-action RL algorithms against our criterion.
D EFINING M ODULAR C REDIT A SSIGNMENT
We ﬁrst separate the model of execution , which speciﬁes the computational graph of the learner, from the model of credit assignment , which speciﬁes the computational graph of the learning algorithm.
Deﬁnitions, background, and proofs are in the Appendix.
Model of execution.
We represent a system of learnable mechanisms as a computational graph G (Def. 10) whose factor nodes, i.e. functions, represent learnable mechanisms
f = { f 1 , ..., f K } and data nodes represent interface between the functions.
Let x f :=
( x in f , x out f ) denote input and output data nodes of function f .
The forward execution generates a topologically-sorted execution trace τ
=
( x f 1 , ..., x f M ) that records the inputs and outputs of the M functions within the computation.
Model of credit assignment.
We represent the credit assignment mechanism Π as a contextconditioned policy Π( τ , f ) → δ that generates modiﬁcations δ =
( δ 1 , ..., δ M ) to the functions f of the system, given τ as context.
Then the computational graph of a learning algorithm can be represented as a controlled Markov process (CMP) C with states f and actions δ .
The type of algorithm deﬁnes the transition function of C , which we will call UPDATE ( f , δ ) → f (cid:48) , and consequently the datatype of the feedback signals δ .
For a learning algorithm based on gradient descent, δ ki is the gradient of G ’s learning objective with respect to the function f k that participated at step i of the execution trace.
1 Then UPDATE performs the following parallel operation UPDATE ( f k , (cid:80) i δ ki )
→ f k (cid:48) over all functions f k where the functional form of UPDATE depends on the choice of optimizer like stochastic gradient descent, Adam (Kingma & Ba, 2014), etc.
We assume a gradient-based learning algorithm, but our results hold for any other optimizer with the structure of UPDATE deﬁned above.
We now adapt the standard deﬁnition of modularity to take the dynamic evolution of the learner into account.
Modularity has traditionally been deﬁned for static computational graphs as the algorithmic independence of functions (Pearl, 2009; Peters et al., 2017):
Deﬁnition 1 ( static modularity ) .
∀
[Equation]
but this deﬁnition is not useful when we expect the functions to learn to share information over the course of learning.
We can instead deﬁne modularity of the functions at iteration n + 1 of learning still as algorithmic independence, but relative to the learning history T n = ( f 1 , τ 1 , ..., f n , τ n ) as background information (Janzing & Sch¨olkopf, 2010, §2.3).
Then, because C is a Markov process, it is sufﬁcient to condition on only the latest execution trace τ n and past learner parameters f n :
Deﬁnition 2 ( dynamic modularity ) .
∀ k
(cid:54) =
j,
[Equation]
f k n +1 denotes the k th function after n credit assignment updates.
Conditioning on the history T n makes sense because any state of the learner f n can be considered the beginning of a learning process.
Static modularity is a special case of dynamic modularity when n = 0 .
As we will see, static modularity does not generally imply dynamic modularity.
We ask what constraint the CAM must satisfy for dynamic modularity to hold for G at every iteration of learning.
Satisfying such a constraint would allow functions to be modiﬁed independently with- out changing in others.
Given execution trace τ and previous learner parameters f , we deﬁne the modularity constraint as that which imposes that the gradients δ 1 , ..., δ M be jointly independent:
Deﬁnition 3 ( modularity constraint ) .
I ( δ 1 , ..., δ M | τ , f ) + = 0 .
Deﬁne a modular credit assignment mechanism as one that satisﬁes the modularity constraint.
If
G were statically modular to start (i.e. its functions share no weights) then learning with a modular CAM enforces dynamic modularity:
Theorem 1 ( modular credit assignment ) .
Dynamic modularity is enforced at learning iteration n if and only if static modularity holds, i.e. I (cid:0) f k 1 : f j 1 (cid:1) + = 0 for all k (cid:54) =
j , and the CAM satisﬁes the modularity constraint.
The key observation, as alluded to in the introduction, is that static modularity alone does not guarantee dynamic modularity – the gradients that the CAM produces must also be independent.
We observe that all modular CAMs must be factorized in the following way:
Theorem 2 ( modular factorization ) .
The credit assignment mechanism Π( τ , f ) → δ is modular if and only if K ( δ | τ , f ) + = (cid:80) Mi =1 K
( δ i | τ , f ) .
For modular CAMs, the complexity of computing feedback for the entire system is minimal because all redundant information among the gradients has been “squeezed out.”
This connection between simplicity and modularity is another way of understanding why if a CAM were not modular it would be impossible for Π to modify a function without simultaneously modifying another, other than due to non-generic instances when δ i has a simple description, i.e. δ i = 0 , which, unless imposed, are not likely to hold over all iterations of learning.
A N A LGORITHMIC C AUSAL M ODEL OF L EARNING
We showed that in general modular CAMs are required to enforce dynamic modularity, but algorithmic mutual information is generally incomputable so the modularity constraint is not practical for evaluating existing CAMs.
We propose to leverage the observation from Janzing & Sch ¨ olkopf (2010) that a computational graph, and in particular the learning process itself, can be represented as a causal graph, We thus bypass the incomputability of algorithmic mutual information by translating the modularity constraint into a criterion on d -separation that can be assessed by direct inspection.
To construct this causal graph, we reconcile the computational graph of execution G , which uses functions f to process x as data, with that of credit assignment C , which uses Π to process f as data.
We can “ﬂatten” G to treat f at the same level of operation as x by introducing the higher-order operation APPLY (Abelson & Sussman, 1996), a principle in λ -calculus known as β -reduction, into the graph construction such that ∀ f , x, APPLY ( f , x ) :
= f ( x ) .
Treating function application as a factor node enables us to treat both f and x as data nodes in the same causal graph.
Lemma 3 ( algorithmic causal model learning ) .
Given a model of execution G and of credit assignment C , deﬁne the algorithmic causal model of learning (ACL) as a computational graph L of the learning process with data nodes x , f , and δ and factor nodes APPLY , Π , and UPDATE , as well as the internal data and factor nodes that constitute the structure of APPLY and Π which vary by the learning algorithm.
Then L is a double trellis over x and f generated via an inner loop governed by
APPLY ( f , x in f ) → x out f that generates the execution trace τ and an outer loop structured as the CMP
C governed by two algorithmically independent operations Π( τ , f )
→ δ and UPDATE ( f , δ ) → f (cid:48) .
Then the strings x in f ki , x out f ki , f ki , δ ki , and the internal data nodes of APPLY and Π , for all steps of credit assignment n , satisfy the algorithmic causal Markov condition with respect to L .
Representing learning algorithms as causal graphs via Lemma 3 brings tools from algorithmic causality (Janzing & Sch ¨ olkopf, 2010) to bear on analyzing general learning algorithms.
Theorem 4 ( modularity criterion ) .
If L is faithful, the modularity constraint holds if and only if for all i the outputs
δ i and δ (cid:54)
= i of Π are d -separated by its inputs τ and f .
Thm. 4 enables us to evaluate, before any training, whether a learning algorithm satisﬁes the modularity constraint by simply inspecting L for d -separation, giving us a practical tool to both design and evaluate learning algorithms on the basis of the modularity of their CAMs.
We generally have access to the true causal graph, because the learning algorithm was programmed by us.
Therefore, assuming that L is faithful is as reasonable as assuming the operations in the programming language used to program the learning algorithm has no side-effects that interfere with L ’s data nodes, which is reasonable since we assume the programming language is agnostic to the learning task.
Henceforth we assume L is faithful, unless otherwise stated.
T HE M ODULARITY OF RL A LGORITHMS
We have developed a formalism for dynamic modularity, a constraint on the CAM for enforcing this property, and a criterion for evaluating whether the constraint is satisﬁed.
We now apply these tools to evaluate major classes of RL algorithms (Sutton & Barto, 2020) – action-value (AV) and policygradient (PG) methods.
The beneﬁt of the modularity criterion (MCn) is that we can use graphical language for our analysis, which simpliﬁes much of the proofs.
For RL the functions f are decision mechanisms that each control a different value of the action variable (e.g. the probability of the action for PG methods, or the Q -value of the action for AV methods).
Which RL algorithms satisfy the modularity criterion?
MCn can be violated through shared hidden variables in Π that couple together gradients δ , which cause the δ ki ’s to not be d -separated given τ and f .
This hidden variable is the normalization term of the softmax for PG methods and the sum of estimated returns for n -step temporal difference methods where n > 1 , abbrv.
TD( n > 1 ).
However, MCn is not violated by TD( 0 ) methods if the decision sequence τ does not contain cycles.
Corollary 4.1.
All PG methods do not satisfy MCn.
Corollary 4.2.
All TD( n > 1 ) methods do not satisfy MCn.
Corollary 4.3.
TD( 0 ) methods satisfy MCn for acyclic τ .
Which RL algorithms enforce dynamic modularity?
Having narrowed down the classes of RL algorithms whose causal structures satisfy MCn to TD( 0 ) methods, we consider the models of execution G of individual TD( 0 ) methods – Q -learning
(Watkins & Dayan, 1992), SARSA (Rummery & Niranjan, 1994), and CVS (Chang et al., 2020) – on whether they satisfy static modularity (and thereby dynamic modularity), i.e. Thm. 1 holds.
We assume that the initial parameters of f are randomly initialized, so the only possible source of dependence among f is if they share parameters.
Corollary 1.1.
Thm. 1 holds for Q -learning, SARSA, CVS in the tabular setting.
Corollary 1.2.
In the general function approximation setting, Thm. 1 holds for CVS.
The main implication is that if we want dynamic modularity, then from the model of credit assignment we need the CAM to produce algorithmically independent gradients, and from the model of execution we need the decision mechanisms not share weights.
Then, an RL algorithm with dynamic modularity makes it possible for individual decision mechanisms to be modiﬁed independently without an accompanying modiﬁcation to other decision mechanisms, which is what we want for transferring to tasks whose optimal solution re-use previously optimal decisions.
D ISCUSSION
We have (1) generalized the static deﬁnition of modularity to encompass dynamic learning systems, (2) proven the necessity of the modularity constraint that the credit assignment mechanism must satisfy to enforce this dynamic modularity, (3) introduced the connection between learning algorithms and causal graphs as means to practically achieve this constraint via the modularity criterion, and (4) evaluated a major classes of RL algorithms against this criterion.
In §E we also show that (5) dynamically modular RL methods tend to correlate with more efﬁcient transfer when previously optimal decisions must be reused.
The key conceptual contribution we add to previous work on modularity is that the concept must be understood through the interaction between the learning system and credit assignment mechanism, not just from the learning system alone.
The connection we have established among credit assignment, modularity, and algorithmic information theory, in particular the link between learning algorithms and algorithmic causality, opens many opportunities for future work, such as new ways of formalizing inductive bias in the algorithmic causal structure of learning systems and the learning algorithms that modify them.
A B ACKGROUND AND A SSUMPTIONS
We present background on algorithmic information theory and standard causality.
For a more thorough treatment on the foundational mathematics and formalism, please refer to Li et al.
(2008) for algorithmic information theory, to Pearl (2009) for standard causality, and to Janzing & Sch¨olkopf (2010)
; Peters et al.
(2017) for algorithmic causality.
This section also states our assumptions for the results that we prove in §D. Lastly, we review the societal decision-making framework from Chang et al.
(2020) that shows we can equivalently re-interpret a learnable discrete-action policy as composed of a society of learnable action-speciﬁc functions and a non-learnable selection mechanism.
A. N OTATION
We denote with bolded uppercase monospace a computation graph at a single level of abstraction (e.g. the model of execution G , the model of credit assignment C ).
We denote with blackboard bold
(e.g. the algorithmic causal model of learning L ) a computation graph that represents multiple levels of abstraction.
We denote binary strings that represent the data nodes in L with lower case (e.g. x or f ), where script ( x ) is used to emphasize that the string typically represents data and monospace ( f ) is used to emphasize that the string typically represents a function.
We use bolded lower case (e.g. τ , δ , f ) to indicate a group of binary strings.
We denote the factor nodes and macro-operations in L (e.g., the transition function T and reward function R of G , the credit assignment mechanism Π , the transition function UPDATE of C ) with uppercase.
We write f ( x ) → y to mean “a program f that takes a string x as input and produces a string y as output.”
A. B ACKGROUND ON ALGORITHMIC CAUSALITY
The formalism of algorithmic causality derives from Janzing & Sch ¨ olkopf (2010); Peters et al.
(2017), which builds upon algorithmic statistics (G ´ acs et al., 2001).
Here we directly restate or paraphrase additional relevant deﬁnitions, postulates, and theorems from Janzing & Sch ¨ olkopf (2010) and G ´ acs et al.
A.2. A LGORITHMIC INFORMATION THEORY Kolmogorov
complexity Kolmogorov complexity (Solomonoff, 1960; 1964; Kolmogorov, 1965;
Chaitin, 1966; 1975; Li et al., 2008) is a function K : { 0 , 1 } ∗ → N from the binary strings { 0 , 1 } ∗ to the natural numbers N that represents the amount of information contained in an object (represented by a binary string).
Deﬁnition 4 ( Kolmogorov-complexity ) .
Given a universal Turing machine and universal programming language as reference, the Kolmogorov complexity K ( s ) is the length of the shortest program that generates s .
The conditional Kolmogorov complexity K ( y | x ) of a string y given another string x is the length of the shortest program that generates y given x as input.
Let the shortest program for string x be denoted as x ∗ .
The joint Kolomogorov complexity K ( x, y ) is deﬁned as:
[Equation]
The invariance theorem (Kolmogorov, 1965) states that the Kolmogorov complexities of two strings written in two different universal languages differ only up to an additive constant.
Therefore, we can assume any reference universal language for deﬁning K (e.g. Python) and work with equalities ( + = ) and inequalities ( + ≥ , + ≤ ) up to an additive constant.
Algorithmic mutual information Algorithmic mutual information I measures the amount of information two objects have in common:
Deﬁnition 5 ( algorithmic mutual information ) .
The algorithmic mutual information of two binary strings x, y is
[Equation]
The conditional algorithmic mutual information of strings x, y given string z is
[Equation]
We can intuitively think of I ( x : y | z ) as, given z , the number of bits that can be saved when describing y knowing the shortest program that generated x .
Then algorithmic independence is the property of two strings that says that the description of one cannot be further compressed given knowledge of the other.
Deﬁnition 6 ( algorithmic conditional independence ) .
Given three strings x, y, z , x is algorithmically conditionally independent of y given z , denoted by x ⊥⊥ y | z , if the additional knowledge of y does not allow for stronger compression of x , given z .
That is:
[Equation]
We further deﬁne the joint conditional independence of strings x 1 , ..., x n given strings y 1 , ..., y m analogously:
Deﬁnition 7 ( algorithmic joint conditional independence ) .
Strings x 1 , ..., x n are algorithmically jointly conditionally independent given strings y 1 , ..., y m if
[Equation]
which means that
[Equation]
meaning that, conditioned on knowing y 1 , ..., y m , the length of the joint description of x 1 , ..., x n cannot be further compressed than sum of the lengths of the descriptions of the individual strings x i .
A proof of the equivalence between Eqs.
1 and 2 is given by the proof of Theorem 3 in Janzing & Sch ¨ olkopf (2010).
We state as a lemma the following result from G´acs et al.
(2001, Corollary Π . 8 ) that states the mutual information of strings x and y cannot be increased by separately processing by functions f and g .
Lemma 5 ( information non-increase ) .
Let f and g be computable programs.
Then
[Equation]
This intuitively makes sense: if K ( f ) is constant with respect to x and K ( g ) is constant with respect to y (i.e. K ( f ) + = 0 and K ( g ) + = 0 ), then mutual information cannot increase between x and y separately with f and g .
In particular, if x and y were independent to begin with (i.e. I ( x : y ) + = 0 )
, then I ( f ( x ) : g ( y )) + = 0 .
The following lemma states that the mutual information between two strings is constant if we condition on one of the strings.
Lemma 6 ( self-conditioning ) .
For two strings x and y ,
I ( x : y | y ) + = 0 .
Proof.
I ( x : y | y )
+
= K ( x | y )
+ K ( y | y )
− K ( x, y | y )
+
= K ( x | y )
+ K ( y | y ) −
[ K ( x | y )
+ K ( y | x ∗ , y )]
+ = K
( x | y )
+ 0 − [ K ( x | y )
+ 0]
+ = K
( x | y ) − K ( x | y )
+ = 0 .
Terminology In this paper, we regard the following statements about a program f ( x ) →
y as equivalent:
• “ K ( f ) + = 0 .”
• “ f is an O (1) -length program.”
• “ K ( f ) is constant with respect to x .”
Note that K ( f ) + = 0 implies that f and x are algorithmically independent (i.e. I ( f : x ) + = 0 ) because
[Equation]
C
[Equation]
Before we review algorithmic causality, we ﬁrst review some key concepts in standard causality: structural causal models and d -separation.
The following deﬁnition deﬁnes standard causal models over random variables as Bayesian networks represented as directed acyclic graphs (DAG).
Deﬁnition 8 ( structural-causal-model ) .
A structural causal model (SCM) (Pearl, 1995; 2009) represents the assignment of random variable X as the output of a function, denoted by lowercase monospace (e.g. f ), that takes as input an independent noise variable N X and the random variables { PA X } that represent the parents of X in a DAG:
[Equation]
Given the noise distributions P ( N X ) for all variables X in the DAG, SCM entails a joint distribution P over all the variables in the DAG (Peters et al., 2017).
The graph-theoretic concept of d -separation is used for determining conditional independencies induced by a directed acyclic graph (see point 3 in Thm. 9):
Deﬁnition 9 ( d -separation ) .
A path p in a DAG is said to be d-separated (or blocked) by a set of nodes Z if and only if
p contains a chain i
→ m → j or fork i ← m → j such that the middle node m is in Z , or 2.
p contains an inverted fork (or collider)
i →
m
← j such that the middle node m is not in Z and
such
that no descendant of m is in Z
A set of nodes Z d -separates a set of nodes X from a set of nodes Y if and only if Z blocks every (possibly undirected) path from a node in X to a node in Y .
A.2. A LGORITHMIC CAUSALITY
We paraphrase Post. 6 and Thm. 4 from Janzing & Sch¨olkopf (2010), which generalize structural causal models, or Bayesian networks (Pearl, 1995; 2009), to general programs, allowing us to use the language of causality to assess the independence of computable objects by inspecting the structure of the generative program that produced them.
Deﬁnition 10 ( computational graph ) .
Deﬁne a computational graph as a directed acyclic factor graph (DAG) of data nodes x 1 , ..., x N and function nodes f 1 , ..., f N , constructed as follows.
Let each x j be computed by a program f j with length O (1) from its parents { pa j } and possibly an additional noise input n j .
Assume the noise n j are jointly independent: n j ⊥⊥ { n (cid:54) =
j } .
Formally, x j := f j ( { pa j } , n j ) , meaning that the Turing machine computes x j from the input { pa j } , n j using the additional program f j and halts.
The computational graph represents a probabilistic program (van de Meent et al., 2018; Goodman et al., 2016; Mansinghka et al., 2009) in the general case, and would represent either a standard
Bayesian network if every f j takes in a noise variable or a deterministic program if none do.
The algorithmic causal Markov condition , which states that d -separation implies conditional independence, generalizes the standard Markov condition to general programs:
Theorem 7 ( algorithmic causal Markov condition ) .
Let { pa j } and { nd j } respectively represent concatenation of the parents and non-descendants (except itself) of x j in a computational graph.
Then ∀ x j , x j ⊥⊥ { nd j } | { pa j } .
It is typical to assume its converse, known as faithfulness:
Postulate 8 ( faithfulness (Spirtes et al., 2000)) .
Given three sets S , T , R of nodes in a computational graph, I ( S : T | R ) + = 0 implies R d-separates S and T .
The following theorem Janzing & Sch¨olkopf (2010, Thm. 3) establishes the connection between the graph-theoretic concept of d -separation with condition algorithmic independence of the nodes of the graph.
Theorem 9 ( equivalence of algorithmic Markov conditions ) .
Given the strings x 1 , ..., x n and a computational graph, the following conditions are equivalent:
Recursive form: the joint complexity is given by the sum of complexities of each node x j , given the optimal compression of its parents { pa j } :
[Equation]
Local Markov Condition:
Every node x j is independent of its non-descendants { nd j } , given the optimal compression of its parents { pa j } :
[Equation]
[Equation]
Global Markov Condition:
Given three sets S , T , R of nodes
[Equation]
if R d-separates S and T .
Together, Thm. 7, Post. 8, and Thm.
9 imply that data nodes in a computational graph are algorithmically independent if and only if they are d -separated in the computational graph
. A. A SSUMPTIONS
This paper analyzes learning algorithms from the perspective of algorithmic information theory, speciﬁcally algorithmic causality.
To perform this analysis, we assume the following, and state our justiﬁcations for such assumptions:
The learning algorithm is implemented in on a universal Turing machine with a universal programming language.
Justiﬁcation: This is a standard assumption in machine learning research that the machine learning algorithm can be implemented on a machine.
Each initial parameter of the learnable functions f is jointly algorithmically independent of the other initial parameters.
Justiﬁcation: This is a standard assumption in machine learning research that the noise from the random number generator is given background knowledge (Janzing & Sch¨olkopf, 2010, §2.3), thus allowing us to ignore possible dependencies among the parameters induced by the random number generator in developing our algorithms.
UPDATE is an O (1) -length program.
Justiﬁcation: This is a standard assumption in machine learning research that the source code that computes the update rule (e.g. a gradient descent step) is agnostic to the feedback signals (e.g. gradients)
it takes as input.
L is faithful.
That is, any conditional independence among the data nodes ( f , x , δ , and the internal data nodes of APPLY and Π ) in L is due to the causal structure of L rather than a non-generic setting of these data nodes.
Justiﬁcation: Faithfulness has been justiﬁed for standard causal models (Meek, 1995).
Deriving an algorithmic analog has been the subject of ongoing work (Lemeire & Janzing, 2013; Lemeire, 2016).
For our work, a violation of faithfulness means that two nodes x , y in the computational graph have I ( x : y )
+ = 0 when they are not d -separated in the computational graph.
This would happen if x and y were tuned in such a way that makes one compressible given the other.
Given assumption (2) above, the source of a violation of faithfulness must be the data experienced by the learning algorithm.
Indeed, the data could be such that after learning certain parameters within f may be conditionally independent given the training history, as suggested by Csord´as et al.
(2020); Filan et al.
(2020); Watanabe (2019).
However, as our focus is on theoretical results that hold regardless of the data distribution the learning algorithm is trained on, we consider the speciﬁc instances where the data does induce such faithfulness violations as “non-generic” and thus out of scope of the paper.
A. S OCIETAL D ECISION -M AKING
An MDP is the standard computational graph for a sequential decision problem with N discrete actions, deﬁned with states s ∈ Ω S , actions a ∈ { 1 , ..., N } , transition function T : Ω S ×{ 1 , ..., N } → Ω S , reward function R : Ω S × { 1 , ..., N } → R , and discount factor γ .
We deﬁne a decision as a value of a .
The MDP objective is to maximize the return (cid:80)
Tt =
γ t R ( s t , a t ) with a policy π : Ω S → { 1 , ..., N } .
However, representing the policy output as a single action a is not granular enough for distinguishing the feedback into possible values of a .
The societal decision-making (SDM) framework (Chang et al., 2020) offers a more ﬁne-grained alternative computational graph to the MDP by interpreting a discrete-action policy as a society of N agents α n that each controls a different possible action value.
Each agent is a tuple α n = ( ψ n , φ n ) of a bidder ψ n : Ω S → Ω B and a ﬁxed transformation mechanism φ n : Ω S → Ω S .
SDM decomposes the policy π of the standard MDP into a composition of two operations: one that computes the bids b ns := ψ n ( s ) for all bidders n , and one that applies a selection mechanism
S : Ω NB → { 1 , ..., N } on the bids to select the index of the agent whose transformation mechanism is the move from s to s (cid:48) that the action a represents.
SDM thus refactors (Curry et al., 1958)
the transition and reward functions as T : { 1 , ..., N } → [Ω S → Ω S ] and R : { 1 , ..., N } → [Ω S → R ] .
Chang et al.
(2020) introduced the cloned Vickrey society (CVS) RL algorithm as an on-policy single-step temporal-difference action-value method.
CVS interprets the Bellman optimality equation as an economic transaction between agents seeking to optimize their utilites in a Vickrey auction (Vickrey, 1961) at each time-step.
The Vickrey auction is the selection mechanism that selects the highest bidding agent i , which receives a utility
U is
[Equation]
and the rest receive a utility of 0 .
In CVS each agent bids twice via an arbitrary function: the highest and second highest bids are produced by the same function parameters.
The auction incentivizes each agent to truthfully bid the Q -value of its associated transformation mechanism, independent of the identities and bidding strategies of other agents.
B F ROM M ONOLITHIC P OLICIES TO D ECISION M ECHANISMS
Deﬁning the model of execution G for RL requires us to deﬁne f , x in , x out .
If we were to use the standard MDP for G , we would represent a single learnable function f as the policy π ( s )
→ a , whose input x in is the current state s i and whose output x out is a tuple ( a i , s i +1 , r i ) of the action, next state, and reward.
However, this is level of granularity restricts us from representing feedback signals to AV methods, which may target only a speciﬁc value the action may take rather than the entire action distribution.
We hence adopt SDM for G , which replaces the monolithic policy with a set of functions that control each a different value of the action variable.
We generalize the “bidder” terminology from §A.4 to encompass standard AV and PG methods by deﬁning the action-speciﬁc decision mechanism as a function that produces a bid b ki for its corresponding action, which would correspond to an actionspeciﬁc estimated Q -value Q ( · , a ) for AV methods or an action probability p ( a = k | · ) for policy gradient methods.
The action is then selected from the bids via algorithmically independent selection mechanism S , such as a stochastic sampler for a PG method or an ε -greedy sampler for Q -learning.
Observe that the agent-environment boundary separates the function we want to learn from other algorithmically independent functions, such as the transition function.
Thus, we can then absorb S into the environment (Fig.
1a-c), thus splitting the formerly monolithic policy into a society of K decision mechanisms as the learnable functions of G , each with input as s i and output as the tuple ( b ks i , s i +1 , r i , w i ) , where w i is a binary ﬂag that indicates whether the S chose its corresponding action.
Deﬁne a decision sequence as the execution trace instantiated for RL with SDM.
C R ELATED W ORK
We synthesize perspectives from multi-agent reformulations of intelligence (Minsky, 1988;
Chang et al., 2020; Balduzzi, 2014; Baum, 1996; Srivastava et al., 2013), computer programming (Liskov, 1972; Parnas, 1971; 1972), and causality (Hausman & Woodward, 1999; Pearl, 2009; Peters et al., 2017) around the theme of algorithmic independence (Janzing & Sch¨olkopf, 2010; Kolmogorov, 1965; Li et al., 2008) to present a formalism for modularity in RL that reframes how RL is traditionally conceptualized (Sutton & Barto, 2020) in several ways.
One conceptual shift is to treat the decision-mechanism (§B), rather than the policy, as the core primitive of decision-making, building directly off the societal decision-making framework (Chang et al., 2020) that refactors the traditional monolithic agent into a multi-agent system of mechanisms that each control a different decision.
The practical beneﬁt of a multi-agent reframing of a traditionally monolithically-framed problem enables the credit assignment (Minsky, 1961) into individual decisions to be decoupled from that into others, which motivates a second conceptual shift: that modularity is a property of not merely how the system itself is statically factorized – the view taken by most previous work on modularity in machine learning (Andreas et al., 2016; Devin et al., 2017; Goyal et al., 2019; Kirsch et al., 2018;
Pathak et al., 2019; Alet et al., 2018;
Chang et al., 2018; Csord´as et al., 2020), but how the outer process that changes the system is factorized as well .
In computer programming this outer process is the human reasoning process (Dijkstra, 1968); in machine learning it is the credit assignment mechanism.
To analyze how the causal structure of the credit assignment mechanism interacts with the causal structure of the decision-maker that it modiﬁes requires a third conceptual shift based on the duality of “code as data” (Abelson & Sussman, 1996) that embeds in the same causal graph both the decision mechanisms that transform states in the MDP and the credit assignment (meta-)mechanism that transforms the decision mechanisms themselves in the learning process.
D P ROOFS
Given the assumptions stated in §A.3, we now provide the proofs for our theoretical results.
We will prove Lemma 3 ﬁrst.
Together with the faithfulness postulate (Post. 8) and the equivalence of algorithmic Markov conditions (Thm. 9) we can prove algorithmic independence by inspecting the graph of L for d -separation.
Lemma 3 (algorithmic causal model of learning).
Given a model of execution G and of credit assignment C , deﬁne the algorithmic causal model of learning (ACL) as a computational graph L of the learning process with data nodes x , f , and δ and factor nodes APPLY , Π , and UPDATE , as well as the internal data and factor nodes that constitute the structure of APPLY and Π which vary by the learning algorithm.
Then L is a double trellis over x and f generated via an inner loop governed by APPLY ( f , x in f ) → x out f that generates the execution trace τ and an outer loop structured as the CMP C governed by two algorithmically independent operations Π( τ , f ) → δ and UPDATE ( f , δ )
→ f (cid:48) .
Then the strings x in f ki , x out f ki , f ki , δ ki , and the internal data nodes of APPLY and Π , for all steps of credit assignment n , satisfy the algorithmic causal Markov condition with respect to L .
Proof.
L is a well-deﬁned computational graph and so by Thm.
7 it satisﬁes the algorithmic causal
Markov condition.
Remark.
By Lemma 5, if a set of data nodes in L are independent, then processing them separately with factor nodes of L will maintain this independence.
For example, given that the UPDATE operation is applied in separately for each pair ( f k , (cid:80) i δ ki ) to produce a corresponding f k (cid:48) , then if ( f k , (cid:80) i δ ki ) were independent of ( f j , (cid:80) i δ ji ) before applying UPDATE , then f k (cid:48) would be independent of f j (cid:48) after applying UPDATE .
Theorem 1 (modular credit assignment).
Dynamic modularity is enforced at learning iteration n if and only if static modularity holds, i.e. I (cid:0) f k 1 : f j 1 (cid:1) + = 0 for all k (cid:54) =
j , and the CAM satisﬁes the modularity constraint.
Proof.
We will prove by induction on n .
The inductive step will make use of the equivalence between d -separation and conditional independence.
Base case: n = 1 .
There is no training history, so static modularity is equivalent to dynamic modularity.
Inductive hypothesis: Assuming that dynamic modularity holds if and only if static modularity and modular credit assignment hold for learning iteration n − 1 , dynamic modularity holds if and only if static modularity and modular credit assignment hold for learning iteration n .
Inductive step: The modularity constraint states
[Equation]
Dynamic modularity at iteration n − 1 states that
[Equation]
These two above statements correspond to the computational graph in Fig.
Note that by Def. 7, disjoint subsets of δ 1 , ..., δ M also have have zero mutual information up to an additive constant.
Letting these subsets be (cid:80)
i δ ki where k is the index of function
f k in f , then
[Equation]
Then, as we can see by direct inspection in Fig. 3, f k n and f j n are d -separated by ( τ n , f n ) , which is equivalent to saying that dynamic modularity holds for iteration n .
Theorem 4 (modularity criterion).
If L is faithful, the modularity constraint holds if and only if for all i the outputs
δ i and δ (cid:54)
= i of Π are d -separated by its inputs τ and f .
Proof.
The forward direction holds by the equivalence of algorithmic causal Markov conditions (Thm. 9), and the backward direction holds by the faithfulness assumption.
Theorem 2 (modular factorization).
The credit assignment mechanism Π( τ , f ) → δ is modular if and only if K ( δ | τ , f ) + = (cid:80) Mi =1 K
( δ i | τ , f ) .
Proof.
The proof comes from the deﬁnition of algorithmic mutual information.
[Equation]
Corollary 4.1 (policy gradient).
All PG methods do not satisfy MCn.
Proof.
It sufﬁces to identify a single shared hidden variable that renders δ 1 , ..., δ M not d -separated.
Computing the policy gradient includes the log probability of the policy as one of its terms.
Computing this log probability for any action involves the same normalization constant (cid:80) k b k .
This normalization constant is a hidden variable that renders δ 1 , ..., δ M not d -separated, as shown in
Fig.
Corollary 4.2 (TD ( n > 1) ).
All TD( n > 1 ) methods do not satisfy MCn.
Proof.
It sufﬁces to identify a single shared hidden variable that renders δ 1 , ..., δ M not d -separated.
TD( n > 1 ) methods include a sum of estimated returns or advantages at different steps of the decision sequence that is shared among multiple δ i ’s.
This sum is the hidden variable that renders δ 1 , ..., δ M not d -separated, as shown in Fig.
Corollary 4.3 (TD (0) ).
TD( 0 ) methods satisfy MCn for acyclic τ .
Proof.
It sufﬁces to identify a single shared hidden variable that renders δ 1 , ..., δ M not d -separated.
TD( 0 ) methods produce gradients as δ k i :
= g ( b k i , s i , s i +1 , r i , f ) if w i =
(the decision mechanism was selected) and δ ki := 0 otherwise, for some function g .
The only hidden variable is [max j b js
i +1 ] , and for acyclic τ there is only one state s i in τ that transitions into s
i +1 .
Therefore the hidden variable is unique to each of δ 1 , ..., δ M , so δ 1 , ..., δ M remain d -separated, as shown in Fig.
Corollary 1.1 (tabular).
Thm. 1 holds for Q -learning, SARSA, CVS in the tabular setting.
Proof.
In the tabular setting, decision mechanisms are columns of the Q -table corresponding to each action.
These columns do not share parameters, so static modularity holds.
Then because
Q -learning, SARSA, and CVS are TD( 0 ) methods, by Corollary 4.3, their credit assignment mechanisms are modular.
Therefore Thm. 1 holds.
Corollary 1.2 (function approximation).
In the general function approximation setting, Thm.holds for CVS.
Proof.
The decision mechanisms of CVS do not share weights, so static modularity holds.
By
Corollary 4.3 its credit assignment mechanism is modular.
Therefore Thm. 1 holds.
E N UMERICAL S IMULATIONS
We have deﬁned dynamic modularity and shown what properties the RL algorithm needs to achieve it, but how dynamic modularity correlates with transfer efﬁciency depends on the task and is thus an empirical question.
Our overarching question is: if the transfer problem requires re-using previously optimal decisions, then how does having a dynamically modular method affect transfer efﬁciency?
There is inevitably a gap between any theoretical model and its empirical implementation, so we cannot ever claim deﬁnitive empirical validation of our theories.
However, viewing our empirical results from the lens of ACL provides a useful framework for generating explanations and future hypotheses for understanding the differences in transfer behavior.
Our analysis of AV and PG methods focuses primarily on on-policy methods CVS and PPO because there are still many factors that inﬂuence the learning of off-policy methods that are still not well understood (Achiam et al., 2019; Kumar et al., 2020; Van Hasselt et al., 2018; Fu et al., 2019).
We also compare with statically modular version of PPO, abbrv.
PPOF, whose decision mechanisms do not share weights.
We consider transfer problems where only one decision in a previously optimal decision sequence needs to be changed.
States are represented as binary vectors.
The reward is only 1 at the end of the sequence if the task is solved otherwise it is 0.
The relationship between the training and transfer
MDP is given by an intervention in the MDP transition function, concretely shown in Fig.
In the same way that analysis of d -separation is conducted with triplets of nodes, we exhaustively enumerated all possible topologies of triplets of decisions: linear chain, common ancestor, and common descendant (Fig. 8, left).
For each topology we exhaustively enumerated all ways of making an isolated change to an optimal decision sequence.
The common ancestor and common descendant topologies involve multi-task training for two decision sequences of length two, while linear chain involves single-task training for a decision sequence of length three.
For each topology we have a training task and three independent transfer tasks that each start with the learner from the training task, where each transfer task represents a different way to modify the MDP used for training.
This exhaustive enumeration enables us to ask a wide range of questions from a single comprehensive plot (Fig. 8).
How does static modularity affect transfer efﬁciency?
We compare PPO and PPOF, which are trained exactly the same, except that the decision mechanisms of PPOF do not share weights while those of PPO do.
The consistent improvement of PPOF over PPO in transfer, often to a greater degree than during training (with the exception of the bottom right of Fig.
8), suggests that decoupling decision mechanisms is itself already useful.
Dynamic modularity vs static modularity.
We compare CVS and PPOF.
While PPOF largely transfers almost as efﬁciently as CVS for other transfer problems, the interesting comparison is the bottom right of Fig.
8, where PPOF transfers about six times slower.
This is the transfer problem where the last decision must be changed between training and transfer.
At the beginning of transfer, the (suboptimal) reward propagates immediately to all the decision mechanisms involved at all timesteps for PPOF during a single credit assignment update, whereas it only propagates back one step for CVS.
The effect of this can be seen in how CVS and PPOF modify their decision mechanisms over time in Fig.
9, where we observe that bids of CVS move independently whereas a change in one bid in PPOF is accompanied by a corresponding change in other bids, even those that do not need to be changed.
This coupling of gradient signal across both time and decision mechanisms could partially explain why CVS recovers much faster than PPOF.
Bandit
The following experiment speciﬁcally targets the transfer differences between CVS, which is dynamically modular, and PPOF, which is statically modular but not dynamically modular, in a way that studies in isolation the effect of independent gradients into different decision mechanisms on transfer efﬁciency.
Both are on-policy algorithms, so the main difference between the two is the gradients in CVS are independent while the gradients in PPOF are not.
We consider a single-step
MDP, or a bandit, with four actions: A , B , C , D .
During training, taking action A gives a reward of 0.8, action B a reward of 0.6, action C a reward of 0.4, and action D a reward of 0.2.
During transfer the rewards for action A and D are swapped.
Thus to transfer requires only to swap the bidding behavior of the decision mechanism for action A and the decision mechanism for action
D .
This is the simplest transfer task to compare static and dynamic modularity because it removes the temporally extended nature of MDPs as a source of dependency in the gradients.
As we can see from Fig.
10, CVS transfers about 3.9 times faster than PPOF even though they have a similar sample efﬁciency during the training task.
This result is signiﬁcant because it shows that even when the transfer task is very simple and the decision mechanisms do not share weights, independence in gradients makes a big difference in transfer efﬁciency, thereby providing support for our overarching hypothesis that modular credit assignment plays a major role in realizing the extensibility beneﬁts of modularity in machine learning.
Disjoint We next test the performance of these methods on transfer to a novel task that shares no components of with the pretraining task.
More speciﬁcally, we begin by training on task (a) in
Fig.
11 which has an optimal action sequence A-C with a reward of 0.8.
Once converged, we transfer the model to task (b) where the optimal action sequence is now B-D, also with a reward of 0.8.
This allows us to investigate any interference of the previously learned behaviour in how the new task is learned.
We see that while all three methods converge within the same number of steps in the environment for Training Task (a), PPO takes 3.8 times more training steps than CVS to converge in Transfer Task (b).
Similarly, PPOF is 2.6 times slower than CVS to converge for this transfer task.
We expect a modular method to be able to transfer efﬁciently when the optimal solution is disjoint since previously learned task’s optimal behavior is isolated to the corresponding set of agents.
Whereas the signiﬁcantly slower performance of the policy gradient methods in comparison to CVS suggests that the dependencies among the gradients of PPO and PPOF and also those in the shared weights for PPO play a role in making them transfer slower.
We also note that both Training and Transfer tasks (a) and (b) are of the same difﬁculty and thus the greater sample efﬁciency of
CVS in task (b) cannot be attributed to CVS being innately more sample efﬁcient on any task as all methods are equally sample efﬁcient for task (a).
In conclusion, how does dynamic modularity affect transfer efﬁciency?
Fig.
9 shows that CVS is able to modify different decisions independently, while this modiﬁcation is coupled for PPOF, which does not have modular credit assignment.
Fig.
8 shows CVS has the highest transfer efﬁciency most consistently.
PPOF is less consistent, possibly due to its lack of modular credit assignment, but its generally better performance than PPO suggests that modularity in some form plays a key role in efﬁcient transfer.
F S IMULATION D ETAILS
We implemented our simulations using the PyTorch library (Paszke et al., 2019).
F. I MPLEMENTATION D ETAILS
The underlying PPO (Schulman et al., 2017) implementation used for CVS, PPO, and PPOF used a policy learning rate of 4 × 10 − 5 , a value function learning rate of 5 × 10 − 3 , a clipping ratio of 0 .
2 , a
GAE (Schulman et al., 2015) parameter of 0 .
95 , a discount factor of 0 . 99 , entropy coefﬁcient of 0 . 1 , and the Adam (Kingma & Ba, 2014) optimizer.
For all algorithms, the policy and value functions for our algorithms were implemented as neural network function approximators that used one hidden layer with rectiﬁed linear units.
For all algorithms, performed a PPO update every 4096 samples with a minibatch size of 256.
These hyperparameters were chosen from
F.
T RAINING D ETAILS
All learning curves are plotted from three random seeds, with a different learning algorithm represented by a different hue.
The dark line represents the mean over the seeds.
The lighter lines represent the curves for individual seeds.
Our protocol for transfer is as follows.
A transfer problem is deﬁned by a (training, transfer) task pair, where the initial network parameters for the transfer task are the network parameters learned the training task for T samples.
In our simulations, we set T to 10 7 because that was about double the number of samples for all algorithms to visually converge on the training task for all seeds.
To calculate the relative sample efﬁciency of CVS over PPO and PPOF (e.g. 4.5x and 6.0x respectively in the bottom-up right corner of Fig.
8), we set the criterion of convergence as the number of samples after which the return deviates by no more than ε = 0 .
01 from the optimal return for 30 epochs of training, where each epoch of training trains on 4096 samples.
F. E NVIRONMENT D ETAILS
The environment for our experiments shown in Fig.
8 represented as discrete-state, discrete-action
MDPs.
Each state is represented by a binary-valued vector.
The structure of the MDP can best be explained via an analogy to a room navigation task, which we will explain in the context of the A → B → C task in the Linear Chain topology.
In this task, there are four rooms, room 0, room 1, room 2, and room 3.
Room 0 has two doors, labeled A and F , that lead to room 1.
Room 1 has two doors, labeled B and E , that lead to room 2.
Room 2 has two doors, labeled C and D .
Doors are unlocked by keys.
The state representation is a concatenation of two one-hot vectors.
The ﬁrst one-hot vector is of length four; the “1” indicates the room id.
The second one-hot vector is of length six; the “1” indicates the presence of a key for door A , B , C , D ,
E , or F .
Only one key is present in a room at any given time.
If the agent goes through the door corresponding to the key present in the room, then the agent transitions into the next room; otherwise the agent stays in the same room.
In the last room, if the agent opens the door corresponding to the key that is present in the room, then the agent receives a reward of 1 .
All other actions in every other state receive a reward of 0 .
Therefore the agent only gets a positive reward if it opens the correct sequence of doors.
For all of our experiments, the optimal policy is acyclic, but a suboptimal decision sequence could contain cycles
Therefore, for the training task in the Linear Chain topology where the optimal solution is A →
B → C , the possible states are
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
For the transfer task whose optimal solution is A → B → D , the possible states are
[1, 0, 0, 0 ; 1, 0, 0, 0, 0, 0] # room 0 with key for A [0, 1, 0, 0 ; 0, 1, 0, 0, 0, 0] # room 1 with key for B [0, 0, 1, 0 ; 0, 0, 0, 1, 0, 0] # room 2 with key for D.
Whereas for the Linear Chain topology the length of the optimal solution is three actions, for the
Common Ancestor and Common Descendant topologies this length is two actions.
Common Ancestor and Common Descendant are multi-task problems.
As a concrete example, in the training task for Common Ancestor is a mixture of two tasks, one whose optimal solution is A → B and one whose optimal solution is A → C .
Following the analogy to room navigation, this task is set up such that after having gone through door A , half the time there is a key to open door B and half there is a key to open door C .
F. C OMPUTING D ETAILS
The computing infrastructure was an AWS c5d.18xlarge instance.
The average runtime training on 10 7 samples was three hours for PPO and PPOF and 6 hours for CVS.