A Direct Approximation of AIXI Using Logical State Abstractions
Abstract
We propose a practical integration of logical state abstraction with AIXI, a Bayesian optimality notion for reinforcement learning agents, to signiﬁcantly expand the model class that AIXI agents can be approximated over to complex historydependent and structured environments.
The state representation and reasoning framework is based on higher-order logic, which can be used to deﬁne and enumerate complex features on non-Markovian and structured environments.
We address the problem of selecting the right subset of features to form state abstractions by adapting the  -MDP optimisation criterion from state abstraction theory.
Exact
Bayesian model learning is then achieved using a suitable generalisation of Context
Tree Weighting over abstract state sequences.
The resultant architecture can be integrated with different planning algorithms.
Experimental results on controlling epidemics on large-scale contact networks validates the agent’s performance.
Introduction
The AIXI agent [1] is a mathematical solution to the general reinforcement learning problem that combines Solomonoff induction [2] with sequential decision theory.
At time t , after observing the observation-reward sequence or 1: t  1 and performing previous actions a 1: t  1 the AIXI agent chooses action a t given by
[Equation]
where U is a universal Turing machine, U ( q, a 1: n ) is the output of U given program q and input a 1: n , m 2 N is the lookahead horizon and l ( q ) is the length of q in bits.
[1] shows that AIXI’s environment model converges rapidly to the true environment and its policy is pareto optimal and self-optimising.
While mathematically elegant, a fundamental issue with AIXI is that it is only asymptotically computable.
The question of whether a good practical approximation of AIXI can be constructed remains an open problem twenty years after AIXI was ﬁrst introduced [3].
The most direct approximation of
AIXI given so far is [4], in which the Bayesian mixture over all Turing machines is approximated over n -Markov environments using Context Tree Weighting [5] and the expectimax search is approximated
Equal using ⇢ UCT, a Monte Carlo planning algorithm [6].
This paper extends [4] in several ways.
To move beyond n -Markov environments and make it easy to incorporate background knowledge, we adopt a formal knowledge representation and reasoning framework based on higher-order logic to allow complex features to be deﬁned on non-Markovian, structured environments.
The Feature
Reinforcement Learning framework [7] is adapted to deﬁne an optimisation problem for selecting the best state abstractions and we present an algorithm based on Binary Decision Diagrams to constrain the set of solutions.
Generalizing binarised CTW [8] to incorporate complex features in its context then allows a Bayesian mixture to be computed over a much larger model class.
Combined with ⇢ UCT, the resulting agent is the richest, direct approximation of AIXI known to date.
The practicality of the resultant agent is shown on epidemic control on non-trivial contact networks.
Background and Related Work 2.1 The General Reinforcement Learning Setting
We consider ﬁnite action, observation and reward spaces denoted by A , O , R respectively.
The agent interacts with the environment in cycles: at any time, the agent chooses an action from
A and the environment returns an observation and reward from O and R .
Frequently we will be considering observations and rewards together, and will denote x 2 O ⇥ R as a percept x from the percept space O ⇥ R .
We will denote a string x 1 x 2 . . .
x n of length n by x 1: n and its length n  1 preﬁx as x <n .
An action, observation and reward from the same time step will be denoted aor t .
After interacting for n cycles, the interaction string a 1 o 1 r 1 . . .
a n o n r n (denoted aor 1: n from here on) is generated.
A history h is an element of the history space H :
=
( A ⇥ O ⇥ R ) ⇤ .
An environment ⇢ is a sequence of probability distributions { ⇢ 0 , ⇢ 1 , ⇢ 2 , . . .
} , where ⇢ n : A n !
D (( O ⇥ R ) n ) , that satisﬁes
[Equation]
or 1: n | a 1: n ) .
We will drop the subscript on ⇢ n when the context is clear.
The predictive probability of the next percept given history and a current action is given by ⇢ ( or n | aor <n , a n ) = ⇢ ( or n | h n  1 a n ) := ⇢ ( or 1: n | a 1: n ) ⇢ ( or <n | a <n ) for all aor 1: n such that ⇢ ( or <n | a <n ) > 0 .
The general reinforcement learning problem is for the agent to learn a policy ⇡ : H !
D ( A ) mapping histories to a distribution on possible actions that will allow it to maximise its future expected reward.
In this paper, we consider the future expected reward up to a ﬁnite horizon m 2 N .
Given a history h and a policy ⇡ the value function with respect to the environment ⇢ is given by V ⇡⇢ ( h t ) :
= E ⇡⇢ hP
[Equation]
i , where R i denotes the random variable distributed according to ⇢ for the reward at time i .
The action value function is deﬁned similarly as V ⇡⇢ ( h t , a t +1 ) :
=
E ⇡⇢ hP t + m i =
t +1 R i  h t , a t
+ i .
The agent’s goal is to learn the optimal policy ⇡ ⇤ , which is the policy that results in the value function with the maximum reward for any given history.
2.2 AIXI agent and its Monte-Carlo approximation
The following equation can be shown to be formally equivalent to Equation (1).
At time t , the AIXI agent chooses action a ⇤ t according to
[Equation]
where m 2 N is a ﬁnite lookahead horizon, M U is the set of all enumerable chronological semimeasures [1], ⇢ ( or 1: t + m | a 1: t + m ) is the probability of observing or 1: t + m given the action sequence a 1: t + m , and K ( ⇢ ) denotes the Kolmogorov complexity [9] of ⇢ .
The Bayesian mixture ⇠ U :=
P ⇢
[Equation]
U
[Equation]
K
[Equation]
⇢ ) ⇢
[Equation]
or 1: t | a
[Equation]
: t ) in
[Equation]
2) is a mixture environment model, with 2  K ( ⇢ ) as the prior for ⇢ .
A mixture environment model enjoys the property that it converges rapidly to the true environment if there exists a ‘good’ model in the model class.
Theorem 1.
[1] Let µ be the true environment and ⇠ be the mixture environment model over a model class M .
For all n 2 N and for all a 1: n , n
[Equation]
where D n ( µ || ⇢ ) is the KL divergence of µ ( ·| a 1: n ) and ⇢ ( ·| a 1: n ) .
To see the rapid convergence of ⇠ to µ , take the limit n !
1 on the l.h.s of (3) and observe that in the case where min ⇢ 2 M sup n D n ( µ || ⇢ ) is bounded, the l.h.s.
can only be ﬁnite if ⇠ ( or k | aor <k a k ) converges sufﬁciently fast to µ ( or k | aor <k a k ) .
From Theorem 1, it can be shown that AIXI’s environment mixture model
⇠ U also enjoys this property with any computable environment.
The ﬁrst practical approximation of AIXI is given in [4], in which the expectimax search is approximated using Monte Carlo Tree Search [6] and Bayesian mixture learning is performed using Context
Tree Weighting (CTW) [5].
There is a body of work on extending the Bayesian mixture learning to larger classes of history-based models [10, 11, 12, 13].
In this paper, we generalise the Bayesian mixture learning in an orthogonal direction by adapting binarised CTW [8] to run on sequences deﬁned by logical state abstractions, thus signiﬁcantly improving the ease with which symbolic and statistical background knowledge can be incorporated into the AIXI framework.
2.3 State Abstraction and Feature Reinforcement Learning
In the general reinforcement learning setting, the state abstraction framework is concerned with ﬁnding a function  : H !
S  that maps every history string to an element of an abstract state space S  .
Given a state abstraction  , the abstract environment is a process that generates a state s 0 and a reward r given a history and action according to the distribution ⇢  ( s 0 , r | h, a ) :=
P o :  ( haor )=
s 0 ⇢ ( or | ha ) .
The interaction sequence of the original process converts to a state-action-reward sequence under  .
A key reason for considering abstractions is that the space H is too large and the underlying process may be highly irregular and difﬁcult to learn on.
The aim is thus to select a mapping  such that the induced process can facilitate learning without severely compromising performance.
Theoretical approaches to this question have typically focused on providing conditions that minimise the error in the action-value function between the abstract and original process [14, 15].
The  MDP framework instead provides an optimisation criteria for ranking candidate mappings  based on how well the state-action-reward sequence generated by  can be modelled as an MDP whilst being predictive of the rewards [7].
A good  results in a Markovian model where the state is a sufﬁcient statistic of the history and thus facilitates reinforcement learning.
Optimisation Criteria for Selecting Mappings.
The  MDP framework ranks candidate state abstractions based on an optimisation criteria deﬁned over the code length of the state-reward sequence.
Let CL ( x 1: n ) denote the code length of a sequence x 1: n and CL ( x 1: n | a 1: n ) denote the code length of sequence x 1: n conditioned on a 1: n .
At time n , the Cost M objective is deﬁned as
[Equation]
where  ( h t ) = s t and h t =
aor 1: t .
For more details on how MDP sequences can be coded see [7].
The objective is then to look for  bestn :=
arg min  Cost M ( n,  ) .
Note that CL (  ) is a term that imposes a Occam’s razor-like penalty on the candidate mapping; if different candidate abstractions result in the same Cost M value, the simplest model is preferred.
For the agent to achieve high expected long-term reward, it is imperative that a state abstraction produces a model that can predict the reward sequence well.
However, the equal weighting given to the code lengths of the state sequence in Cost M can be suboptimal; in fact a simple environment can be constructed where Cost M gives unintuitive results (see Appendix F).
Given that CL ( s 1: n | a 1: n ) is optimised independent of any reward signal it does not inherently provide any information about the long-term reward.
Thus, it is best thought of as a regularisation term to make the resulting MDP easier to learn.
We drop the code length of the state sequence from the optimisation criteria to focus purely on reward predictability and hence consider Cost 0 M ( n,  ) =
CL ( r 1: n | s 1: n , a 1: n ) + CL (  ) .
2.4 Knowledge Representation and Reasoning Formalism
The formal agent knowledge representation and reasoning language is the one described in [16, 17].
The syntax of the language are the terms of the  -calculus [18], extended with type polymorphism to increase its expressiveness.
The inference engine has two interacting components: an equationalreasoning engine and a tableau theorem prover [19].
Thus, a predicate evaluation at the internal node of a Context Tree takes the general form  ` p ( h ) , where  is a theory consisting of equations and implications that reﬂects the agent’s current belief base, h is the history of agent-environment interacts, and p is a predicate mapping histories to boolean values.
There is also a predicate rewrite system for deﬁning and enumerating a set of predicates.
Suitable alternatives to our logical formalism are studied in the Statistical Relational Artiﬁcial
Intelligence literature [20, 21], including a few that caters speciﬁcally for relational reinforcement learning [22, 23] and Symbolic MDP/POMDPs [24, 25].
Our choice is informed by the standard arguments given in [26, 27] and the practical convenience of working with (the functional subset of) a language like Python.
Our thesis that formal logic can be used to incorporate domain knowledge and broaden the model class of AIXI approximations remains unchanged with an alternate formal logic.
The  -AIXI-CTW Agent
We now present the  -AIXI-CTW agent.
The actions and rewards are considered in their binary representation.
The agent has access to a large set P of predicates on histories, and each subset of P constitutes a candidate state abstraction.
We show that a  -BCTW data structure performs a Bayesian mixture over all abstractions deﬁned by P and will optimise Cost 0 M .
However, this is computationally infeasible in practice if P is large.
We ﬁrst reduce this computational burden by performing feature selection to shrink the set of predicates for  -BCTW.
Finally, the agent selects actions according to the ⇢ UCT algorithm.
3.1 Shrinking the Model Class via Feature Selection
Our method for selecting candidate state abstractions is centred around the detection of predicate features that result in low entropy conditional reward distributions since lower entropy results in smaller code lengths and lower Cost 0 M .
A simple heuristic that we employ is to look for candidate state abstractions that contain states that maximise the probability of receiving a given reward.
We call such abstractions sharp mappings.
Deﬁnition 1.
Let p  ( r ) :=
[Equation]
a ) and p  ( r c ) :=
P r 0 6 = r 2 R p  ( r 0 ) for all r 2 R .
For a D > p  ( r ) /p
( r c ) , an MDP mapping  is D -sharp if for all r 2 R , there exists s 2 S  , a 2 A such that p  ( r | s,a ) p  ( r c | s,a )  D .
Sharp mappings are deﬁned in this manner to help select candidate state abstractions that are predictive of all rewards.
It is often the case in reinforcement learning environments that informative reward signals are sparse during learning.
Treating rewards as class labels, the problem of predicting rewards can be viewed as an unbalanced data classiﬁcation task.
If the unbalanced nature of the data is not taken into account, feature selection methods will tend to ignore features that may be predictive of rewards that occur infrequently but are important for performance.
Thus, we adjust for this fact by ensuring that the threshold chosen for each reward is bounded from below by p  ( r ) p  ( r c ) ; if a reward r occurs infrequently relative to the frequency of all other rewards, then the lower bound on D will be small.
Our approach is to select predicate features that lead to sharp mappings.
Feature Selection Using Decision Boundaries.
Using the sharpness threshold, we frame the problem as a binary classiﬁcation task.
For each reward r 2 R , the sharpness of the reward distribution at a state-action pair determines the binary class by a decision rule deﬁned by
F r  ,D ( s, a ) :
=
I
[Equation]
i with indicator function I .
Feature selection on the binary classiﬁcation task can then be performed by eliminating conditionally redundant features, deﬁned next.
Deﬁnition 2.
Let  1 = { p 1 , . .
p n } and  2 =  1 \ { p j } for j 2 { 1 , . .
. , n } where p i is a predicate function.
If F r
[Equation]
D ( s a for all
s 2 S  and
{+s 
+}
= s 1: j  1 s j +1: n , then p j is
a conditionally redundant feature.
A conditionally redundant feature does not affect the output of the decision rule given the feature ordering and thus does not contribute to discriminating between different classes.
As this property only depends upon the feature ordering upon construction, removing a conditionally redundant feature does not change whether the remaining features are redundant or not.
To select candidate predicate features, we set up a decision rule for each reward r 2 R and eliminate the predicate features that are redundant across all rewards.
In practice, the probabilities used by the decision rules are estimated as frequency counts.
To remove conditionally redundant features, we use a Binary Decision Diagram (BDD) data structure [28] to represent each decision rule.
A reduction procedure on the BDD representation then guarantees that only informative features remain.
Theorem 2.
Let F r  ,D : S  ⇥ A ! { 0 , 1 } be the decision rule for reward r 2 R .
Then the reduced
BDD representation of F r  ,D contains only conditionally informative features.
The proof is given in Appendix E.
In practice there exists simple procedures to reduce a BDD.
For each reward, we determine a decision rule and perform BDD reduction to get a set of informative predicates.
The union of the sets of selected predicates is then chosen.
One issue with BDD reduction is that the given variable ordering can impact whether a feature is conditionally redundant or not; a sub-optimal variable ordering will keep more features than necessary.
We alleviate this issue by performing BDD reduction on multiple randomly selected subspaces of the feature space, corresponding to randomly selected subsets of predicates.
We name our approach Random Forest-BDD (RF-BDD).
For each selected subset of predicates P i , let  i be the corresponding MDP mapping.
We perform BDD reduction on the decision rule constructed from P i given by F r P i ( s, a ) :
=
I
[Equation]
i .
Once BDD reduction has been performed for all chosen subsets, a choice has to be made on which predicates to keep.
We employ a voting scheme where a predicate receives a vote if it is kept by a BDD reduction.
The votes are then normalised by the number of times each predicate was selected.
The predicates that have a retention rate above a certain threshold will then be kept.
The kept predicates for each percept are then combined into a single set.
We emphasise that selecting features based on a sharpness threshold does not directly optimise the
Cost 0 M criteria but rather constrains the set of possible solutions by their reward predictability.
Thus the set of predicates selected by the RF-BDD procedure above, when input into  -BCTW, lead to a model that approximately minimises Cost
M over the initial set of predicates.
3.2  -Binarized Context Tree Weighting
The binarized context-tree weighting (BCTW) algorithm is a universal data compression scheme that extends the context-tree weighting algorithm to non-binary alphabets.
The context-tree weighting algorithm is a data structure that computes an online Bayesian mixture model over prediction sufﬁx trees [5].
The BCTW approach is to binarize a target symbol and predict each bit using a separate context tree.
We present the  -BCTW algorithm which modiﬁes BCTW by allowing nodes to correspond to arbitrary predicates on histories.
We show that this extension allows  -BCTW to compute a Bayesian mixture model over environments that can be modelled by a set of predicate functions and will optimise Cost 0 M .
The  -BCTW algorithm modiﬁes the BCTW algorithm to allow it to model the state and reward transitions of an abstract environment deﬁned under a set of predicate functions.
Suppose we are given a set of predicates that deﬁnes the state space.
The  -BCTW models the abstract environment by assuming it has MDP structure and thus conditions on an initial context given by the previous state and action.
Suppose the length of a state and action symbol together is d and let [ x ] denote the binary representation of a symbol x and [ x ] i , [ x ] <i denote the i -th bit and the ﬁrst i  1 bits of x respectively.
Given the sequence sra 1: n , the distribution of the ﬁrst bit is computed as a Bayesian mixture model by a depth d context tree:
[Equation]
Here T ([ s n  1 a n ]) denotes the leaf node in the PST T that is reached by following the sufﬁx of [ s n  1 a n ] .
Limiting the context to the previous state-action symbols encodes the Markov assumption into the modelling of the state-reward transition.
The distribution ˆ P ([ sr n ] 1 | T ([ s n  1 a n ]) , sr 1: n  1 ) of [ sr n ] 1 under the model T is provided by a KT estimator [29] at T ([ s n  1 a n ]) that is computed by accumulating the frequency of bits following [ s n  1 a n ] from the history sr 1: n .
The function  measures the complexity of a model via a preﬁx coding and the weighting term 2   ( T ) provides an
Occam-like penalty on the model T .
For more details on the prior weighting see [4].
For the l -th bit, the distribution conditions on the previous state-action as well as the previous l  1 bits of sr n : ˆ P ( [Equation] ([ sr
[Equation]
l | [ sr n ] <
[Equation]
, asr [Equation] [Equation] The prediction for sr n using  -BCTW is then given by composing the predictions for each bit:
[Equation]
Note that the conditional reward distribution ˆ P ( r n | sra 1: n  1 , s n ) is modelled by the last | [ r n ] | context trees.
The following result states that  -BCTW computes a mixture environment model.
Proposition 1.
Let ˆ P
[Equation]
sr
[Equation]
and
[Equation]
the form
[Equation]
The following guarantees fast reward distribution convergence for mixture environment models.
Theorem 3.
Let µ be the true environment and ⇠ be the mixture environment model over a model class M .
For all n 2 N , a 1: n , o 1: n n
[Equation]
where D 1: n [Equation] µ [Equation]
:=
[Equation]
The proofs of Proposition 1 and Theorem 3 can be found in Appendix A.
As a mixture environment model,  -BCTW’s conditional distribution over reward sequences is guaranteed to converge quickly by Theorem 3 provided the true environment can be modelled by the provided predicates.
Combined with  -BCTW’s
Occam-like model size penalty,  -BCTW can be viewed as optimising Cost
M over the set of abstractions deﬁned by the given predicates.
3.3 Expectimax Approximation With ⇢ UCT
We use Monte-Carlo Tree Search to perform an approximation of the ﬁnite horizon expectimax operation in AIXI.
We employ the ⇢ UCT algorithm
[4] where actions are selected according to the UCB policy criteria.
Unlike ⇢ UCT however, we are able to store search nodes of the tree and maintain value estimates for the duration of the agent’s life-span.
This is due to the feature selection and modelling by  -BCTW reducing the environment model to an MDP with a ‘small’ state space.
Thus, the number of samples through the search tree that need to be generated per step is greatly reduced.
Experiments
We evaluate our  -AIXI-CTW agent’s performance across a number of different domains.
We begin by evaluating our agent’s performance on four simpler domains before considering the epidemic control problem.
All experiments were performed on a 12-Core AMD Ryzen Threadripper 1920x processor and 32 gigabytes of memory.
4.1 Simple Domains
We consider four simple domains: Biased rock paper scissors (RPS) [30], Taxi [31], Jackpot and
Stop Heist.
A full description of each of the environments is provided in Appendix C.1.
As baseline comparison methods we compare against two decision-tree based, iterative state abstraction methods using splitting criteria as deﬁned in U-Tree [32] and PARSS-DT [33].
In U-Tree, an existing node (representing a state) is split if splitting results in a statistically signiﬁcant difference in the resulting
Q-values as computed by a Kolmogorov-Smirnov test.
In PARSS-DT, nodes are split if the resulting value functions are sufﬁciently far apart.
Model learning is simply done by frequency estimation.
To ensure our method does not have an informational advantage, both baseline methods are given the same initial set of predicate functions to consider node splits from.
In both baselines, leaf nodes were tested for splits every 100 steps.
From Figure 1, it can be seen that our  -AIXI-CTW agent either outperforms or performs on par with the two baseline decision tree methods.
U-Tree was able to perform well in the biased RPS and Stop
Heist domains.
The PARSS based method was able to perform well on the Stop Heist domain and was displaying evidence of learning, albeit slowly, on the biased RPS domain.
U-Tree and PARSS were both unable to perform well in the Jackpot and Taxi environments.
One possible issue is that
Jackpot and Taxi require more predicates than biased RPS and Stop Heist to be able to perform well.
Also, both methods can be susceptible to producing spurious splits.
This can make the state space too large and the resulting transition model can become too difﬁcult to learn with.
4.2 Epidemic Control over Contact Networks
We now evaluate the agent’s performance on the novel task of learning to control an epidemic process over large-scale contact networks, a highly structured environment that exhibits an exponential state space, partial observability and history dependency.
Epidemic Processes on Contact Networks.
In this paper epidemic processes are modelled on contact networks [34, 35, 36].
A contact network is an undirected graph where the nodes represent individuals and the edges represent interactions between individuals.
Each individual node in the graph is labelled by one of four states corresponding to their infection status: Susceptible ( S ), Exposed ( E ), Infectious ( I ), and Recovered ( R ).
Every node also maintains an immunity level ! .
The model evolves as a
POMDP.
At time t , the temporal graph is given by G t =
( V, E t ) with V as the set of nodes and E t as the set of edges at time t .
A function ⇣ t : V ! { S , E , I , R } ⇥ { 1 , ⌘ 1 , ⌘ 2 } maps each node to its label and one of three immunity levels where ⌘ 1 , ⌘ 2 2 R + .
Together, ( G t , ⇣ t ) constitute the state.
At time t , a Susceptible node with k t Infectious neighbours becomes Exposed with probability 1  (1   ) kt ! , where  2 [0 , 1] is the rate of transmission.
An Exposed node becomes Infectious at rate  .
Similarly, an Infectious node becomes Recovered at a rate  and becomes Susceptible again at a rate ⇢ .
At every time step, the process emits an observation on each node from { + ,  , ? } corresponding to whether a node tests positive, negative or is unknown/untested.
The observation model is parametrised such that positive tests are more likely if the underlying node is Infectious, corresponding to the realistic scenario where sick individuals are more likely to test and also test positive.
A more detailed exposition on the exact transition and observation models is given in Appendix B.
The agent can perform an action at each time step.
We consider a set of 11 possible actions { DoNothing , Vaccinate ( i, j ) , Quarantine ( i ) } , where i 2 [0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0] and j =
i +0 .
A Vaccinate ( i, j ) action increases the immunity level of the top i th to j th percent of nodes as ranked by betweenness centrality by one level, up to the maximum value.
Note that a conferred immunity level lasts for the entire length of an episode unless increased.
A Quarantine ( i ) action quarantines the top i th percent of nodes by removing all edges incident on those nodes for one time step.
The reward function is parametrised to evaluate the agent’s ability to balance the cost of its actions against the cost of the epidemic.
At each time step, the instantaneous reward is given by
[Equation]
where  2 R + , Positives ( o t ) counts the number of positive tests in the observation o t and
Action _ Cost ( a t  1 ) is a function determining the cost of each action.
Thus changing  changes the ratio of contributions to the reward received at each time step.
If the agent successfully terminates the epidemic, i.e. there are no more Exposed or Infectious nodes, the agent receives a positive reward of 2 per node.
During a run, it is possible the environment never reaches an absorbing state.
If the agent has not successfully stopped the epidemic process after 1000 steps, we terminate the episode.
Epidemic control is a topical subject given the prevalance of COVID-19 and recent approaches to this question vary wildly in terms of the how the problem is modelled.
Most approaches vary in the different ways they model the dynamics of an epidemic and the action space considered is usually fairly small [37, 38, 39, 40, 41, 42, 43].
Ultimately, the differences in the modelling of epidemic processes makes comparison between results difﬁcult.
With no consensus on the appropriate model to use, our model is chosen as it is sufﬁciently complex to demonstrate the efﬁcacy of our agent.
Experimental Setup.
We use an email network dataset as the underlying contact network, licensed under a Creative Commons Attribution-ShareAlike License, containing 1133 nodes and 5451 edges [44, 45].
The transition model, observation model, Action _ Cost ( a t ) are parametrised the same way across all experiments (see Table 1 in Appendix B).
A Quarantine ( i ) action imparts a cost of 1 per node that is quarantined at the given time step.
A Vaccinate ( i, j ) action imparts a lower cost of 0 .
5 per node.
The parameters  , ⌘ 1 , ⌘ 2 are varied across experiments.
We generate a set ofpredicate functions consisting of predicates on top of functions such as the percentage each action was selected in the last N steps, the observed infection rate on different subsets of nodes, and also the infection rate on different subset of nodes as computed by particle ﬁltering.
A full description of how the predicates were generated is given in Appendix D.
The  -AIXI-CTW agent is trained in an online fashion.
The agent explores with probability ✏↵ t at each step t until ✏↵ t < 0 .
03 , where the agent performs in an ✏ -greedy way with exploration rate 0 .
RF-BDD was performed with a threshold value of 0 .
9 across all rewards.
4.2.1 Results
Selected Features.
The set of predicates chosen in each experiment by RF-BDD are described in Appendix D. In general, RF-BDD tended to select predicates representing bits in the binary representation of the observed infection rate and the rate of change of the observed infection rate on different subsets of the betweenness centrality nodes.
A random selection of different action sequence indicator functions were also kept.
Note that predicates on the infection rate as computed by particle ﬁltering were not selected and this can be attributed to the particle degeneracy issue that occurs in high dimensional problems [46].
These choices by RF-BDD highlight its ability to select a small set of predicates that provided useful information for epidemic control.
The inclusion of a random selection of different action sequence indicator functions also demonstrates that it is not perfect.
A more thorough investigation is the topic of future work.
Comparison to Baseline Methods.
Both U-Tree and PARSS were used as baseline methods in the same conﬁguration as in the simpler domains.
From Figure 2,  -AIXI-CTW outperforms the two baseline methods and both U-Tree and PARSS were unable to improve.
The epidemic problem as formulated is likely difﬁcult for both U-Tree and PARSS as the number of predicates required to perform well is fairly large.
Also, both methods can be susceptible to producing spurious splits.
This can make the resulting state space and transition model too large and difﬁcult to learn with.
RF-BDD vs. Random Forest Feature Selection.
We also compare the performance of our agent under RF-BDD or Random Forest feature selection.
Other feature selection methods such as forward selection methods [47] were also considered but were found to offer no advantage over Random
Forest.
For each experiment, we plot four graphs.
The ﬁrst graph plots the learning curve computed as a moving average over 5000 steps of the reward per cycle.
The remaining graphs provide insight into the learnt policy.
The second graph plots the percentage each type of action is performed over a moving average of 500 steps as well as a moving average of the infection rate over the same window. [Figure]
[Figure]
The remaining two graphs provide an example of the agent’s behaviour and its impact on the infection rate over an episode collected toward the end of training.
Figure 3 depicts the  -AIXI-CTW agent’s performance with RF-BDD feature selection over two experiments with differing vaccination efﬁcacy and an infection rate weighting of  = 10 .
RF-BDD was run with 500 trees generated using random samples of 8 features and used a threshold value of 0.9.
In both experiments, the agent learns to be more aggressive in terminating the epidemic.
In
Experiment 1 the agent’s behaviour is to quarantine almost always, and occasionally vaccinate.
This is reﬂected in the plots of the agent’s behaviour in an episode toward the end of training.
Quarantine actions are performed the entire time except once, resulting in a fast termination of the epidemic process.
In contrast, Experiment 2 shows that the agent prefers to vaccinate more often.
This preference can be attributed to the added effectiveness that a stronger vaccination action imparts to lowering the infection rate.
When analysing the agent’s behaviour over an episode, it is clear that the agent chooses to vaccinate almost always whilst only quarantining at timely moments.
This results in the agent also learning to terminate the episode early, albeit not as quickly.
In comparison to
Experiment 1, the average reward per cycle is also higher at the end of learning.
Figure 4 depicts the  -AIXI-CTW agent’s performance with Random Forest feature selection.
Random Forest was initialized with 500 trees and splits were chosen using either the gini impurity or entropy and the top thirty highest weighting features were chosen.
The resulting predicate sets were very similar for the two criteria and so we only present results for the entropy criteria case.
When comparing the agent’s performance when using RF-BDD (Figure 3) versus Random Forest (Figure 4), it is quite clear that the agent performs better with the predicates chosen using RF-BDD.
As can be seen from Figure 4a) and 4b), the agent’s learning curve when using the predicates chosen by
Random Forest did not improve in either case and actually decreased.
The remaining three plots also show that there is no clear discernible pattern of behaviour learnt by the agent.
This suggests that the features chosen by Random Forest were not informative enough to allow the agent to learn to perform well.
The inability of traditional feature selection methods like Random Forest to pick out useful features for the agent was a strong motivation for constructing an alternative in RF-BDD. [Figure]
Overall, the results demonstrate that the  -AIXI-CTW agent is able to learn plausible behaviours under the different environment settings.
This demonstrates the potential of our approach in reducing a complex environment to a simpler problem whilst still maintaining performance.
Nevertheless, it is also quite clear that the agent did not learn optimal behaviour except potentially in the case of
Experiment 1.
In Experiment 2, the agent did not learn that vaccinating a node more than twice does not further reduce the cost received.
However, this is not surprising as this is a difﬁcult causal relationship to learn from data alone.
Furthermore, the full set of generated predicates do not contain enough information to isolate this particular history dependency.
In fact, the predicates selected generally capture different statistics about the infection rate at the current time step.
If the value estimate of vaccinating in a highly visited state is high due to it lowering future received costs, the agent will learn to perform the vaccination action every time the state is encountered, even if it has done so many times before.
We expect that importing better domain knowledge into the original set of predicates would solve this issue for the  -AIXI-CTW agent.
Conclusion
We have introduced the  -AIXI-CTW agent model, an agent that expands the model class that AIXI may be approximated over to structured and non-Markovian domains via logical state abstractions.
We show that our approach is competitive on simple domains and is sufﬁciently powerful to handle control of epidemic processes on contact networks, a class of problems with a highly structured exponential state space, partial observability and history dependency.
In doing so, our result provides another piece of strong evidence that AIXI is not just a mathematically interesting theory, but can eventually lead to practically useful agents.
Whilst RF-BDD has demonstrated its usefulness empirically in our experiments, a better understanding of how it works theoretically and how it behaves in other contexts is important.
In [48] the authors identify the epidemic control problem within a class of problems they call the control of diffusion processes over networks through nodal interventions.
The epidemic control problem constitutes an instance of this class of problems and it would be interesting to extend our agent setup to other problems within this domain.
A key concern for the real-world application of reinforcement learning on contact networks for control is the preservation of privacy for individual nodes.
Our agent setup does not restrict the usage of any form of domain knowledge on the complete graph such as an individual node’s contact network and extended neighbourhood.
Investigating the performance of our agent under privacy preserving constraints such as differential privacy [49] is an interesting topic for future research.