V ARIATIONAL O RACLE G UIDING FOR R EINFORCEMENT L EARNING
A BSTRACT
How to make intelligent decisions is a central problem in machine learning and artiﬁcial intelligence.
Despite recent successes of deep reinforcement learning (RL) in various decision making problems, an important but under-explored aspect is how to leverage oracle observation (the information that is invisible during online decision making, but is available during ofﬂine training) to facilitate learning.
For example, human experts will look at the replay after a Poker game, in which they can check the opponents’ hands to improve their estimation of the opponents’ hands from the visible information during playing.
In this work, we study such problems based on Bayesian theory and derive an objective to leverage oracle observation in RL using variational methods.
Our key contribution is to propose a general learning framework referred to as variational latent oracle guiding (VLOG) for DRL.
VLOG is featured with preferable properties such as its robust and promising performance and its versatility to incorporate with any value-based
DRL algorithm.
We empirically demonstrate the effectiveness of VLOG in online and ofﬂine RL domains with tasks ranging from video games to a challenging tilebased game Mahjong.
Furthermore, we publish the Mahjong environment and an ofﬂine RL dataset as a benchmark to facilitate future research on oracle guiding 1 .
I NTRODUCTION
Deep reinforcement learning (DRL) has undergone rapid development in recent year (Sutton &
Barto, 2018; Mnih et al., 2015; Vinyals et al., 2019).
However, there is an common and important but under-explored aspect in RL: imagine that after playing a Poker game, a human player may look at the replay to check opponents’ hands and analyze this information to improve his/her playing strategy (or policy ) for the next time.
We refer to the information like opponents’ hands as oracle observation , deﬁned as the information invisible to the agent during online task execution but available during ofﬂine training.
By contrast, the information available during task execution is called executor observation .
Such a scenario has been refered to as oracle guiding for RL (Li et al., 2020; Fang et al., 2021)
(see Sec.
3 for a formal deﬁnition).
Oracle guiding is common in real life.
For example, when taking an examination (the oracle observations are the answers to similar questions, which are available only during preparing); and training a robot to perform some tasks on the Moon (when training the robot, we can provide it with the information of the territory, which are not available during execution).
The type of oracle observations can be diverse, including hindsight information (Harutyunyan et al., 2019;
Guez et al., 2020), human feedback (Knox & Stone, 2009;
Loftin et al., 2016; MacGlashan et al., 2017), re-calibrated data with post-processing, and hidden states in a partially observed setting (Li et al., 2020).
While humans naturally perform oracle guiding when learning to make decisions, it remains challenging in RL.
The difﬁculties include: (1) how to guarantee that learning with oracle observation improves the main decision model using executor observation only, and (2) if introducing an auxiliary loss leveraging oracle observation, how to tradeoff between the main loss and auxiliary loss.
While recent studies attempted to model oracle guiding in RL (Guez et al., 2020; Li et al., 2020;
Fang et al., 2021), none of them addressed these difﬁculties (refer to the Related Work section for more details).
In particular, all these proposed methods are heuristic: although empirical results showed performance gain with oracle guiding, it is not theoretically guaranteed that the usage of oracle observation improves execution performance.
In this paper, we propose a fundamentally new idea for oracle guiding based on Bayesian theory.
Taking Poker as an example, we know that the learning of optimal strategy is tractable if knowing the global, true state of the environment (or simply state 2 ), including all visible or invisible cards, the opponents’ playing style, etc.
(Azar et al., 2017; Jin et al., 2018; 2020).
A key part of skill improvement is learning to estimate the probabilistic distribution of environmental state from executor observation.
The common way to do this by human experts is to watch match replays where the oracle observation (e.g. opponents’ hands) is available, and then use the oracle-estimated state to correct the executor-estimated state.
We interpret this to Bayesian language: executor-estimated state as the prior distribution, and the oracle-estimated one as the posterior distribution.
Thus, the training objective can be considered two-fold: learning to make decisions based on posterior estimation of state, and learning a prior distribution of state closer to the posterior one.
We formulate this idea by proposing a novel learning framework for general oracle guiding problems based on variational Bayes (VB) (Kingma & Welling, 2014), referred to as variational latent oracle guiding (VLOG).
VLOG owns several preferable properties.
First, VLOG is theoretically guaranteed to leverage oracle observation for improving the decision model using executor observation.
Second, VLOG is a versatile DRL framework that can be integrated into any value-based
RL algorithm and is agnostic to the type of oracle observation.
Third, VLOG does not suffer from the necessity of tuning additional hyper-parameters.
Finally, we empirically show that VLOG contributes to better performance in a variety of decision-making tasks in both online and ofﬂine RL domains.
The tasks include a simple maze navigation, video games playing, and a particularly challenging tile-based game Mahjong in which humans heavily leverage oracle observations in learning (Li et al., 2020).
We also contribute to the community by taking Mahjong as a benchmarking task for oracle guiding and publish the RL environment and dataset to facilitate future research.
R ELATED WORK
In the past few years, research interests have grown on DRL and imitating learning (Chen et al., 2020) with leveraging oracle or hindsight information.
For DRL, Guez et al.
; Fang et al.
(2021) considered hindsight observation (executor observation at future steps) as the oracle observation during training.
Guez et al.
(2020) used hindsight observation to facilitate learning a representation of current state.
Another method (Fang et al., 2021) was used for stock trading: the authors trained a teacher (oracle) policy with hindsight information, and employed network distillation to make the student policy behaves more similar to the teacher policy.
Both the methods (Guez et al., 2020; Fang et al., 2021) are heuristic and focused on making use of future observation for a better sequential modeling, while VLOG is theoretically guaranteed for any kind of oracle observation.
For applications in imperfect-information games, Suphx (Li et al., 2020), a DRL-based AI for
Mahjong, also introduced a method to leverage oracle observation (opponents’ hand) for stronger performance.
They concatenate oracle observation with executor observation as the input of policy network, where the oracle observation is timed by a scalar variable which is annealed from 1 toduring the training course.
However, the method used in Li et al.
(2020) is also heuristic and has only been tested in one task.
Variational Bayes (VB) is a well-established method and has been taken advantage of in RL.
For example, control as probabilistic inference uses VB to connect the objective function of RL and variational lower bound of a probabilistic inference problem (Furmston & Barber, 2010; Weber et al., 2015; Levine, 2018).
Our idea differs since it frames a value regression problem by the maximumlikelihood problem, and then, it applies VB to solve it (see Sec. 4).
Also, the usage of VBian network models for DRL has captured attention from researchers recently.
For example, Ha & Schmidhuber (2018) proposed to employ a VAE to reduce the high dimensionality of image observation; Igl et al.
; Han et al.
; Lee et al.
(2020) proposed variational RNNs as the state-transition models to encode the belief states of the agent; Yin et al.
(2021) utilized a variational sequential generative model for predicting future observation and used the prediction error to infer intrinsic reward to encourage exploration; and Okada et al.
(2020) demonstrated performance gain by using a deep
Bayesian planning model in continuous control tasks.
Our study differs from the mentioned works by focusing on oracle guiding, and VLOG does not involve learning a state transition model.
O RACLE GUIDING POMDP
Here we deﬁne the problem based on the Partially Observable Markov decision process (POMDP) (Sondik, 1978).
An oracle guiding POMDP distinguishes from the original POMDP by having two types of observations: executor and oracle.
The executor observation x is always available to the agent, on which the agent’s decision making (execution) relies.
The oracle observation ˆ x is not accessible during execution, but can be obtained afterward.
x is included in ˆ x since the former is always available.
Thus ˆ x contains no less information of the underlying environment state than x .
A formal deﬁnition of oracle guiding POMDP is a tuple (cid:104) S , A , P 0 , T , X , ˆ X , O , ˆ O , γ (cid:105) , where S and A are the state and action spaces, respectively.
P 0 speciﬁes the initial state distribution such that P 0 ( s ) is the probability of a state s ∈ S being an initial state.
T speciﬁes the state transition probability such that T ( s (cid:48) , r | s , a ) is the probability of reaching to a new state s (cid:48) ∈ S with an immediate reward r ∈ R after taking an action a ∈ A at a state s ∈ S .
X denotes the executor observation space and ˆ X denotes the oracle observation space.
O speciﬁes the executor observation probability such that O ( x | s ) is the probability of an executor observation x ∈ X at a state s ∈ S .
Similarly, ˆ O speciﬁes the oracle observation probability such that ˆ O ( x | s ) is the probability of an oracle observation ˆ x ∈ ˆ X at a state s ∈ S .
γ ∈ [0 , 1) is the discount factor.
Value functions are the expected value of return from a particular state or state-action pair.
For a policy π , its Q-function is deﬁned by q π ( s , a ) :
=
E π [ (cid:80) ∞ n =0 γ n r t + n | s t = s , a t = a ] , where E π indicates the expectation when the policy π is followed.
The state value function v π is deﬁned by v π ( s ) :
= E π ( a | s ) [ q π ( s , a )] .
The agent aims to maximize E P 0 ( s ) v π ( s ) with respect to π , and the value-functions play an important role to this end (Sutton & Barto, 2018).
VLOG: VARATIONAL LATENT ORACLE GUIDING
Let us introduce a latent vector z t , which is a probabilistic variable representing the environmental state s t .
From a Bayesian perspective, we consider the prior distribution p ( z t | x t ) as the agent’s estimated probability density function (PDF) for z t based on executor observation x t .
Meanwhile, the posterior distribution PDF q ( z t | ˆ x t ) is modeled based on oracle observation ˆ x t .
In RL, the most basic requirement is to make a good estimation of return by a value function approximator v ( x t ) :
= (cid:82) v ( z t ) p ( z t | x t ) d z t (we denote it by v , but it can also be a Q function by simply replacing x t with ( x t , a t ) ) based on available information, i.e. executor observation x t .
The target of return, denoted by v tart , can be estimated by any value learning algorithm such as TD(0)
(Sutton & Barto, 2018), Peng’s Q( λ ) (Kozuno et al., 2021), etc.
(generally, v tar can always be given by Bellman equations.
However, one can also use Monte-Carlo return as v tar if available).
In particular, we employed double Q-learning with dueling architecture (Wang et al., 2016) to compute v tar due to its effectiveness and simplicity (Sec. 5 and B.1).
We want to maximize the log-likelihood objective of the estimation of return based on executor observation x t (i.e., for the executor model)
[Equation]
By Jensen’s inequality, we have
[Equation]
Thus we can maximize our initial objective L via maximizing L VLOG , which is also known as the variational lower bound (Kingma & Welling, 2014), but in our oracle-guiding scheme.
Since p ( z t | x t ) and q ( z t | ˆ x t ) represents the PDF of latent vector obtained from executor observation and oracle observation, respectively, the meanings of the two terms in L VLOG appear clear now: the ﬁrst term, i.e., oracle prediction error, helps to improve value estimation from posterior latent state distribution ( z t computed with oracle observation); and the second term, i.e., regularization term, helps to shape the prior representation of z t closer to the posterior one as latent oracle guiding.
We would like to highlight that the VLOG objective is the lower bound of the objective for the prior executor model v ( x t ) (the estimation of return using executor observation x t ) with the usage of oracle observation ˆ x t .
This lower bound guarantees that the usage of oracle observation is positively affecting the learning of executor model, which is what we ﬁnally want.
Now we have derived the theoretical objective of VLOG, the next section will explain its implementation in practice.
Remark 1.
One may use any shape of the approximate posterior q , depending on which different instance of VLOG is possible.
Furthermore, one may directly use v ( x t ) instead of p ( z t | x t ) .
These design choices allow users to incorporate any prior knowledge on oracle observation.
For example, if one knows the range of a state-value at x t is a closed interval [ l, u ] , the approximate posterior q ( v t | ˆ x t ) can be restricted to a family of probability distributions supported on [ l, u ] . observation observation ˆ x t use two
4. I MPLEMENTATION WITH NEURAL NETWORKS
Inspired by the implementation of variational auto-encoder (VAE, Kingma & Welling (2014)), we propose the neural network architecture of VLOG (Fig. 1).
The executor observation x t and oracle observation ˆ x t are processed by two distinct encoder networks and to compute the prior and posterior distribution of the latent vector, respectively.
During training, both x t and ˆ x t are available, and all the network parameters are updated by maximizing the VLOG objective in an end-to-end manner (Fig.
1A).
During execution, the agent computes prior distribution p ( z | x t ) for decision making (Fig. 1B) without using oracle observation.
z t is computed by parameterized normal distribution p ( z t | x t ) =
N ( µ pt , exp (log σ pt )) , ( µ pt , log σ pt ) =
prior encoder ( x t ) , [Equation]
posterior encoder [Equation]
For computing P ( v ( z t ) = v tart | z t ) in Eq.
1, we simply assume it follows normal distribution, and estimate it with mean square error between v ( z t ) and v tart in practice.
The reparameterization trick is used to perform end-to-end training as in VAE (Kingma & Welling, 2014).
Then the output of the decoder (value function) can be obtained by v ( z t ) =
decoder ( z t ) .
Note that z t is obtained using the posterior encoder during training, and using the prior encoder during execution.
4. T ASK - AGNOSTIC LATENT BOTTLENECK CONTROL
To learn a better representation, we borrow the idea from β -VAE (Higgins et al., 2016) to apply a coefﬁcient β to the regularization term.
Thus we have the loss function (negative lower bound) as
[Equation]
The hyper-parameter β controls the capacity of the latent information bottleneck (Tishby & Zaslavsky, 2015;
Alemi et al., 2017).
We found the choice of β is important for the performance of
VLOG in RL (see Appendix B.3).
However, having extra hyper-parameters are not desired.
Inspired by the method used in Burgess et al.
(2017) for controlling the scale of KL divergence in β -VAE, we propose a task-agnostic method to automatically adjust β by setting a target of KL divergence
D tarKL .
In particular, we minimize the auxiliary loss function ( β as the being optimized parameter)
[Equation]
The intuition here is to strengthen the regularization by increasing β when the divergence between prior and posterior is too large, and vice versa.
This method is similar to that used in soft actorcritic for automatically adjusting the entropy coefﬁcient (Haarnoja et al., 2019), but we used for KL divergence coefﬁcient.
Importantly, we found a well-performing value D tarKL = 50 agnostic to other design choices.
It worked well in a range of different tasks and networks (Sec. 5).
Therefore, we do not need to tune β .
We provide more discussion about this method in Appendix D.
E XPERIMENTS
How does VLOG perform in practice?
We investigated the empirical performance of VLOG in three types of tasks using online or ofﬂine RL, from simple to difﬁcult.
In the following experiments, we used double DQN with dueling network architecture (van Hasselt et al., 2016;
Wang et al., 2016) as the base RL algorithm, the model and loss functions for RL are deﬁned in Appendix.
B.1.
As DRL is susceptible to the choice of hyper-parameters, introducing any new hyper-parameters might obscure the effect of oracle guiding.
Double DQN and dueling architecture are preferable for the base algorithm since they require no additional hyper-parameters, in contrast to other DQN variants (Hessel et al., 2018), such as prioritized experience replay (Schaul et al., 2016), noisy network (Fortunato et al., 2018), categorical DQN (Bellemare et al., 2017), and distributed RL (Kapturowski et al., 2018).
Importantly, we used the same hyper-parameter setting for all methods and environments as much as possible (see Appendix B.2).
5. M AZE
We ﬁrst demonstrate how VLOG helps to shape the latent representation by leveraging oracle observations in learning.
The testbed is a maze navigation task 3 (Fig.
2A) with 10 × 10 grids.
The executor observation is the ( x , y ) position, where x , y are continuous values randomly sampled within each grid of the maze (thus the observations in two adjacent but wall-separated grids may be very close).
At each step, the agent selects an action (going up, down, right, or left) and moves to another grid if not blocked by wall.
We provided the VLOG agent with oracle observation ( x c , y c , d g ) during training, where x c , y c are coordinates of the center of the current grid, and d g is the (shortest) path distance to goal from the current grid.
It is intuitive that although the raw observation is ( x , y ), d g matters more in a maze navigation task.
We empirically investigated how much such oracle observation used in learning of VLOG could help to shape the latent representation of z w.r.t.
d g rather than position.
The encoder and decoder were both 2-layers multi-layer perceptrons (MLP) with width 256 and ReLU activation.
The size of latent vector z t for VLOG was 128 since we computed both µ and σ (Appendix C).
Experiments show that the baseline agent struggled to reach the goal (Fig. 2B), while VLOG agents stably solved the task after learning.
To check how the usage of VLOG affected the learned latent representation, we visualize the latent representation of both VLOG and baseline models with principal component analysis (PCA, Pearson F.R.S. (1901)).
In Fig.
2C, we map path distance to goal d g to color and plot the scores of the ﬁrst 2 PCs of z (computed using executor observation) for VLOG and the corresponding hidden state for baseline using successful trials (“hidden layer” in Fig.
1C).
The latent state of VLOG showed a relatively smoother and more interpretable representation of distance-to-goal comparing to that of baseline.
We then plot the latent representations for different positions in the maze in Fig.
2D.
The latent state of VLOG more clearly represented d g , consistent with the result (Fig.
2C).
In particular, we looked into a rectangle region (denoted by rectangles in Fig.
2D) inside which the left 2 grids and right 2 grids are segregated by a wall.
We found the corresponding areas in the latent PC space and circled them in Fig.
2C.
While these 4 grids are close in ( x , y ) (executor observation), their distances-to-goal (oracle observation) are highly distinguished.
By leveraging oracle guiding using VLOG, the agents can clearly differentiate the left 2 grids and the right 2 grids in latent space as shown in Fig.
2C, D, left (note that the latent state z here of VLOG was computed using executor observation only).
By contrast, the latent representations of these grids were overlapped for the baseline model, which did not utilize the oracle observation (Fig.
2C, D, right).
In sum, we demonstrated with a toy example that VLOG effectively helped the latent space to couple with oracle state useful for the task.
The following sections will transfer to experiments on more complicated tasks and discuss how VLOG can improve practical performance.
5. N OISY M IN A TAR
To evaluate how VLOG scales with more high-dimensional state space, we tested it on a set of
MinAtar videos games.
MinAtar (Young & Tian, 2019) is a test platform for AI agents, which implements 5 miniaturized Atari 2600 games with discrete actions (Seaquest, Breakout, Space Invaders, Freeway and Asterix).
MinAtar is inspired by Arcade Learning Environment (Bellemare et al., 2013) but simpliﬁes the environments for efﬁciency.
The observations are 10 × 10 pixels with multiple channels indicating different objects.
In real-world, the observation usually contains some noise.
Thus it is natural to consider the noisy observation as the partially-observable executor observation, and the original, non-noisy observation as the oracle one.
Suppose that at each frame, each pixel may “break” randomly with an independent probability of 1/8 (Fig.
3A).
The original observation at a broken pixel is erased and is replaced by a different value in all channels.
We consider such noisy MinAtar environments with the noisy pixels as the executor observation and the original pixels as the oracle observation.
The network structure was the same as that for Maze, but the encoder was replaced by a CNN (Appendix C).
We ran experiments on all the 5 environments of MinAtar with VLOG as well as baseline, oracle and alternative oracle guiding methods (see Appendix A for details).
Baseline model always uses executor observation as network input (Fig. 1C).
Oracle is the same as baseline except that it always receives oracle observation (i.e., cheating, we ran the experiments of oracle for reference).
VLOG-no oracle is an ablation of VLOG where we use the executor observation as the input to posterior encoder in VLOG (oracle observation is not used).
Suphx-style oracle guiding is the oracle guiding method used in the Mahjong AI Suphx (Li et al., 2020), in which the executor observation and the dropout-ed oracle observation (with dropout probability p dropout ) are concatenated as the input to the network.
With training proceeds, p dropout is gradually increased from 0 to 1, thus the trained network does not need oracle observation for input (Appendix A).
OPD-style oracle guiding is the oracle guiding method used in oracle policy distillation (OPD) (Fang et al., 2021).
OPD-style oracle guiding ﬁrst trains a teacher model using oracle observation as input, and then trains the executor model using an auxiliary loss, which is the error between the executor and the teacher models’ estimation for value function (Appendix A).
The results show that oracle performed usually the best, as expected.
We normalized the performance of non-oracle models using oracle model as reference, for more clear comparison (Fig.
3B).
Among all the oracle guiding methods (VLOG, OPD-style and Suphx-style), VLOG consistently performed the best.
It is notable that VLOG and VLOG-no oracle performed surprisingly well in Seaquest.
This can be explained by that Seaquest is a task with a local optimum (see Appendix E), while the stochasticity in hidden states of VLOG helped exploration in latent space to escape from the local optimum (a similar idea is Fortunato et al.
(2018), but their noise was added to the network weights).
Except in Seaquest, VLOG-no oracle did not show signiﬁcant performance different with baseline, showing that the performance gain of VLOG in this task set mainly came from leveraging oracle observation for shaping latent distribution; and the usage of variational Bayesian model was, at least not harming the performance when there is no helpful oracle information.
5. O FFLINE LEARNING ON M AHJONG
Mahjong is a popular tile-based game with hundreds of millions of players worldwide (here we consider the Japanese variant).
The game is like many other card games (but using tiles instead of cards), in which multiple (usually four) players draw and discard tiles (totally 136 tiles) alternatively to satisfy winning conditions.
It is a highly challenging game characterized with (1) imperfect information in executor observation (a player cannot see opponents’ private tiles and the remaining tiles to be drawn), (2) stochastic state transitions as in many card games, and (3) extremely high game complexity (i.e., the number of distinguished, legal game states).
Complexity of Mahjong is much larger than 10 166 (Appendix F.1).
For reference, complexity of Go is ∼ 10 172 (Silver et al., 2016) and complexity of no-limit Poker is ∼ 10 162 (Johanson, 2013).
In Mahjong, it is hard to make optimal decisions based on executor observation because the outcomes heavily depend on invisible information, and complexity of invisible state space is as high as 10 48 on average (Li et al., 2020).
In response to this challenge, Li et al.
(2020) introduced suphx-style oracle guiding and demonstrated a performance gain.
Thus, we consider Mahjong as a promising test platform for oracle guiding methods.
Since the number of possible states in Mahjong is extremely large.
It is costly to explore with random actions in an online RL manner, and no previous work could train a strong Mahjong AI with purely online RL.
Also, we would like to examine the effectiveness of VLOG in ofﬂine RL settings.
For these reasons, we transferred to ofﬂine RL (Levine et al., 2020) for the Mahjong task using expert demonstrations.
We processed about 23M steps of human experts’ plays from the online Mahjong game platform
Tenhou (https://tenhou.net/mjlog.html) to a dataset for ofﬂine RL (data were augmented using the symmetry in Mahjong, see Appendix F).
Also, we created a simulator of Mahjong as the testing environment.
Though there are sophisticated ways to encodes the state and action space of Mahjong (Li et al., 2020), we attempts to make simpliﬁcations with reasonable amounts of approximations since our goal is to not create an strong Mahjong AI, but to use Mahjong as a platform to study oracle guiding problems.
In our case, the action space is composed of 47 discrete actions covering all decisions in Mahjong.
An executor observation is a matrix encoding public information and the current player’s private hand; An oracle observation concatenates the executor observation with the information of the opponents’ private hands.
(see Appendix F).
We used an 1-D CNN as the encoder as done in Mahjong AIs commonly (Li et al., 2020), and the size of z t and decoder network width was increased to 512 and 1024, respectively (Appendix C).
Note that although Mahjong is a 4-player game, using ofﬂine RL data to train an agent does not involve multi-agent RL (Zhang et al., 2021) because the ofﬂine dataset is ﬁxed: the opponents have ﬁxed policies and thus can be considered as parts of the environment.
Our experiments focused on single-agent RL to avoid the complexity caused by considering multi-agent RL.
We investigated two kinds of ofﬂine RL settings.
The ﬁrst is conservative Q-learning (CQL) (Kumar et al., 2020).
Our
CQL setting differed from the online RL setting in previous sections by adding auxiliary CQL loss (Kumar et al., 2020) to the Q-learning loss function (Appendix B.1.2).
The other is behavior cloning (BC).
Although VLOG was designed for value-based RL, we could straightforwardly incorporate VLOG with BC by letting the network predict action (and the decoder does not need action input).
The learning was conducted by minimizing the cross entropy between the output and the target action (demonstration) as in a classiﬁcation problem.
Note that we did not test OPD-style oracle guiding in BC setting because it was equal to the baseline since we can directly use demonstration actions as oracle policy for distillation.
[Table]
without oracle guiding (Fig. 1C)). To decrease the variance, we sum up the results of two players with the same model, and use it as the performance of the corresponding method.
Average payoff is the mean of points change per game. Each type of match was repeated 12,500 times (100,000 games).
Data are Mean ± S.E.M..
Because Mahjong is a zero-sum, four-player game, we tested the performance of trained models in two scenarios: playing with the (trained) baseline model (Table 1) and playing with each other (ﬁght each other, Table 2).
In the ﬁrst scenario, four agents were playing the same matches on the game table, where two of them were being tested agents and the other two were the baseline models.
Although each agent played for itself and there was no communication between players, we simply added up the payoff of the two being tested agents, and consider they won a match if one of them ranked top (so the match win rate will be 50% if equally strong as baseline), for statistics (Table 1).
[Table]
-in a punishment to the
, thus
For CQL, the results (Table 1 left and 2 upper) show that VLOG substantially outperformed the baseline and alternative methods (because Mahjong is a highly random game, 55.7% match win rate indicates a large skill gap).
Interestingly, VLOG was even comparable to oracle.
This can be explained by that VLOG also beneﬁted from its Bayesian property, which is consistent with that
VLOG-no oracle showed a signiﬁcant performance gain over the baseline model (Table 1 left).
Still, the oracle model learned to reduce deal-ins (i.e., player discards a tile and another player wins the game by picking up this tile to compose a winning hand) since it could explicitly see the opponents’ private tiles, showing a much lower deal-in rate than other non-cheating models (Table 2 upper).
In BC setting, the agents did not learn a value function, but tried to predict human experts’ actions.
Therefore, the training procedure did not involve reasoning the relationship between playing outcome and oracle observation, but just imitating human behaviors.
This can be seen from the results that oracle did not substantially outperform baseline in BC (Table 1 right and 2 lower).
However,
VLOG and VLOG-no oracle still showed performance gain, thanks to the stochastic modeling.
S UMMARY
We have proposed VLOG – a variational Bayesian learning framework for leveraging oracle observations to facilitate DRL especially in partially observable environments.
VLOG is available for any
RL problem in which there is oracle observation that may help the executor to make decision.
We ﬁrst introduced a latent vector z to represent the environmental state.
The prior and posterior distribution of z is modeled using executor and oracle observation, respectively.
Then, we derived a variational lower bound (Eq. 2) by maximizing which we can optimize the executor model using the oracle observations.
We developed the corresponding methodology for DRL, which can be incorporated with most RL algorithms that need to estimate a value function.
If oracle observation contains more information to retrieval the true environmental state (or, it is the true environmental state), VLOG’s oracle guiding in latent space helps to shape a latent representation in neural networks closer to the true one.
We demonstrated this advantage of VLOG using the maze task.
Then, we scaled VLOG up to solve image-based videos games, and compared it with alternative oracle-guiding methods.
Though all oracle-guiding methods showed performance gain over the baseline model, VLOG performed consistently the best.
Finally, we transferred to ofﬂine
RL domain using a challenging tile-based game Mahjong in which an executor plays with hidden information and random state transitions, and observed VLOG achieved best overall performance.
We also conducted ablation study of VLOG (VLOG-no oracle) in which posterior model did not receive oracle observation, but executor one.
VLOG-no oracle demonstrated performance gain in the tasks that may beneﬁt from the stochasticity; otherwise, it performs similar to the deterministic baseline.
This clariﬁed that the source of VLOG’s promising performance is two-fold: oracle guiding and stochastic modeling.
Finally, we publish the dataset of Mahjong for ofﬂine RL and the corresponding RL environment so as to facilitate future research on oracle guiding. A CKNOWLEDGEMENT
This work was supported by Microsoft Research Asia.
Kenji Doya was supported by Japan Society for the Promotion of Science KAKENHI Grant Numbers JP16K21738, JP16H06561 and
JP16H06563, as well as by Okinawa Institute of Science and Technology.
R EPRODUCIBILITY STATEMENT
The source code of VLOG can be found in Supplementary Material.
E THICS STATEMENT
We declare no conﬂict of interests.
We tried to use friendly colors to people with color recognition disabilities (Fig.
2C, D) and distinguishable markers for performance curves of different models (Fig.
2B, Fig. 3 and Fig6).
Our Mahjong dataset was generated using downloadable, public game replay data from Tenhou.net with post-processing.
The dataset contains no private information about players.
Since VLOG is a general framework for leveraging the oracle information, we cannot foresee any direct application of VLOG to malicious purposes.
However, any new RL algorithm might confer increased autonomy on an agent, and eventually lead to a completely autonomous agent, which can be used for malicious purposes, e g., fully autonomous soldiers.
A I MPLEMENTATION OF ALTERNATIVE ORACLE GUIDING METHODS
In the original works of Suphx (Li et al., 2020) and OPD (Fang et al., 2021), the authors were performing oracle guiding to different RL objectives to ours.
For comparison, we re-implemented
Suphx-style and OPD-style oracle guiding in our settings using similar network structures (see
Fig. 4) and used the same hyper-parameters wherever possible.
However, the key methodology of Suphx-style and OPD-style oracle guiding remains the same as those in their original works.
As shown in Fig.
4, left, Suphx-style model receives input
[Equation]
where δ t is the dropout matrix whose elements are Bernoulli variables with P ( δ t ( i, j ) = 1)
= 1 − p dropout , and (cid:12) denotes element-wise multiplication.
During training, p dropout was linearly increased from 0 to 1 in the ﬁrst 2 / 3 training epochs, and kept to be 0 in the remaining 1 / 3 epochs.
The other parts of training were the same as the baseline model.
For OPD-style oracle guiding, we directly used the trained oracle model as the teacher model (Fig. 4, right).
The loss function (in Q-learning) was
[Equation]
where Q tart is the Q-learning target obtained by bootstrapping the student model (The ﬁrst term (MSE ( Q tart , Q ( x t , a t )) ) was the same as the loss function for the baseline model).
And the second term is the distillation loss with an coefﬁcient µ (note that the teacher model is ﬁxed during training).
We selected the best-performing µ for each task using grid search (see Table 3).
B RL ALGORITHMS AND HYPER - PARAMETERS B. RL ALGORITHMS B.1. D UELING D OUBLE DQN FOR M AZE AND M IN A TAR TASKS
As we discussed in Sec. 5, we used double DQN with dueling network architecture (van Hasselt et al., 2016;
Wang et al., 2016) as the base RL algorithm, because it work relatively well (Hessel et al., 2018) without introducing additional hyper-parameter.
The Dueling architecture of DQN (Wang et al., 2016) is deﬁned as follows (see Appendix C for hidden layer size):
[Table]
et al., 2016) was used to compute Q target in Fig.
1 A (one can use any other algorithm to compute Q target without changing other parts).
In particular, as in Wang et al.
(2016), we have
[Equation]
where r t is the reward at step t , γ is the discount factor (Table 3), θ denotes the parameters of the Q network (MLP decoder) for computing the Q-function (Appendix C).
Note that z is given by the posterior decoder with oracle observation x as input, since the oracle prediction error term E q ( z t | ˆ x t )
[log P ( v ( z t ) =
v tart | z t )] in Eq.
1 is the expectation over the posterior distribution q
( z | ˆ x ) .
Like normally done in RL (Mnih et al., 2015; Wang et al., 2016; van Hasselt et al., 2016), we used a target Q network with the same structure as the original Q network, of which the parameters are denoted as θ − (Table 3).
Every 1,000 steps, the target Q network copies the parameters from the original Q network (Table 3).
Then the ﬁrst term of the VLOG loss function (Eq. 2) is simply given by the mean square error between Q target and the output of Q network (MLP decoder) for the Maze and MinAtar tasks.
B.1. D UELING D OUBLE DQN WITH CONSERVATIVE Q LEARNING FOR M AHJONG
In Mahjong, as we transfer to ofﬂine RL domain (Sec. 5.3), directly using off-policy RL algorithm usually results to very unsatisfying performance (Levine et al., 2020).
Therefore, we complement the loss function of VLOG (Eq. 2) with an auxiliary conservative Qlearning (CQL) loss (Kumar et al., 2020), which is an state-of-the-art ofﬂine RL algorithm:
[Equation]
where D is the ofﬂine dataset we used for Mahjong and α = 1 .
The combined loss function used for Mahjong (CQL) is J β VLOG + J CQL .
B.1. B EHAVIOR CLONING IN M AHJONG
We also tested behavior cloning with VLOG (Sec. 5.3) for Mahjong.
we implement VLOG with
BC by letting the decoder network predict action (and the decoder does not need action input).
The learning was conducted by minimizing the cross entropy between the output and the target action (demonstration) as in a classiﬁcation problem.
B. H YPER - PARAMETER SELECTION
We summarize the hyper-parameters in Table 3. [Table] [Table] [Table]
online RL
[Table]
Adam
[Table]
Fixed in all experiments
[Table] OPD [Table] style coefﬁcient of policy distillation loss
[Table]
By grid search
B. S ENSITIVITY ANALYSIS FOR β
As we mentioned in Sec.
4.2, the coefﬁcient β is important to the learning in VLOG.
If we ﬁx the value of β throughout training, a too large or too small β will result in worse performance.
The corresponding results are shown in Fig.
C N ETWORK STRUCTURES
For simplicity, we tried to use the same hidden layer size for all fully connected layers in the same model, where the hidden layer size is 256 for maze and MinAtar, and 1024 for Mahjong.
C. E NCODER
Since we targeted at various tasks, we used different encoder network for each type of environment.
The prior and posterior encoder has the same structure except different sizes of input features/channels.
For the maze task, the encoder was a 2-layers MLP with ReLU activation.
The output size is also equal to hidden layer size .
For the MinAtar tasks, we used 2-D CNN encoder deﬁned as follows: import t o r c h .
nn as nn c n n m o d u l e l i s t = nn .
ModuleList ( ) c n n m o d u l e l i s t .
append ( nn . Conv2d ( n channels , 16 , 3 , 1 , 0 ) )
c n n m o d u l e l i s t .
append ( nn . ReLU ( ) )
c n n m o d u l e l i s t .
append ( nn . Conv2d (16 , 32 , 3 , 1 , 0 ) )
c n n m o d u l e l i s t .
append ( nn . ReLU ( ) )
c n n m o d u l e l i s t .
append ( nn . Conv2d
c n n m o d u l e l i s t .
append ( nn . ReLU ( ) )
c n n m o d u l e l i s t .
append ( nn . Conv2d (128 , 256 , 2 , 1 , 0 ) )
c n n m o d u l e l i s t .
append ( nn . ReLU ( ) )
c n n m o d u l e l i s t .
append ( nn .
F l a t t e n ( ) )
cnn minatar
= nn .
S e q u e n t i a l ( * c n n m o d u l e l i s t ) where n channels is the number of channels of executor or oracle observation.
The output size is equal to hidden layer size .
For Mahjong, because the second dimension of observation (tile ID) has local contextual relationship (Li et al., 2020), we used 1-D CNN (convolution along the tile ID dimension) as the encoders, deﬁned as follows:
[Equation]
The output size is 1088, close to hidden layer size .
C.
[Equation]
L ATENT LAYER
For VLOG and VLOG (no-oracle), the size of z layer is half of hidden layer size because we need to estimate both the mean and the variance of z .
For all other models, the latent layer is one fully connected layer with size hidden layer size and ReLU activation.
[Equation]
D ECODER
The decoders for all models were 2-layers MLPs with size hidden layer size .
Except BC on
Mahjong, the input of decoder was the output of the latent layer and the action, and we used the dueling Q-network structure (Wang et al., 2016) to output a scalar Q value.
For BC on Mahjong, the input of decoder was the output of the latent layer.
The outputs of decoder was logit of actions, and the actions could be obtained using softmax.
D T ASK - AGNOSTIC LATENT BOTTLENECK CONTROL
As we discussed in Sec. 4.2, the coefﬁcient β of the regularization term in the VLOG loss function (Eq. 2) is adaptive regularized by Eq. 3, provided with D tarKL , which is another hyper-parameter.
While our experiments demonstrated the effectiveness of this approach, the follows discuss more thoughts behind this design choice.
In principle, replacing a hyper-parameter with another does not always make training easier.
However, in practice (especially deep RL), the performance will be highly sensitive to some hyperparameters (e.g., the entropy coefﬁcient α in the original soft-actor critic algorithm (Haarnoja et al., 2018) need to be tuned in each robotic task.
This is because the reward magnitude is different among tasks.).
As a result, it will be beneﬁcial to replace a sensitive hyper-parameter with another one that does not need ﬁne tuning.
For example, in the follow-up paper of soft-actor critic, the authors let the entropy coefﬁcient α to be adaptive by introducing another hyper-parameter, the entropy target (Haarnoja et al., 2019).
They empirically found that it is good to set the entropy target equal to the negative of the degree of freedom of the agent, thus to avoid tuning α .
Our idea of replacing β with D tarKL is due to similar reasons.
One obvious problem is that the magnitude of the “oracle prediction error” term (Eq. 2) relies on the reward magnitude of the task.
Therefore β should also be adjusted to match with the magnitude of task reward.
However, D tar KL is only relevant to the magnitude of prior z and posterior z t , which does not differ too much among tasks (usually at the order of 1).
In practical we found that a single value for the target KLD,
D tarKL = 50 , work well for all tasks including Maze, MinAtar and Mahjong.
Another way of regularizing β is to employ a linear or exponential scheduler.
For example, in
Burgess et al.
(2017), the authors used a linear scheduler for the target KL-divergence and got good results.
However, using a scheduler introducing more hyper-parameters (at least two: initial β and ﬁnal β ), which is against our willing to reduce the impact of hyper-parameters.
E S EAQUEST LOCAL OPTIMUM
In Seaquest, the agent drives a submarine diving into the sea to shoot at enemies and rescue divers to earn scores.
However, the submarine has limited oxygen.
It must surface to replenish the oxygen before running out for surviving, by doing which it temporarily cannot earn scores.
A typical local optimum is to use the last remaining oxygen for diving instead of surfacing.
F M AHJONG F. E STIMATION OF GAME COMPLEXITY OF M AHJONG
We consider 4-players Japanese Mahjong 4 .
Although there are minor variants of rules, the following estimation applies to general cases.
For easier computation, we make two major simpliﬁcations (i.e., we estimate a lower bound of the game complexity): (1) melding from discard 5 is not considered and (2) the information other than tiles, such as results of contextual games, points of players, is not considered.
There are 34 types of tiles, each with 4 duplicates (so totally 136 tiles).
We further restrict our estimation to the last turn (i.e., the last tile is drawn).
Among 136 tiles, 53 tiles are in someone’s hand (the 4 players have 14, 13, 13, 13 tiles in hands, respectively).
Permutation of tiles in one’s hand does not make a different, while the permutation matters if not in one’s hand.
The number of distinguishable conﬁgurations of 136 tiles thus can be computed as
[Table]
Meanwhile, for each discarded tile, it is important to know whether it was discarded immediately after being drawn or not.
For 70 discarded tiles, the number of possibilities is simply 2 70 ∼ 10 21 .
Therefore, the lower bound of game complexity of Mahjong is estimated as 10 145 × 10 21 ∼ 10 166 .
If considering the being simpliﬁed information, the state space could be much larger.
For example, let’s consider the current points of each player.
The most common rule is that each player starts with 25,000 points, with 100 points as the minimal unit (totally 1,000 units), and the game terminates if someone got negative points.
So the number of possibilities can be converted to the answer of “how many ways to distribute 1000 candies to 4 kids”, which is (1000+1) (4 − 1) (4 − 1)!
∼ 10 8 .
F. D ETAILS OF OBSERVATION SPACE AND ACTION SPACE ENCODING
In our Mahjong environment 6 , the action space is composed of 47 discrete actions (Table 5).
Because not all actions are available at a certain step, we also provide the set of valid actions among allactions at each step according to the rules, which can be used during playing and learning.
The oracle observation has the shape of 111 × 34 (executor observation is a part of oracle observation, with shape 93 × 34) (Fig. 5).
The ﬁrst dimension corresponds to 111 features (channels).
The second dimension of observation (with size 34) corresponds to 34 Mahjong tiles (the order is
Character 1-9, Dot 1-9, Bamboo 1-9, East, South, West, North, White, Green, Red).
We used 1-D
CNNs with convolution along the second dimension for the encoders.
Suppose the current player is player 0, and other players are numbered 1, 2, 3 counter-clockwise.
The value of any element in an observation is 1 or 0, explained in Table 4.
[Table] [Table]
Action index
Explanation
Discard Character Discard Character
Discard Character Discard Character
Discard Character 5 (non-Red Dora with higher priority)
Discard Character Discard Character
Discard Character Discard Character
Discard Dot Discard Dot
Discard Dot Discard Dot
Discard Dot 5 (non-Red Dora with higher priority)
Discard Dot Discard Dot
Discard Dot Discard Dot
Discard Bamboo Discard Bamboo
Discard Bamboo Discard Bamboo
Discard Bamboo 5 (non-Red Dora with higher priority)
Discard Bamboo Discard Bamboo
Discard Bamboo Discard Bamboo
Discard East Wind
Discard South Wind Discard West Wind
Discard North Wind
Discard White Dragon Tile (Haku)
Discard Green Dragon Tile (Hatsu)
Discard Red Dragon Tile (Chu)
Chi (the picked up tile is the smallest in the meld)
Chi (the picked up tile is the middle in the meld)
Chi (the picked up tile is the largest in the meld)
Pon
An-Kan
Kan
Ka-Kan
Riichi
Ron
Tsumo
Restart the game with Kyushukyuhai
Not to response (when Chi, Pon, Kan, Ron, etc. is possible)
Not to Riichi (When Riichi is possible)
F. D ATA AUGMENTATION
In (Japanese) Mahjong, there are 3 suit sets of (characters, dots, bamboos) tiles, 4 wind tiles anddragon tiles. The 3 suit sets are symmetric and thus exchangeable with each other 7 . So are thewind tiles and the 3 dragon tiles. According to such symmetry, we augmented the ofﬂine RL dataset by randomly exchanging the 3 suit sets, the 4 wind tiles and the 3 dragon tiles, respectively.