S OLVING THE R UBIK ’ S C UBE WITH A PPROXIMATE P OLICY I TERATION
A BSTRACT
Recently, Approximate Policy Iteration (API) algorithms have achieved superhuman proﬁciency in two-player zero-sum games such as Go, Chess, and Shogi without human data.
These API algorithms iterate between two policies: a slow policy (tree search), and a fast policy (a neural network).
In these two-player games, a reward is always received at the end of the game.
However, the Rubik’s Cube has only a single solved state, and episodes are not guaranteed to terminate.
This poses a major problem for these API algorithms since they rely on the reward received at the end of the game.
We introduce Autodidactic Iteration: an API algorithm that overcomes the problem of sparse rewards by training on a distribution of states that allows the reward to propagate from the goal state to states farther away.
Autodidactic Iteration is able to learn how to solve the Rubik’s Cube without relying on human data.
Our algorithm is able to solve 100% of randomly scrambled cubes while achieving a median solve length of 30 moves — less than or equal to solvers that employ human domain knowledge.
I NTRODUCTION
The Rubik’s Cube is a classic combination game that poses unique and interesting challenges for AI and machine learning.
Although the state space is astronomically large ( 4 . 2 × 10 19 different states for the 3x3x3 cube), only a single state is considered solved.
Furthermore, unlike the game of Go or
Chess, the Rubik’s Cube is a single-player game and a sequence of random moves, no matter how long, is unlikely to end in the solved state.
Developing reinforcement learning algorithms to deal with this property of the Rubik’s Cube might provide insight into other sparse-reward environments.
While methods for solving the Rubik’s Cube have been developed, only relatively recently have methods been derived that can compute the minimal number of moves required to solve the cube from any given starting conﬁguration (Rokicki et al., 2014; Rokicki, 2014).
In addition, the Rubik’s Cube and its solutions are deeply rooted in group theory, raising interesting and broader questions about the applicability of machine learning methods to complex symbolic systems, including mathematics.
Finally, the classical 3x3x3 Rubik’s Cube is only one representative of a much larger family of possible combination puzzles, broadly sharing the characteristics described above and including: (1) Rubik’s
Cubes with longer edges (e.g. 4x4x4); (2) Rubik’s Cubes in higher dimensions (e.g. 2x2x2x2); as well as (3) Rubik’s cube on non-cubic geometries (Pyriminix, etc) and their combinations.
As the length of the sides and dimensions are increased, the complexity of the underlying combinatorial problems rapidly increases and, for instance, God’s numbers for the 4x4x4 cube is not known.
In short, for all these reasons, the Rubik’s Cube and its variations pose interesting challenges for machine learning.
Here we develop deep reinforcement learning methods, in particular a new form of Approximate
Policy Iteration (API), for addressing these challenges.
Approximate Policy Iteration (Bertsekas & Tsitsiklis, 1995;
Bagnell et al., 2004; Kakade & Langford, 2002; Scherrer, 2014; Lazaric et al., 2010) is a core Reinforcement Learning (RL) algorithm.
Recently, a new type of API algorithm, called Dual Policy Iteration (Sun et al., 2018), has achieved success in two player zero-sum games such as Go, Chess, Shogi and Hex (Anthony et al., 2017; Silver et al., 2017a;c).
AlphaZero
(Silver et al., 2017c) and ExIt (Anthony et al., 2017) are examples of Dual
Policy Iteration.
These algorithms update the policy in a more sophisticated way than traditional API methods by using two policies: a fast policy (usually a neural network) and a slow policy (usually tree search).
The fast policy is trained via supervised learning on data generated from gameplay from the slow policy.
The slow policy then uses the fast policy to guide the tree search.
In this way, the fast policy is used to improve the slow policy, and the slow policy generates better data to train the fast policy.
This work is the ﬁrst to solve the Rubik’s Cube with reinforcement learning.
Although DPI works for two-player games, our work is the ﬁrst DPI algorithm to succeed in an environment with a high number of states and a small number of reward states.
A number of these "needle in a haystack" environments such as generating meaningful sentences or code are seen as long-term goals for the ﬁeld of AI.
There is no clear way to apply current DPI algorithms such as AlphaZero or ExIt to sparse-reward environments such as the Rubik’s Cube.
This is because a randomly initialized policy will be unlikely to encounter the single reward state.
This will cause the value function to be biased or divergent and the fast policy will not converge to the optimal policy.
In our work we overcome this problem by training the fast policy on a distribution of states that propagate the reward signal to from the goal state to the further states.
Our algorithm, called Autodidactic Iteration (ADI), trains a neural network value and policy function through an iterative process.
These neural networks are the "fast policy" of DPI described earlier.
In each iteration, the inputs to the neural network are created by starting from the goal state and randomly taking actions.
The targets seek to estimate the optimal value function by performing a breadth-ﬁrst search from each input state and using the current network to estimate the value of each of the leaves in the tree.
Updated value estimates for the root nodes are obtained by recursively backing up the values for each node using a max operator.
The policy network is similarly trained by constructing targets from the move that maximizes the value.
After the network is trained, it is combined with MCTS to effectively solve the Rubik’s Cube.
We call the resulting solver DeepCube.
R ELATED WORK
Erno Rubik created the Rubik’s Cube in 1974.
After a month of effort, he came up with the ﬁrst algorithm to solve the cube.
Since then, the Rubik’s Cube has gained worldwide popularity and many human-oriented algorithms for solving it have been discovered (Ruwix).
These algorithms are simple to memorize and teach humans how to solve the cube in a structured, step-by-step manner.
Human-oriented algorithms to solve the Rubik’s Cube, while easy to memorize, ﬁnd long suboptimal solutions.
Since 1981, there has been theoretical work on ﬁnding the upper bound for the number [Figure] of moves necessary to solve any valid cube conﬁguration (Thistlethwaite, {+
 ;+} Reid, 1995; Radu, 2007; Kunkle & Cooperman, 2007; Rokicki, 2010).
Finally, in 2014, it was shown that any valid cube can be optimally solved with at most 26 moves in the quarter-turn metric, or 20 moves in the half-turn metric
(Rokicki et al., 2014; Rokicki, 2014).
The quarter-turn metric treats 180 degree rotations as two moves, whereas the half-turn metric treats 180 degree rotations as one move.
For the remainder of this paper we will be using the quarter-turn metric.
This upper bound on the number of moves required to solve the Rubik’s cube is colloquially known as God’s Number.
Algorithms used by machines to solve the Rubik’s Cube rely on hand-engineered features and group theory to systematically ﬁnd solutions.
One popular solver for the Rubik’s Cube is the Kociemba two-stage solver (Kociemba).
This algorithm uses the Rubik’s Cube’s group properties to ﬁrst maneuver the cube to a smaller sub-group, after which ﬁnding the solution is trivial.
Heuristic based search algorithms have also been employed to ﬁnd optimal solutions.
In 1985, it was shown that iterative deepening A* (IDA*) could be used to ﬁnd optimal solutions to combination puzzles (Korf, 1985).
This algorithm was then combined with pattern databases (Culberson & Schaeffer, 1998) to successfully ﬁnd optimal solutions to the Rubik’s Cube (Korf, 1997).
Later, (Sturtevant & Chen, 2016) used advances in compute power (5TB of external memory and parallel processing), to show that bidirectional brute-force search could ﬁnd optimal solutions to the Rubik’s Cube in up tohours.
More recently, there has been an attempt to train a DNN – using supervised learning with hand-engineered features – to act as an alternative heuristic (Brunetto & Trunda, 2017).
These search algorithms, however, take an extraordinarily long time to run and usually fail to ﬁnd a solution to randomly scrambled cubes within reasonable time constraints.
Besides hand-crafted algorithms, attempts have been made to solve the Rubik’s Cube through evolutionary algorithms (Smith et al., 2016; Lichodzijewski & Heywood, 2011).
However, these learned solvers can only reliably solve cubes that are up to 5 moves away from the solution.
We solve the Rubik’s Cube using API (Bertsekas & Tsitsiklis, 1995;
Bagnell et al., 2004; Kakade & Langford, 2002; Scherrer, 2014; Lazaric et al., 2010).
API is a general purpose reinforcement learning algorithm, and includes API with stochastic policies (Schulman et al., 2015; Kakade, 2001; Bagnell & Schneider, 2003; Baxter & Bartlett, 2001), conservative API (Kakade & Langford, 2002), and API with learned critics (Rummery & Niranjan, 1994).
Even though many of these
API algorithms have certain theoretical optimality guarantees, they rely on random exploration (such as REINFORCE (Williams, 1992)-type policy gradient or (cid:15) -greedy) and as a result are not sample-efﬁcient.
Dual Policy Iteration (Sun et al., 2018) algorithms such as AlphaZero and Exit are a new type of API algorithm that has achieved success in two player zero-sum games such as Go, Chess, Shogi and
Hex (Anthony et al., 2017; Silver et al., 2017a;c) .
These algorithms use two policies: a fast policy and a slow policy.
The fast policy is trained via supervised learning on data generated from gameplay from the slow policy.
The slow policy then uses the fast policy to guide the tree search.
The main difference between DPI and traditional API algorithms is the use of a forward dynamics model.
With the use of this model, the slow policy can perform a tree search, abandoning the need for random exploration.
Using this forward dynamics model allows for improved sample efﬁciency compared to traditional API algorithms.
It is now well-known that reinforcement learning with nonlinear function approximation is prone to diverge (Baird, 1995; Boyan & Moore, 1995; Gordon, 1995).
Wang & Dietterich (1999) approaches the problem of learning a stable value function with nonlinear approximation for combinatorial optimization by using a composite error function.
Learning a value function with nonlinear approximation within API is not much of a problem when trying to solve Go or Chess.
This is because the problem is simply binary classiﬁcation based on the current board state.
With the Rubik’s Cube, on the other hand, approximating the value function is very difﬁcult because even a naive monte-carlo approach would not work since it will never encounter the solved state.
ADI ensures stable and approximately unbiased training of the value function by sampling states in a way that ensures the goal state’s learning signal will propagate during the training.
This approach is somewhat similar to the method proposed in Boyan & Moore (1995).
T HE R UBIK ’ S C UBE
The Rubik’s Cube consists of 26 smaller cubes called cubelets.
These are classiﬁed by their sticker count: center, edge, and corner cubelets have 1, 2, and 3 stickers attached respectively.
There arestickers in total with each sticker uniquely identiﬁable based on the type of cubelet the sticker is on and the other sticker(s) on the cubelet.
Therefore, we may use a one-hot encoding for each of thestickers to represent their location on the cube.
However, because the position of one sticker on a cubelet determines the position of the remaining stickers on that cubelet, we may actually reduce the dimensionality of our representation by focusing on the position of only one sticker per cubelet.
We ignore the redundant center cubelets and only store the 24 possible locations for the edge and corner cubelets.
This results in a 20x24 state representation which is demonstrated in Figures 2c and 2d.
Moves are represented using face notation originally developed by David Singmaster: a move is a letter stating which face to rotate.
F , B , L , R , U , and D correspond to turning the front , back , left , right , up , and down faces, respectively.
A clockwise rotation is represented with a single letter, whereas a letter followed by an apostrophe represents a counter-clockwise rotation.
For example: R and R’ would mean to rotate the right face 90 ◦ clockwise and counter-clockwise, respectively.
Formally, the
Rubik’s Cube environment consists of a set of 4 .
3 × 10 19 states S which contains one special state, s solved , representing the goal state.
At each timestep, t , the agent observes a state s t ∈ S and takes an action a t ∈ A with A :=
{ F, F’, ..., D, D’ } .
After selecting an action, the agent observes a new state s t +
=
A ( s t , a t ) and receives a scalar reward, R ( s t +1 ) , which is 1 if s t +1 is the goal state and − 1 otherwise.
It is important to note that this environment is deterministic with known forward and backwards dynamics.
M ETHODS
We develop a novel algorithm called Autodidactic Iteration which is used to train a joint value and policy network.
Once the network is trained, it is combined with MCTS to solve randomly scrambled cubes.
The resulting solver is called DeepCube.
Autodidactic Iteration is a DPI algorithm like AlphaZero and ExIt.
However, since AlphaZero and ExIt cannot solve the Rubik’s Cube,
Autodidactic Iteration uses a state sampling procedure during training to prevent divergence of the value function and to ensure that the learning signal from the single reward state is propagated from the solved cube during training.
In particular, the sampling distribution is generated by starting from the solved state and randomly taking actions, weighting cubes closer to the solved state more heavily.
This sampling procedure is described in the State Sampling section below.
4. A UTODIDACTIC I TERATION
ADI is an iterative supervised learning procedure which trains a deep neural network f θ ( s ) with parameters θ which takes an input state s and outputs a value and policy pair ( v, p ) .
The policy output p is a vector containing the move probabilities for each of the 12 possible moves from that state.
Once the network is trained, the policy is used to reduce breadth and the value is used to reduce depth in the MCTS.
In each iteration of ADI, training samples for f θ are generated by starting from the solved cube.
This ensures that some training inputs will be close enough to have a positive reward when performing a shallow search.
Targets are then created by performing a depth-1 breadth-ﬁrst search (BFS) from each training sample.
The current value network is used to estimate each child’s value.
The value target for each sample is the maximum value and reward of each of its children, and the policy target is the action which led to this maximal value.
Figure 3 displays a visual overview of
ADI.
Formally, we generate training samples by starting with s solved and scrambling the cube k times to generate a sequence of k cubes.
We do this l times to generate N = k ∗ l training samples
X =
[ x i ]
Ni =1 .
Each sample in the series has the number of scrambles it took to generate it, D ( x i ) , associated with it.
Then, for each sample x i ∈ X , we generate a training target
Y i =
( y v i , y p i ) .
To do this, we perform a depth-1 BFS to get the set of all children states of x i .
We then evaluate the current neural network at each of the children states to receive estimates of each child’s optimal value and policy: ∀ a ∈ A , ( v x i ( a ) , p x i ( a ))
=
f θ ( A ( x i , a )) .
We set the value target to be the maximal value from each of the children y v i ← max a ( R ( A ( x i , a ))
+ v x i ( a )) ,
and we set the policy target to be the move that results in the maximal estimated value y p i ← argmax a ( R ( A ( x i , a ))
+ v x i ( a )) .
We then train f θ on these training samples and targets [ x i , y v i ]
Ni =1 and [ x i , y p
i ]
Ni =1 to receive new neural network parameters θ (cid:48) .
For training, we used the RMSProp optimizer (Hinton et al., 2014) with a mean squared error loss for the value and softmax cross entropy loss for the policy.
Although we use a depth-1 BFS for training, this process may be trivially generalized to perform deeper searches at each x i .
4. S TATE S AMPLING
Correct sampling of training data is vital or the convergence of API.
It is particularly important for the Rubik’s Cube, since a naive state sampling will never include the solved state and the reward signal will not be learned by the value function.
ADI trains on states that are generated by starting from the solved cube and taking random actions.
This allows the reward signal to propagate from the solved state to states that are farther away.
However, ensuring the reward signal is propagated is not the only consideration for the Rubik’s Cube.
Training a nonlinear approximate value function often leads to unstable or divergent behavior (Boyan & Moore, 1995; Baird, 1995; Gordon, 1995).
This is because although the optimal value function is usually smooth and easy to approximate, the intermediate value functions generated by approximate policy iteration are rarely smooth, resulting in divergent behaviour.
This problem is especially true for the Rubik’s Cube since only one state has a positive reward.
One solution to value function divergence is to only train on true value data.
Such
Initialization: θ initialized using Glorot initialization repeat X ← N scrambled cubes for x i ∈
X do for a ∈ A do ( v x i ( a ) , p x i ( a )) ← f θ ( A ( x i , a )) y v i ← max a ( R ( A ( x i , a )) + v x i ( a )) y p i ← argmax a ( R ( A ( x i , a ))
+ v x i ( a ))
Y i ← ( y v i , y p i ) θ (cid:48) ← train ( f θ , X, Y )
θ ← θ (cid:48) until iterations =
M ; a solution is presented by Boyan & Moore (1995), who only train the approximate value function on a subset of support states, whose value has been explicitly computed via tree search or similar methods.
We take a similar approach by weighting states that are earlier in the rollout from the solved state more heavily during training.
This way, the algorithm trains on data that it is more conﬁdent of the true value.
We assign the loss weight W ( x i )
= 1 D ( x i ) to each sample x i .
Although since we are dealing with nonlinear function approximation and TD-learning we do not have any theoretical guarantees, empirically this state sampling procedure prevented divergent behaviour.
Since the algorithm prioritized learning states close to the solved state, it was then able to use these predictions to come up with more accurate targets for states farther away.
Without this novel state sampling procedure, Algorithm 1 would often diverge after several iterations, but with the state sampling procedure it did not diverge in any of our tests.
4. S OLVER
We employ an asynchronous Monte Carlo Tree Search augmented with our trained neural network f θ to solve the cube from a given starting state s 0 .
We build a search tree iteratively by beginning with a tree consisting only of our starting state, T = { s 0 } .
We then perform simulated traversals until reaching a leaf node of T .
Each state, s ∈ T , has a memory attached to it storing: N s ( a ) , the number of times an action a has been taken from state s , W s ( a ) , the maximal value of action a from state s ,
L s ( a ) , the current virtual loss for action a from state s , and P s ( a ) , the prior probability of action a from state s .
Every simulation starts from the root node and iteratively selects actions by following a tree policy until an unexpanded leaf node, s τ , is reached.
The tree policy proceeds as follows: for each timestep t , an action is selected by choosing, A t = argmax a U s t ( a ) + Q s t ( a ) where
U s t ( a ) = cP s t ( a ) (cid:112) (cid:80) a (cid:48)
N s t ( a (cid:48) ) /
(1 + N s t ( a )) , and Q s t ( a ) =
W s t ( a ) − L s t ( a ) with an exploration hyperparameter c .
The virtual loss is also updated L s t ( A t ) ← L s t ( A t ) + ν using a virtual loss hyperparameter ν .
The virtual loss prevents the tree search from visiting the same state more than once and discourages the asynchronous workers from all following the same path (Segal, 2011).
Once a leaf node, s τ , is reached, the state is expanded by adding the children of s τ , { A ( s τ , a ) , ∀ a ∈
A} , to the tree T .
Then, for each child s (cid:48) , the memories are initialized: W s (cid:48) ( · ) = 0 , N s (cid:48) ( · ) = 0 ,
P s (cid:48) ( · )
= p s (cid:48) , and L s (cid:48) ( · ) = 0 , where p s (cid:48) is the policy output of the network f θ ( s (cid:48) ) .
Next, the value and policy are computed: ( v s τ , p s τ ) =
f θ ( s τ ) and the value is backed up on all visited states in the simulated path.
For 0 ≤ t ≤ τ , the memories are updated: W s t ( A t ) ← max ( W s t ( A t ) , v s τ ) , N s t ( A t ) ←
N s t ( A t ) + 1 , L s t ( A t ) ← L s t ( A t ) − ν .
Note that, unlike other implementations of MCTS, only the maximal value encountered along the tree is stored, and not the total value.
This is because the
Rubik’s Cube is deterministic and not adversarial, so we do not need to average our reward when deciding a move.
The simulation is performed until either s τ is the solved state or the simulation exceeds a ﬁxed maximum computation time.
If s τ is the solved state, then the tree T of the simulation is extracted and converted into an undirected graph with unit weights.
We expand the graph by adding the children of any unexplored states and we perform a full breath-ﬁrst search on T to ﬁnd the shortest predicted path from the starting state to solution.
Alternatively, the last sequence Path =
{ A t | 0 ≤ t ≤ τ } may be used, but this produces longer solutions.
R ESULTS
As a baseline, we compare DeepCube against two other solvers.
The ﬁrst baseline is the Kociemba two-stage solver (Kociemba; Tsoy, 2018).
This algorithm relies on human domain knowledge of the group theory of the Rubik’s Cube.
Kociemba will always solve any cube given to it, and it runs very quickly.
However, because of its general-purpose nature, it often ﬁnds a longer solution compared to the other solvers.
The other baseline is the Korf Iterative Deepening A* (IDA*) with a pattern database heuristic (Korf, 1997; Brown, 2017).
Korf’s algorithm will always ﬁnd
the optimal solution from any given starting state; but, since it is a heuristic tree search, it will often have to explore many different states and it will take a long time to compute a solution.
We also compare the full DeepCube solver against two variants of itself.
First, we do not calculate the shortest path of our search tree and instead extract the initial path from the MCTS: this will be named Naive DeepCube .
We also use our trained value network as a heuristic in a greedy best-ﬁrst search for a simple evaluation of the value network: this will be named Greedy .
We compare our results to Kociemba using 640 randomly scrambled cubes.
Starting from the solved cube, each cube was randomly scrambled 1000 times.
Both DeepCube and Kociemba solved allcubes within one hour.
Kociemba solved each cube in under a second, while DeepCube had a median solve time of 10 minutes.
The systematic approach of Kociemba explains its low spread of solution lengths with an IQR of only 3 .
Although DeepCube has a much higher variance in solution length, it was able to match or beat Kociemba in 55% of cases.
We also compare DeepCube against Naive
DeepCube to determine the beneﬁt of performing the BFS on our MCTS tree.
We ﬁnd that the BFS has a slight, but consistent, performance gain over the MCTS path ( − 3 median solution length).
BFS removes any trivial cycles in the path such as a move followed by its inverse, but it also improves solution lengths by ﬁnding a slightly more efﬁcient path between states that are within 3 moves of each other.
A comparison of solution length distributions for these three solvers is presented in the left graph of Figure 4.
We could not include Korf in the previous comparison because its runtime is prohibitively slow: solving just one of the 640 cubes took over 6 days.
We instead evaluate the optimality of solutions found by DeepCube by comparing it to Korf on cubes closer to solution.
We generated a new set of 100 cubes that were only scrambled 15 times.
At this distance, all solvers could reliably solve all 100 cubes within an hour.
We compare the length of the solutions found by the different solvers in the right graph of Figure 4.
Noticeably, DeepCube performs much more consistently for close cubes compared to Kociemba, and it almost matches the performance of Korf.
The median solve length for both DeepCube and Korf is 13 moves, and DeepCube matches the optimal path found by Korf in 74% of cases.
However, DeepCube seems to have trouble with a small selection of the cubes that results in several solutions being longer than 15 moves.
Note that Korf has one outlier that is above 15 moves.
This is because Korf is based on the half-turn metric while we are using the quarter-turn metric.
Furthermore, our network also explores far fewer tree nodes when compared to heuristic-based searches.
The Korf optimal solver requires an average expansion ofbillion different nodes for fully scrambled cubes before ﬁnding a solution (Korf, 1997).
Our MCTS algorithm expands an average of only 7,823 nodes with a maximum of 24,175 expanded nodes on the longest running sample.
This is why DeepCube is able to solve fully scrambled cubes much faster than Korf.
Furthermore, this small amount of nodes explored means that the algorithm can potentially be more efﬁcient with an implementation that better exploits parallel processing.
D ISCUSSION
DeepCube is based on similar principles as AlphaZero and ExIt, however, these methods receive their reward when the game reaches a terminal state, which is guaranteed to occur given enough play time.
On the other hand, one is not guaranteed to ﬁnd a terminal state in the Rubik’s Cube environment and therefore may only encounter rewards of -1, which does not provide enough information to solve the problem.
DeepCube addresses this by selecting a state distribution for the training set that allows the reward to be propagated from the terminal state to other states.
In addition, while AlphaZero and
ExIt make use advanced tree-search algorithms to update the policy, DeepCube is able to ﬁnd success using the faster and simpler depth-1 BFS.
The depth-1 BFS used to improve the policy can also be viewed as doing on-policy temporal difference learning (Sutton, 1988), speciﬁcally TD(0) with function approximation.
In this case, the TD(0) algorithm uses a deterministic greedy policy where each episode is one step long.
Off-policy methods are often used to train a deterministic policy by using a stochastic behavior policy to facilitate exploration.
An alternative approach, called exploring starts, instead changes the distribution of starting states and can also be used to train a deterministic policy in an on-policy fashion (Sutton & Barto, 1998).
We use a similar approach to exploring starts by ensuring exploration through the selection of the starting state distribution.
The Rubik’s Cube can be thought of as a classical planning problem.
While traditional planning algorithms, such as Dijkstra’s algorithm, would require an infeasible amount of memory to work on environments a state space as large as the Rubik’s Cube, we show that Dual Policy Iteration can ﬁnd a solution path in such an environment.
For future work, we look to apply Autodidactic Iteration to a variety of other problems with similar characteristics such as robotic manipulation, two-player games, and path ﬁnding.
Léon Bottou deﬁnes reasoning as “algebraically manipulating previously acquired knowledge in order to answer a new question”(Bottou, 2013).
Many machine learning algorithms do not reason about problems but instead use pattern recognition to perform tasks that are intuitive to humans, such as object recognition.
By combining neural networks with symbolic AI, we are able to create algorithms which are able to distill complex environments into knowledge and then reason about that knowledge to solve a problem.
DeepCube is able to teach itself how to reason in order to solve a complex environment with only one positive reward state using pure reinforcement learning.
B T RAINING AND S OLVER D ETAILS
Each layer is fully connected.
We use elu activation on all layers except for the outputs.
A combined value and policy network results in more efﬁcient training compared to separate networks.
(Silver et al., 2017b).
We used a feed forward network as the architecture for f θ as shown in Figure 7.
The outputs of the network are a 1 dimensional scalar v , representing the value, and a 12 dimensional vector p , representing the probability of selecting each of the possible moves.
The network was then trained using ADI for 2,000,000 iterations.
The network witnessed approximately 8 billion cubes, including repeats, and it trained for a period of 44 hours.
Our training machine was a 32-core
Intel Xeon E5-2620 server with three NVIDIA Titan XP GPUs.
During play, the neural network prediction is the major bottleneck for performance.
In order to counteract this, we implemented a parallel version of MCTS that had 32 independent workers that shared a single MCTS search tree.
Each of the workers queue their prediction requests, and the neural network batches all requests and processes them simultaneously.
This parallelization sped up the solver 20x compared to a single core implementation.
C O VERALL S OLVER C OMPARISON
[Table]
Mean
[Table]
Length
Memory Requirement [Table] [Table] 86E
[Table]
[Table]
[Table]
86E
[Table]
[Table]
182 GB [Table] 122 billion [Table]
[Table]
2 GB
[Table]
[Table]
823E
[Table]
[Table]
[Table]
1 GB
[Table]
N/A
[Table]
N/A
[Table]
30 MB