Original: R EASONING
Revised: R EASONING

Original: W ITH
Revised: W ITH

Original: H IERARCHICAL
Revised: H IERARCHICAL

Original: S YMBOLS :
Revised: S YMBOLS :

Original: R ECLAIMING S YMBOLIC P OLICIES F OR V ISUAL R EINFORCEMENT
Revised: R ECLAIMING S YMBOLIC P OLICIES F OR V ISUAL R EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep vision models are nowadays widely integrated into visual reinforcement learning (RL) to parameterize the policy networks.
Revised: Deep vision models are nowadays widely integrated into visual reinforcement learning (RL) to parameterize the policy networks.

Original: However, the learned policies are overparameterized black boxes that lack interpretability, and are usually brittle under input distribution shifts.
Revised: However, the learned policies are overparameterized black boxes that lack interpretability, and are usually brittle under input distribution shifts.

Original: This work revisits this end-to-end learning pipeline, and proposes an alternative stage-wise approach that features hierarchical reasoning.
Revised: This work revisits this end-to-end learning pipeline, and proposes an alternative stage-wise approach that features hierarchical reasoning.

Original: Speciﬁcally, our approach progressively converts a policy network into the interpretable symbolic policy, composed from geometric and numerical symbols and operators.
Revised: Speciﬁcally, our approach progressively converts a policy network into the interpretable symbolic policy, composed from geometric and numerical symbols and operators.

Original: A policy regression algorithm called RoundTourMix is proposed to distill the symbolic rules as teacher-student.
Revised: A policy regression algorithm called RoundTourMix is proposed to distill the symbolic rules as teacher-student.

Original: The symbolic policy can be treated as discrete and abstracted representations of the policy network, but are found to be more interpretable, robust and transferable.
Revised: The symbolic policy can be treated as discrete and abstracted representations of the policy network, but are found to be more interpretable, robust and transferable.

Original: The proposed symbolic distillation approach is experimentally demonstrated to maintain the performance and “denoise” the CNN policy: on six speciﬁc environments, our distilled symbolic policy achieved compelling or even higher scores than the CNN based RL agents.
Revised: The proposed symbolic distillation approach is experimentally demonstrated to maintain the performance and “denoise” the CNN policy: on six speciﬁc environments, our distilled symbolic policy achieved compelling or even higher scores than the CNN based RL agents.

Original: Our codes will be fully released upon acceptance.
Revised: Our codes will be fully released upon acceptance.

Original: [Figure]
Revised: [Figure]

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) is effective to explore complex environments through interactions (Kaiser et al., 2019).
Revised: Reinforcement learning (RL) is effective to explore complex environments through interactions (Kaiser et al., 2019).

Original: A central target in building stronger RL algorithms is to learn reliable and generalizable policies, robust to unseen real-world data with possible shifts.
Revised: A central target in building stronger RL algorithms is to learn reliable and generalizable policies, robust to unseen real-world data with possible shifts.

Original: This is a difﬁcult task, especially when the input space is high-dimensional or continuous, e.g., the visual data, and the sampled observations during training are hence destined to be extremely sparse.
Revised: This is a difﬁcult task, especially when the input space is high-dimensional or continuous, e.g., the visual data, and the sampled observations during training are hence destined to be extremely sparse.

Original: To learn sophisticated policies from those data, it is common to parameterize policies as numerical predictors, e.g, neural networks (Francois-Lavet et al., 2018).
Revised: To learn sophisticated policies from those data, it is common to parameterize policies as numerical predictors, e.g, neural networks (Francois-Lavet et al., 2018).

Original: Despite their success, the neural network based RL algorithms require prohibitive amounts of data to train, and the resultant rules are “black boxes” that are hardly interpretable (Mnih et al., 2013; Heuillet et al., 2021) nor stable under input domain shifts (Schmeckpeper et al., 2020).
Revised: Despite their success, the neural network based RL algorithms require prohibitive amounts of data to train, and the resultant rules are “black boxes” that are hardly interpretable (Mnih et al., 2013; Heuillet et al., 2021) nor stable under input domain shifts (Schmeckpeper et al., 2020).

Original: To mitigate this gap, let us take inspirations from ourselves.
Revised: To mitigate this gap, let us take inspirations from ourselves.

Original: Humans integrate information at multiple levels of abstraction: from raw sensory, to concept symbols, and to more underlying
Revised: Humans integrate information at multiple levels of abstraction: from raw sensory, to concept symbols, and to more underlying

Original: “computational” mechanisms (Timpf et al., 1992).
Revised: “computational” mechanisms (Timpf et al., 1992).

Original: The hierarchical and progressive integration of concept abstraction serves as the key for intermediate states to enable interpretable logical reasoning as well as for generalization: the latter because appropriate abstraction can eliminate nuisances (Krawczyk et al., 2011).
Revised: The hierarchical and progressive integration of concept abstraction serves as the key for intermediate states to enable interpretable logical reasoning as well as for generalization: the latter because appropriate abstraction can eliminate nuisances (Krawczyk et al., 2011).

Original: Therefore, we are motivated to program our visual RL policies into an alternative representation of symbolic rules.
Revised: Therefore, we are motivated to program our visual RL policies into an alternative representation of symbolic rules.

Original: Compared to its numerically parameterized counterparts, a symbolic rule is signiﬁcantly lighter-weight, likely more interpretable, and potentially more generalizable.
Revised: Compared to its numerically parameterized counterparts, a symbolic rule is signiﬁcantly lighter-weight, likely more interpretable, and potentially more generalizable.

Original: Abstracting pure symbolic rules from data is not new.
Revised: Abstracting pure symbolic rules from data is not new.

Original: The family of symbolic regression (SR) methods (Cranmer et al., 2020; Runarsson & Jonsson, 2000; Gustafson et al., 2005; Orchard & Wang, 2016) have been studied toto directly search from the discrete symbolic space to compose target equations.
Revised: The family of symbolic regression (SR) methods (Cranmer et al., 2020; Runarsson & Jonsson, 2000; Gustafson et al., 2005; Orchard & Wang, 2016) have been studied to directly search from the discrete symbolic space to compose target equations.

Original: Different from conventional regression techniques that optimize a pre-speciﬁed model structure,
Revised: Different from conventional regression techniques that optimize a pre-speciﬁed model structure,

Original: SR infers both model structures and parameters from data.
Revised: SR infers both model structures and parameters from data.

Original: Most popular SR algorithms rely on genetic programming to evolve symbolic math equations from scratch (Runarsson & Jonsson, 2000;
Revised: Most popular SR algorithms rely on genetic programming to evolve symbolic math equations from scratch (Runarsson & Jonsson, 2000;

Original: Gustafson et al., 2005; Orchard & Wang, 2016).
Revised: Gustafson et al., 2005; Orchard & Wang, 2016).

Original: However, compared to the differentiable learning of numerical policy networks, those SR methods are slow on large/complicated problems and rely on many heuristics to work (Runarsson & Jonsson, 2000; Gustafson et al., 2005; Orchard & Wang, 2016).
Revised: However, compared to the differentiable learning of numerical policy networks, those SR methods are slow on large/complicated problems and rely on many heuristics to work (Runarsson & Jonsson, 2000; Gustafson et al., 2005; Orchard & Wang, 2016).

Original: Hence, naively plugging those methods as policy learners for visual RL, e.g., to initialize a population of simple symbolic policies and then gradually mutate/select better candidates while interacting with the environment, will be computationally prohibitive (Runarsson & Jonsson, 2000;
Revised: Hence, naively plugging those methods as policy learners for visual RL, e.g., to initialize a population of simple symbolic policies and then gradually mutate/select better candidates while interacting with the environment, will be computationally prohibitive (Runarsson & Jonsson, 2000;

Original: Gustafson et al., 2005; Orchard & Wang, 2016), in addition to suffering from the sparse rewards.
Revised: Gustafson et al., 2005; Orchard & Wang, 2016), in addition to suffering from the sparse rewards.

Original: We thus propose to synergize the numerical and symbolic policy learning, hoping to gain the best from both worlds.
Revised: We thus propose to synergize the numerical and symbolic policy learning, hoping to gain the best from both worlds.

Original: Inspired by the idea of “knowledge distillation” (Hinton et al., 2015), we propose a symbolic policy distillation approach: it does not directly compose and evolve the symbolic policy through RL interaction, but instead, distills a symbolic policy from the behavior of a numerical policy network as the “teacher”.
Revised: Inspired by the idea of “knowledge distillation” (Hinton et al., 2015), we propose a symbolic policy distillation approach: it does not directly compose and evolve the symbolic policy through RL interaction, but instead, distills a symbolic policy from the behavior of a numerical policy network as the “teacher”.

Original: After the teacher is learned via conventional RL, our approach extracts a symbolic form that approximates the relationship between the input visual observations and the output actions of the teacher.
Revised: After the teacher is learned via conventional RL, our approach extracts a symbolic form that approximates the relationship between the input visual observations and the output actions of the teacher.

Original: A key barrier here is, however, the fact that the teacher’s input is an image, while a symbolic policy cannot easily consume a high-dimension continuous input.
Revised: A key barrier here is, however, the fact that the teacher’s input is an image, while a symbolic policy cannot easily consume a high-dimension continuous input.

Original: A robust parsing mechanism is thus needed to abstract an image into a set of symbolic input operands, on top of which more symbolic operators can be deﬁned to compose them.
Revised: A robust parsing mechanism is thus needed to abstract an image into a set of symbolic input operands, on top of which more symbolic operators can be deﬁned to compose them.

Original: That requires us to cross the famous “semantic gap” (Hein, 2010) in vision, by properly discretizing the perception-decision procedure.
Revised: That requires us to cross the famous “semantic gap” (Hein, 2010) in vision, by properly discretizing the perception-decision procedure.

Original: In view of the above challenges, we study the progressive discrete symbolization of representations, from the continuous visual observations to the discrete actions, and accordingly design a symbolic policy distillation procedure that leverages the resultant hierarchical symbols.
Revised: In view of the above challenges, we study the progressive discrete symbolization of representations, from the continuous visual observations to the discrete actions, and accordingly design a symbolic policy distillation procedure that leverages the resultant hierarchical symbols.

Original: The progressive symbolization for visual RL bridges the raw image input, the desired symbolic input operands, and the output action space.
Revised: The progressive symbolization for visual RL bridges the raw image input, the desired symbolic input operands, and the output action space.

Original: To achieve so, we ﬁrst use an object detection module to achieve the “ﬁrst level” of discretization, i.e., from the input image to a discrete set of objects (both classes and locations).
Revised: To achieve so, we ﬁrst use an object detection module to achieve the “ﬁrst level” of discretization, i.e., from the input image to a discrete set of objects (both classes and locations).

Original: We then proceed to the “second level”, to (iteratively) search the symbolic rule that maps the input set of objects to the output set of actions.
Revised: We then proceed to the “second level”, to (iteratively) search the symbolic rule that maps the input set of objects to the output set of actions.

Original: The second level is accomplished using a symbolic teacher-student distillation algorithm named RoundTourMix .
Revised: The second level is accomplished using a symbolic teacher-student distillation algorithm named RoundTourMix .

Original: Our entire process yields a hierarchical, explainable, and generalizable symbolic rule in the form of symbolic compositions, that can be treated as discrete and abstracted representations of the visual policy network.
Revised: Our entire process yields a hierarchical, explainable, and generalizable symbolic rule in the form of symbolic compositions, that can be treated as discrete and abstracted representations of the visual policy network.

Original: As observed through extensive experiments, our symbolic policy distillation approach captures the “causal structure” through data-driven experience, and meanwhile gets rid of visual domain nuisances through state abstraction (e.g., detection).
Revised: As observed through extensive experiments, our symbolic policy distillation approach captures the underlying structure in a white-box form through data-driven experience, and meanwhile gets rid of visual domain nuisances through state abstraction (e.g., detection).

Original: In particular, the distilled symbolic policy leverages the representation of object location, velocity and class information, which naturally group two scenarios (visual scenes) together if the same logical speciﬁcations are congruent even if the scenes’ pixel-level attributes are markedly different.
Revised: In particular, the distilled symbolic policy leverages the representation of object location, velocity and class information, which naturally group two scenarios (visual scenes) together if the same logical speciﬁcations are congruent even if the scenes’ pixel-level attributes are markedly different.

Original: The distillation procedure also requires orders of magnitude less data than policy neural network training.
Revised: The distillation procedure also requires orders of magnitude less data than policy neural network training.

Original: Our approach paves a new way to make the visual RL more interpretable, reliable and generalizable, and also narrows the gap between numerical and symbolic approaches in RL.
Revised: Our approach paves a new way to make the visual RL more interpretable, reliable and generalizable, and also narrows the gap between numerical and symbolic approaches in RL.

Original: Our concrete technical contributions are summarized as follows:
Revised: Our concrete technical contributions are summarized as follows:

Original: • We propose a novel progressive discretization of the perception-decision procedure with three logical hierarchies: from a raw image to its object-level abstraction (called “geometric symbols” hereinafter), to the numerical state and discrete action spaces.
Revised: • We propose a novel progressive discretization of the perception-decision procedure with three logical hierarchies: from a raw image to its object-level abstraction (called “geometric symbols” hereinafter), to the numerical state and discrete action spaces.

Original: • We propose a novel policy regression algorithm that distills the teacher behaviors of policy networks into symbolic rules.
Revised: • We propose a novel policy regression algorithm that distills the teacher behaviors of policy networks into symbolic rules.

Original: Our approach termed RoundTourMix is gradient-free and data-efﬁcient to train, and the output rule enjoys good interpretability and reliability.
Revised: Our approach termed RoundTourMix is gradient-free and data-efﬁcient to train, and the output rule enjoys good interpretability and reliability.

Original: • Through performance/interpretability/transferability evaluation, our distilled symbolic policies demonstrate comparable or even stronger effectiveness in seven highly challenging visual RL environments.
Revised: • Through performance/interpretability/transferability evaluation, our distilled symbolic policies demonstrate comparable or even stronger effectiveness in seven highly challenging visual RL environments.

Original: They achieve the comparative or higher scores with interpretable actions while generalizing better on the new environments than their teacher policy networks.
Revised: They achieve the comparative or higher scores with interpretable actions while generalizing better on the new environments than their teacher policy networks.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Our work tightly intersects with interpretable reinforcement learning, and mathematical symbolic regression.
Revised: Our work tightly intersects with interpretable reinforcement learning, and mathematical symbolic regression.

Original: In reinforcement learning, the way to boost interpretability can be roughly categorized into (a) intrinsic interpretability based methods and (b) post-hoc explanation based methods.
Revised: In reinforcement learning, the way to boost interpretability can be roughly categorized into (a) intrinsic interpretability based methods and (b) post-hoc explanation based methods.

Original: The
Revised: The

Original: Intrinsic interpretability based methods require the learned models to be self-understandable by learning over interpretable architectures.
Revised: Intrinsic interpretability based methods require the learned models to be self-understandable by learning over interpretable architectures.

Original: Among the existing methods, the authors in Lyu et al.
Revised: Among the existing methods, the authors in Lyu et al.

Original: (2019) proposed to learn an interpretable policy online, Zambaldi et al.
Revised: (2019) proposed to learn an interpretable policy online, Zambaldi et al.

Original: (2018); Jiang & Luo (2019); Dong et al.
Revised: ; Jiang & Luo (2019); Dong et al.

Original: (2019) to introduce architectural inductive biases to learn a interpretable policy, and Ma et al.
Revised: (2019) to introduce architectural inductive biases to learn a interpretable policy, Dittadi et al.

Revised: (2020) to perform width-based planning on relevant features extracted from game frames, and Ma et al.

Original: (2021) preserves interpretability by learning a set of logical rules.
Revised: (2021) preserves interpretability by learning a set of logical rules.

Original: Other methods also leverage imitation learning (Verma et al., 2018; 2019; Bastani et al., 2018).
Revised: Other methods also leverage imitation learning (Verma et al., 2018; 2019; Bastani et al., 2018).

Original: For post-hoc explanation based methods, the papers in Zahavy et al.
Revised: For post-hoc explanation based methods, the papers in Zahavy et al.

Original: ; Greydanus et al.
Revised: ; Greydanus et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: (2019) propose to use t-SNE and saliency map to explain the learned policy.
Revised: (2019) propose to use t-SNE and saliency map to explain the learned policy.

Revised: (Liang et al., 2015) proposes to capture key features of a DQN into practical linear representations.

Original: Other methods include attention based method (Shi et al., 2020), visual summary based method (Sequeira & Gervasio, 2020), reward decomposition based method (Juozapaitis et al., 2019), casual model based method (Madumal et al., 2020) and Markov chain based method (Topin & Veloso, 2019).
Revised: Other methods include attention based method (Shi et al., 2020), visual summary based method (Sequeira & Gervasio, 2020), reward decomposition based method (Juozapaitis et al., 2019), casual model based method (Madumal et al., 2020), Markov chain based method (Topin & Veloso, 2019) and case-based expert-behaviour retrieval method (Ontan ´ on et al., 2007).

Original: Our symbolic distillation falls within the post-hoc methods, as we ﬁrst learn a CNN policy that enjoys the neural network’s free form optimization, then distills it into an explainable symbolic policy.
Revised: Our symbolic distillation falls within the post-hoc methods, as we ﬁrst learn a CNN policy that enjoys the neural network’s free form optimization, then distills it into an explainable symbolic policy.

Original: Connecting to existing methods that follow this path, the work in (Coppens et al., 2019) distill the learned policy into a soft decision tree.
Revised: Connecting to existing methods that follow this path, the work in (Coppens et al., 2019) distill the learned policy into a soft decision tree.

Original: Their visual observations are coarsely quantized into 10 ×cells, and the soft decision tree policy is learned over the 100 dimensional space.
Revised: Their visual observations are coarsely quantized into 10 ×cells, and the soft decision tree policy is learned over the 100 dimensional space.

Original: Garnelo et al.
Revised: Garnelo et al.

Original: (2016); Garcez et al.
Revised: (2016); Garcez et al.

Original: (2018) also learn symbolic RL algorithms, but the states of the environments they considered are clean and simple, and are not as challenging as video games.
Revised: (2018) also learn symbolic RL algorithms, but the states of the environments they considered are clean and simple, and are not as challenging as video games.

Original: Symbolic Regression (SR) (Runarsson & Jonsson, 2000; Orchard & Wang, 2016; Schmidt & Lipson, 2009; Cranmer et al., 2020; Cranmer, 2020; Petersen et al., 2019; Gustafson et al., 2005) is a recently emerging and promising approach for discovering underlying symbolic rules of the observed data.
Revised: Symbolic Regression (SR) (Runarsson & Jonsson, 2000; Orchard & Wang, 2016; Schmidt & Lipson, 2009; Cranmer et al., 2020; Cranmer, 2020; Petersen et al., 2019; Gustafson et al., 2005) is a recently emerging and promising approach for discovering underlying symbolic rules of the observed data.

Original: In discovering the mathematical equations, several algorithms adopted the gradient-free genetic programming method.
Revised: In discovering the mathematical equations, several algorithms adopted the gradient-free genetic programming method.

Original: These algorithms generate populations of candidates equations, mutate them and select better candidates, then repeat this evolution procedure until certain thresholds of maximum interation number or performance level are met.
Revised: These algorithms generate populations of candidates equations, mutate them and select better candidates, then repeat this evolution procedure until certain thresholds of maximum interation number or performance level are met.

Original: Examples include Eureqa (Schmidt & Lipson, 2009) and PySR (Cranmer et al., 2020; Cranmer, 2020), and a few others (Runarsson & Jonsson, 2000;
Revised: Examples include Eureqa (Schmidt & Lipson, 2009) and PySR (Cranmer et al., 2020; Cranmer, 2020), and a few others (Runarsson & Jonsson, 2000;

Original: Orchard & Wang, 2016; Gustafson et al., 2005).
Revised: Orchard & Wang, 2016; Gustafson et al., 2005).

Original: Some other methods leverage the deep neural network to parse out equation trees, such as (Petersen et al., 2019; Bello et al., 2017).
Revised: Some other methods leverage the deep neural network to parse out equation trees, such as (Petersen et al., 2019; Bello et al., 2017).

Original:  T HREE T YPES O F I MAGE D ATA R EPRESENTATIONS
Revised:  T HREE T YPES O F I MAGE D ATA R EPRESENTATIONS

Original: In this work, we focus on the visual Reinforcement Learning (RL) domain, and distill the teacher agent behavior into a symbolic form.
Revised: In this work, we focus on the visual Reinforcement Learning (RL) domain, and distill the teacher agent behavior into a symbolic form.

Original: We use the Gym Retro games (Nichol et al.)
Revised: We use the Gym Retro games (Nichol et al.)

Original: as primary examples to study policy distillation.
Revised: as primary examples to study policy distillation.

Original: For the most common visual RL case, the agent is modeled by an end-to-end neural network (usually based on a convolutional neural network, or CNN) that maps the image type data to the action.
Revised: For the most common visual RL case, the agent is modeled by an end-to-end neural network (usually based on a convolutional neural network, or CNN) that maps the image type data to the action.

Original: As pixel level data representation, the raw image offers rich low level information but without abstraction.
Revised: As pixel level data representation, the raw image offers rich low level information but without abstraction.

Original: On the other hand, as a high level decision regime, the symbolic rule takes robust abstracted symbols as its input operands.
Revised: On the other hand, as a high level decision regime, the symbolic rule takes robust abstracted symbols as its input operands.

Original: To bridge this gap, it is hence necessary to summarize the image pixels into symbolic operands (also refered to as geometric objects in this work) as required by the symbolic policy.
Revised: To bridge this gap, it is hence necessary to summarize the image pixels into symbolic operands (also refered to as geometric objects in this work) as required by the symbolic policy.

Original: Besides the object level abstraction of the raw image, richer numerical form representations are also needed, as the numbers are the basic building blocks to effectively represent knowledge and conduct learning.
Revised: Besides the object level abstraction of the raw image, richer numerical form representations are also needed, as the numbers are the basic building blocks to effectively represent knowledge and conduct learning.

Original: Summarizing these motivations, we introduce three types of representations, the visual observation , the geometric symbol representation , and the numerical state .
Revised: Summarizing these motivations, we introduce three types of representations, the visual observation , the geometric symbol representation , and the numerical state .

Original: The initially obtainable representation is the visual observation , which contains the raw information and serves as the input for the CNN based model.
Revised: The initially obtainable representation is the visual observation , which contains the raw information and serves as the input for the CNN based model.

Original: With these visual observations, we run a (pretrained and ﬁxed) object detection algorithm to obtain object lists in the images.
Revised: With these visual observations, we run a (pretrained and ﬁxed) object detection algorithm to obtain object lists in the images.

Original: The object detection module will provide us the class, location, as well as the velocity of every single detected object in the image sequence.
Revised: The object detection module will provide us the class, location, as well as the velocity of every single detected object in the image sequence.

Original: The output of the object detection module is the second type of image data representations, which we term as the geometric symbol representation .
Revised: The output of the object detection module is the second type of image data representations, which we term as the geometric symbol representation .

Original: The geometric symbol representation is the desired symbol abstraction of the pixel level raw image, and supplies the input operands for the symbolic distillation to work on.
Revised: The geometric symbol representation is the desired symbol abstraction of the pixel level raw image, and supplies the input operands for the symbolic distillation to work on.

Original: We use optical ﬂow to segment and estimate the velocity of the objects, and further classify the objects based on their shapes.
Revised: We use optical ﬂow to segment and estimate the velocity of the objects, and further classify the objects based on their shapes.

Original: More details are in
Revised: More details are in

Original: Appendix A. Eventually, the third type of representation is the numerical state , which is obtained by drawing auxiliary lines, and measuring/quantizing certain relationships of the geometric symbols (distances, orientations, etc).
Revised: Appendix B. Eventually, the third type of representation is the numerical state , which is obtained by drawing auxiliary lines, and measuring/quantizing certain relationships of the geometric symbols (distances, orientations, etc).

Original: The numerical state is then used to predict the ﬁnal action.
Revised: The numerical state is then used to predict the ﬁnal action.

Original: Overall as illustrated in Fig.
Revised: Overall as illustrated in Fig.

Original: 2, in order to parse a raw image into a symbolic policy, the perception-decision procedure is discretized into the following steps: visual observation → geometric symbol representation → numerical state → action.
Revised: 2, in order to parse a raw image into a symbolic policy, the perception-decision procedure is discretized into the following steps: visual observation → geometric symbol representation → numerical state → action

Original:  L EARNING T O D ISTILL T EACHER B EHAVIOR I NTO S YMBOLIC P OLICY
Revised:  L EARNING T O D ISTILL T EACHER B EHAVIOR I NTO S YMBOLIC P OLICY

Original: With the discretized decision procedure deﬁned above, the remaining question is how to concretely distill the teacher behavior into the symbolic policy.
Revised: With the discretized decision procedure deﬁned above, the remaining question is how to concretely distill the teacher behavior into the symbolic policy.

Original: In what follows, we summarize the learning/distillation objectives in section 4.1, then describe the components of symbolic search space in section 4.2, and ﬁnally describe the distillation algorithm in section 4.3.
Revised: In what follows, we summarize the learning/distillation objectives in section 4.1, then describe the components of symbolic search space in section 4.2, and ﬁnally describe the distillation algorithm in section 4.3.

Original: 4. T HE L EARNING O BJECTIVES
Revised: 4. T HE L EARNING O BJECTIVES

Original: Over the three representations discussed above, a hierarchical symbolic reasoning procedure can be built.
Revised: Over the three representations discussed above, a hierarchical symbolic reasoning procedure can be built.

Original: In this work, we posit that the step [visual observation → geometric symbol representation] could be robustly implemented and is reliable for the usage of future steps, therefore, we set it as learning-less.
Revised: In this work, we posit that the step [visual observation → geometric symbol representation] could be robustly implemented and is reliable for the usage of future steps, therefore, we set it as learning-less.

Original: Based on the observed geometric symbol representation, we learn the rest of reasoning steps [geometric symbol representation → numerical state → action] through the proposed symbolic distillation.
Revised: Based on the observed geometric symbol representation, we learn the rest of reasoning steps [geometric symbol representation → numerical state → action] through the proposed symbolic distillation.

Original: Intuitively, there are three reasoning components subject to learning: (cid:202) what geometric relationship to measure out of the geometric symbol combinations, so as to compose the numerical state; (cid:203) what is the condition to take action based on the measured numerical state; and (cid:204) how to take action , under the current condition.
Revised: Intuitively, there are three reasoning components subject to learning: (cid:202) what geometric relationship to measure out of the geometric symbol combinations, so as to compose the numerical state; (cid:203) what is the condition to take action based on the measured numerical state; and (cid:204) how to take action , under the current condition.

Original: In this research, the learning targets (cid:203) and (cid:204) are usually found to be stacked with each other in a hierarchical way, yielding a tree-shapedsymbolic decision procedure (Botvinick et al., 2009).
Revised: In this research, the learning targets (cid:203) and (cid:204) are usually found to be stacked with each other in a hierarchical way, yielding a tree-shaped decision procedure (Sutton et al., 1999; Dietterich, 2000; Botvinick et al., 2009).

Original: To connect these learning targets with the discretized decision procedures in section 3,  the learning target (cid:202) rules the [geometric symbols → numerical state] step, andthe targets (cid:203) and (cid:204) rule the [numerical state → action] step.
Revised: To connect these learning targets with the discretized decision procedures in section 3, (cid:202) determines the learning target for the [geometric symbols → numerical state] step, and (cid:203) and (cid:204) determine the learning target for the [numerical state → action] step.

Original: In the following section, we deﬁne and construct the basic building block to compose these learning targets.
Revised: In the following section, we deﬁne and construct the basic building block to compose these learning targets.

Original: 4. S YMBOLIC P OLICY S EARCH S PACE
Revised: 4. S YMBOLIC P OLICY S EARCH S PACE

Original: To make symbolic policy distillation executable, we have to answer two open questions: what are the available building blocks to construct symbolic policy, and how can we construct it?
Revised: To make symbolic policy distillation executable, we have to answer two open questions: what are the available building blocks to construct symbolic policy, and how can we construct it?

Original: In the following, we answer them by deﬁning the symbolic search space, which contains the geometric operator search space and the numerical operator search space.
Revised: In the following, we answer them by deﬁning the symbolic search space, which contains the geometric operator search space and the numerical operator search space.

Original: The geometric operator search space .
Revised: The geometric operator search space .

Original: The geometric operator search space is designed for the learning target (cid:202) in section 4.1, and the components within this space are geometric operator combinations applied to detected objects.
Revised: The geometric operator search space is designed for the learning target (cid:202) in section 4.1, and the components within this space are geometric operator combinations applied to detected objects.

Original: To enable stronger expressiveness, we include two types of geometric operators: the auxiliary line drawer and the geometric attribute measurer .
Revised: To enable stronger expressiveness, we include two types of geometric operators: the auxiliary line drawer and the geometric attribute measurer .

Original: The auxiliary line drawer takes the input of detected objects, and draw auxiliary lines according to certain given descriptions.
Revised: The auxiliary line drawer takes the input of detected objects, and draw auxiliary lines according to certain given descriptions.

Original: This type of operators include velocity extension (extend the line along certain object’s velocity direction, and keep track of it), static line drawer (drawing static vertical/horizontal/tilt auxiliary lines), and intersection marker (mark the intersection between the auxiliary lines and/or detected objects, and keep track of it).
Revised: This type of operators include velocity extension (extend the line along certain object’s velocity direction, and keep track of it), static line drawer (drawing static vertical/horizontal/tilt auxiliary lines), and intersection marker (mark the intersection between the auxiliary lines and/or detected objects, and keep track of it).

Original: The newly drawn auxiliary lines/intersection points are added into the collection of the geometric symbol representation.
Revised: The newly drawn auxiliary lines/intersection points are added into the collection of the geometric symbol representation.

Original: For the second type of geometric operators, the geometric attribute measurer is responsible for evaluating the exact geometric relationships given speciﬁed geometric objects.
Revised: For the second type of geometric operators, the geometric attribute measurer is responsible for evaluating the exact geometric relationships given speciﬁed geometric objects.

Original: The evaluated geometric relationships could be real/boolean values, or another geometric object.
Revised: The evaluated geometric relationships could be real/boolean values, or another geometric object.

Original: The real valued operators include velocity of (certain given object), location of (certain given object), distance between (two given objects), orientation between (two objects against each other).
Revised: The real valued operators include velocity of (certain given object), location of (certain given object), distance between (two given objects), orientation between (two objects against each other).

Original: The boolean valued operators include has intersection (detect intersection between two objects or an object and an auxiliary line), is moving , is changing shape (inspect one given object).
Revised: The boolean valued operators include has intersection (detect intersection between two objects or an object and an auxiliary line), is moving , is changing shape (inspect one given object).

Original: Another special operator is find nearest object , which returns the nearest
Revised: Another special operator is find nearest object , which returns the nearest

Original: class-
Revised: class-

Original: i ( i is given) object from the protagonist in the current observation, and the attributes of this object could be further measured by other geometric attribute measurer.
Revised: i ( i is given) object from a reference object in the current observation, and the attributes of this object could be further measured by other geometric attribute measurer.

Original: This operator is found to be frequently used, as it is usually the case that when multiple objects are present, the agent tends to deal with the nearest object ﬁrst.
Revised: This operator is found to be frequently used, as it is usually the case that when multiple objects are present, the agent tends to deal with the nearest object ﬁrst.

Original: The numerical operator search space .
Revised: The numerical operator search space .

Original: In correspondence with the geometric search space, the numerical search space is designed for learning targets (cid:203) and (cid:204) , and its components are numerical operator combinations applied to the variables in the measured numerical state.
Revised: In correspondence with the geometric search space, the numerical search space is designed for learning targets (cid:203) and (cid:204) , and its components are numerical operator combinations applied to the variables in the measured numerical state.

Original: There are also two types of numerical operators.
Revised: There are also two types of numerical operators.

Original: The ﬁrst type is the bool function/logical operators, which include is ( x < y ), is ( x ≤ y ), is ( x = y ), a | b , a & b , ¬ a , and conditional combinations: a * x +( ¬ a )
Revised: The ﬁrst type is the bool function/logical operators, which include is ( x < y ), is ( x ≤ y ), is ( x = y ), a | b , a & b , ¬ a , and conditional combinations: a * x +( ¬ a )

Original: * y .
Revised: * y .

Original: The second type is the math operators, which include + , − , ∗ , /, sin , cos , tan , cot , ( · ) 2 , ( · ) 3 , √· , exp , log .
Revised: The second type is the math operators, which include + , − , ∗ , /, sin , cos , tan , cot , ( · ) 2 , ( · ) 3 , √· , exp , log .

Original: To illustrate with two concrete examples, in the environment CircusCharlie (illustrated in Fig.and Fig. 2), the agent’s target is to jump right into the ﬁre ring and over the pot.
Revised: To illustrate with two concrete examples, in the environment CircusCharlie (illustrated in Fig.and Fig. 2), the agent’s target is to jump right into the ﬁre ring and over the pot.

Original: One sample distilled symboblic policy is presented in Fig 3.
Revised: One sample distilled symboblic policy is presented in Fig 3.

Original: The equation 1 also represents a subtree of Figfor simplicity.
Revised: The equation 1 also represents a subtree of Figfor simplicity.

Original: In Eq 1, x ring , x pot and x 0 are the x -coordinate of the detected ﬁre ring, pot and the protagonist objects.
Revised: In Eq 1, x ring , x pot and x 0 are the x -coordinate of the detected ﬁre ring, pot and the protagonist objects.

Original: The distilled policy allows the agent to only jump when the condition ¬ (10 < x pot − x ring < 50) ×
Revised: The distilled policy allows the agent to only jump when the condition ¬ (10 < x pot − x ring < 50) ×

Original: ( x ring − x 0 < 40) is met.
Revised: ( x ring − x 0 < 40) is met.

Original: In Pong (illustrated in Fig. 1), the agent needs to control the right racket so as to hit the pong, which can collide and bounce with the walls.
Revised: In Pong (illustrated in Fig. 1), the agent needs to control the right racket so as to hit the pong, which can collide and bounce with the walls.

Original: One sample distilled symbolic policy is in Eq.
Revised: One sample distilled symbolic policy is in Eq.

Original: 2, where x pong is the x coordinate of pong, S is the intersection of the speed orientation and the right edge (drawn and tracked by the auxiliary line drawer ).
Revised: 2, where x pong is the x coordinate of pong, S is the intersection of the speed orientation and the right edge (drawn and tracked by the auxiliary line drawer ).

Original: Such policy allows the agent to ignore the bouncing procedure of the pong, only pay attention to the speed orientation of the pong when it is near the edge ( x pong > 40 ).
Revised: Such policy allows the agent to ignore the bouncing procedure of the pong, only pay attention to the speed orientation of the pong when it is near the edge ( x pong > 40 ).

Original: More examples of distilled policies and their interpretations are in Appendix.
Revised: More examples of distilled policies and their interpretations are in Appendix.

Original: B.
Revised: C.

Original: One possible difﬁculty of the symbolic distillation procedure is the huge size of the search space: it grows exponentially with longer time span.
Revised: One possible difﬁculty of the symbolic distillation procedure is the huge size of the search space: it grows exponentially with longer time span.

Original: Denote the number of symbolic compositions in one frame as N .
Revised: Denote the number of symbolic compositions in one frame as N .

Original: If one consider stacking T consecutive frames without duplicate removal or other engineering tricks, the total number of symbolic compositions will be N T .
Revised: If one consider stacking T consecutive frames without duplicate removal or other engineering tricks, the total number of symbolic compositions will be N T .

Original: To reduce the size of the search space, we primarily focus on single-frame policy distillation, i.e., infer the action only based on the current frame observation.
Revised: To reduce the size of the search space, we primarily focus on single-frame policy distillation, i.e., infer the action only based on the current frame observation.

Original: For the environments that require multi-frame tracking, those long-term informations could be attached to the object as an attributes of the object, and are carried and updated across time.
Revised: For the environments that require multi-frame tracking, those long-term informations could be attached to the object as an attributes of the object, and are carried and updated across time.

Original: For example, in the shooting based environments, whether an object has been set as the “next shooting target” by the teacher agent could be set as an attribute of this object, and this attribute could be measured by observing the teacher behavior, and backtrack in time to mark previous frames (backtrack is allowed during distillation, since our distillation algorithm works with the ofﬂine stored teacher behavior dataset), and could be presented as a component in the measured numerical state.
Revised: For example, in the shooting based environments, whether an object has been set as the “next shooting target” by the teacher agent could be set as an attribute of this object, and this attribute could be measured by observing the teacher behavior, and backtrack in time to mark previous frames (backtrack is allowed during distillation, since our distillation algorithm works with the ofﬂine stored teacher behavior dataset), and could be presented as a component in the measured numerical state.

Original: Under this strategy, the size of the symbolic search space is greatly reduced, and a plenty of environments are found to be viable for symbolic distillation (approximately 85% of 1000+ gym retro environments).
Revised: Under this strategy, the size of the symbolic search space is greatly reduced, and a plenty of environments are found to be viable for symbolic distillation (approximately 85% of 1000+ gym retro environments).

Original: A sampled list of such environments are put in Appendix C. policy-CircusCharlie 
Revised: A sampled list of such environments are put in Appendix D. policy-CircusCharlie =

Original: [Equation]
Revised: [Equation]

Revised: left

Original: [Equation]
Revised: [Equation]

Original: policy-pong =( x pong > 40) × (( y pong > y S ) × down + ¬ ( y pong > y S ) × up
Revised: right

Original: [Equation]
Revised: [Equation]

Original: action
Revised: 1) policy-pong =( x pong > 40) × (( y pong > y S ) × down + ¬ ( y pong > y S ) × up ) + ¬ ( x pong >

Original: [Equation]
Revised: [Equation]

Revised: × no action (2)

Original: 4. R OUND T OUR M IX : T HE P ROCEDURE T O D ISTILLT HE S YMBOLIC P OLICY
Revised: 4. R OUND T OUR M IX : T HE P ROCEDURE T O D ISTILL

Revised: T HE S

Revised: YMBOLIC

Revised: P OLICY

Original: [Figure]
Revised: [Figure]

Original: Having established the search space and clariﬁed the learning targets, in this section, we propose the RoundTourMix: a gradient-free policy distillation approach that iteratively optimize the three learning target discussed in Section 4.1, by switching back and forth between observing teacher’s behavior and interacting with the environments.
Revised: Having established the search space and clariﬁed the learning targets, in this section, we propose the RoundTourMix: a gradient-free policy distillation approach that iteratively optimize the three learning target discussed in Section 4.1, by switching back and forth between observing teacher’s behavior and interacting with the environments.

Original: It follows the straightforward philosophy of genetic programming: a mainline guess of symbolic policy tree is maintained, new variants are repeatedly mutated from the mainline, and those better variants which minimize the loss with the teacher’s actions are captured – for continuous action space, we use Mean Squared
Revised: It follows the straightforward philosophy of genetic programming: a mainline guess of symbolic policy tree is maintained, new variants are repeatedly mutated from the mainline, and those better variants which minimize the loss with the teacher’s actions are captured.

Original: Error loss, and for discrete action space, we take the Cross-Entropy loss as the metric.
Revised: We take the Cross-Entropy loss as the metric under the discrete action space we tested.

Revised: We also note the choice to use Mean Squared Error loss if the action space is continuous.

Original: Through this evolution procedure, the more complex and better performing mutations of symbolic policy are iteratively discovered.
Revised: Through this evolution procedure, the more complex and better performing mutations of symbolic policy are iteratively discovered.

Original: We note that the decision procedure of a wide range of CNN policy networks could be efﬁciently represented as high-ﬁdelity tree shaped symbolic policy similar to Fig.
Revised: We note that the decision procedure of a wide range of CNN policy networks could be efﬁciently represented as high-ﬁdelity tree shaped symbolic policy similar to Fig.

Original: Therefore, we extend it into a generalizable form, where there are condition node and the action node co-existed.
Revised: Therefore, we extend it into a generalizable form, where there are condition node and the action node co-existed.

Original: The condition node serves as the branching node, which corresponds to the learning target (cid:203) , and the action node serves as the leaf node that corresponds to the learning target (cid:204) .
Revised: The condition node serves as the branching node, which corresponds to the learning target (cid:203) , and the action node serves as the leaf node that corresponds to the learning target (cid:204) .

Original: The condition node has three key properties: the condition , a LEFT , and a RIGHT .
Revised: The condition node has three key properties: the condition , a LEFT , and a RIGHT .

Original: Given a sample of numerical state, the condition is a symbolic composition that evaluates into a boolean value.
Revised: Given a sample of numerical state, the condition is a symbolic composition that evaluates into a boolean value.

Original: a LEFT and a RIGHT are two of its children tree nodes, which can be either condition nodes or action nodes.
Revised: a LEFT and a RIGHT are two of its children tree nodes, which can be either condition nodes or action nodes.

Original: Given a numerical state, if the condition of this node is evaluated to be True, a LEFT will be executed, otherwise a RIGHT will be executed.
Revised: Given a numerical state, if the condition of this node is evaluated to be True, a LEFT will be executed, otherwise a RIGHT will be executed.

Original: The action node also have three key properties: the total condition , the depth , and the policy .
Revised: The action node also have three key properties: the total condition , the depth , and the policy .

Original: The total condition is the cascaded condition of its father condition nodes, in other words, given any sample state, the total condition of an action node is true only if the condition of all of its father nodes are met (either True or False), so that this action node can be reached along the path from the root node.
Revised: The total condition is the cascaded condition of its ancestral condition nodes, in other words, given any sample state, the total condition of an action node is true only if the condition of all of its ancestral nodes are met (either True or False), so that this action node can be reached along the path from the root node.

Original: The depth of the action node means the length of the path from the root node to the action node.
Revised: The depth of the action node means the length of the path from the root node to the action node.

Original: The policy can either be a single action, or a mapping from the state space (a subspace that already meets the total condition ) to the action space.
Revised: The policy can either be a single action, or a mapping from the state space (a subspace that already meets the total condition ) to the action space.

Original: In practice, we encourage the policy to be a single action, but if the tree search reached certain predeﬁned maximum depth, we also allow this policy to be the symbolic mapping recovered by symbolic regression (Cranmer et al., 2020; Cranmer, 2020), which is a subtree itself.
Revised: In practice, we encourage the policy to be a single action, but if the tree search reached certain predeﬁned maximum depth, we also allow this policy to be the symbolic mapping recovered by symbolic regression (Cranmer et al., 2020; Cranmer, 2020), which is a subtree itself.

Original: Algorithm: Distilling Teacher Behavior into Symbolic Tree
Revised: Algorithm: Distilling Teacher Behavior into Symbolic Tree

Original: Require: Temporary dataset D train containing
Revised: Require: Temporary dataset D train containing

Original: X (numerical states), Y (actions)
Revised: X (numerical states), Y (actions)

Original: Return: r : the root of symbolic policy tree
Revised: Return: r : the root of symbolic policy tree

Original: Maintain: S : the set of unsolved action nodes 1: Initializations 2: r ← newActionNode ( depth = 0) 3:
Revised: Maintain: S : the set of unsolved action nodes 1: Initializations 2: r ← newActionNode ( depth = 0) 3:

Original: S ← { r } ; cnt ←4:
Revised: S ← { r } ; cnt ←4:

Original: While S (cid:54) = {} & cnt < cnt MAX : 5: cnt ← cnt +6: n ← pop ( S ) (cid:3)
Revised: While S (cid:54) = {} & cnt < cnt MAX : 5: cnt ← cnt +6: n ← pop ( S ) (cid:3)

Original: Sample action node 7:
Revised: Sample action node 7:

Original: Y sub ← Y [ n. total condition] (cid:3) Slices 8:
Revised: Y sub ← Y [ n. total condition] (cid:3) Slices 8:

Original: IF Entropy ( Y sub ) <
Revised: IF Entropy ( Y sub ) <

Original: Θ entropy : 9: n. policy ← Mean ( Y sub ) 10:
Revised: Θ entropy : 9: n. policy ← Mean ( Y sub ) 10:

Original: ELSE: (cid:3)
Revised: ELSE: (cid:3)

Original: Single action cannot ﬁt 11:
Revised: Single action cannot ﬁt 11:

Original: IF n. depth < depth MAX : 12:
Revised: IF n. depth < depth MAX : 12:

Original: With probability p 1 : (cid:3) Split condition 13: n ← newConditionNode () 14:
Revised: With probability p 1 : (cid:3) Split condition 13: n ← newConditionNode () 14:

Original: S ← S + { n.a LEFT , n.a RIGHT } 15:
Revised: S ← S + { n.a LEFT , n.a RIGHT } 15:

Original: With probability 1 − p 1 : (cid:3) De-noise 16: n. policy ← default action 17:
Revised: With probability 1 − p 1 : (cid:3) De-noise 16: n. policy ← default action 17:

Original: ELSE: (cid:3) Too deep, stop branching further 18:
Revised: ELSE: (cid:3) Too deep, stop branching further 18:

Original: With probability p 2 : 19:
Revised: With probability p 2 : 19:

Original: X sub ← X
Revised: X sub ← X

Original: [ n. total condition] 20:
Revised: [ n. total condition] 20:

Original: n. policy ← runSR ( X sub , Y sub ) 21:
Revised: n. policy ← runSR ( X sub , Y sub ) 21:

Original: With probability p 3 : (cid:3) De-noise 22: n. policy ← default action 23:
Revised: With probability p 3 : (cid:3) De-noise 22: n. policy ← default action 23:

Original: With probability 1 − p 2 − p 3 : 24: n (cid:48)
Revised: With probability 1 − p 2 − p 3 : 24: n (cid:48)

Original: ← Sample ( pathToRoot ( n )) 25: removeSubtree ( n (cid:48) ) 26: n (cid:48) ← newConditionNode () 27:
Revised: ← Sample ( pathToRoot ( n )) 25: removeSubtree ( n (cid:48) ) 26: n (cid:48) ← newConditionNode () 27:

Original: S ← S + { n (cid:48) 
Revised: S ← S + { n (cid:48) .a

Revised: LEFT , n (cid:48) .a

Revised: RIGHT } 28: Return r

Original: Speciﬁcally, the RoundTourMix is composed of four mutually interactive stages:
Revised: Speciﬁcally, the RoundTourMix is composed of four mutually interactive stages:

Original: Record Teacher Behavior .
Revised: Record Teacher Behavior .

Original: In the ﬁrst stage, the teacher interacts with the environment, and the visual observations are fully recorded and converted to geometric symbols frame-by-frame.
Revised: In the ﬁrst stage, the teacher interacts with the environment, and the visual observations are fully recorded and converted to geometric symbols frame-by-frame.

Original: We align these frames of symbols with actions and store as an ofﬂine teacher behavior dataset D 

Revised: We align these frames of symbols with actions and store as an ofﬂine teacher behavior dataset D

Original: (no need to store teacher’s reward).
Revised: (no need to store teacher’s reward).

Original: This stage serve as the initialization of RoundTourMix, and is the only stage that will be executed only once.
Revised: This stage serve as the initialization of RoundTourMix, and is the only stage that will be executed only once.

Original: Guess and Observe .
Revised: Guess and Observe .

Original: In this stage, we make a random guess for the target geometric relations to be measured, then replay the teacher’s interactions in D 0 .
Revised: In this stage, we make a random guess for the target geometric relations to be measured, then replay the teacher’s interactions in D 0 .

Original: During the replay, the numerical states are evaluated using geometric operators, and are recorded (instead of measure all possible geometric relations and yield gigantic numerical states).
Revised: During the replay, the numerical states are evaluated using geometric operators, and are recorded (instead of measure all possible geometric relations and yield gigantic numerical states).

Original: We collect the numerical states and teacher’s actions as a temporary dataset D , and split it into a training set D train and a validation set D val .
Revised: We collect the numerical states and teacher’s actions as a temporary dataset D , and split it into a training set D train and a validation set D val .

Original: Solve .
Revised: Solve .

Original: In this stage, we iteratively search the symbolic policy tree composed of the numerical operators.
Revised: In this stage, we iteratively search the symbolic policy tree composed of the numerical operators.

Original: Table 1 presents the details of one round symbolic tree generation.
Revised: Table 1 presents the details of one round symbolic tree generation.

Original: This procedure is repeated for multiple times, after which a few of the best policy candidates on D val are collected and sent to the next stage.
Revised: This procedure is repeated for multiple times, after which a few of the best policy candidates on D val are collected and sent to the next stage.

Original: Verify and Optimize .
Revised: Verify and Optimize .

Original: In this stage, we let go the best few symbolic policies collected from last stage to interact with the environment.
Revised: In this stage, we let go the best few symbolic policies collected from last stage to interact with the environment.

Original: If any candidate symbolic policy is found to match or surpass the teacher’s score, the RoundTourMix procedure is terminated and the distilled symbolic policy is found.
Revised: If any candidate symbolic policy is found to match or surpass the teacher’s score, the RoundTourMix procedure is terminated and the distilled symbolic policy is found.

Original: Otherwise, return to the Guess and Observe stage and make new guess of which geometric relations to measure.
Revised: Otherwise, return to the Guess and Observe stage and make new guess of which geometric relations to measure.

Original: Remark 1: More Details in Stage 3.
Revised: Remark 1: More Details in Stage 3.

Original: The core part of the algorithm in Table.
Revised: The core part of the algorithm in Table.

Original: 1 is to repeatedly guess branching conditions of the condition nodes (lines 13/26), slice out the subset of data that obey the total condition (line 7), and ﬁt action nodes with this subset of data (lines 9/16/20/22).
Revised: 1 is to repeatedly guess branching conditions of the condition nodes (lines 13/26), slice out the subset of data that obey the total condition (line 7), and ﬁt action nodes with this subset of data (lines 9/16/20/22).

Original: The more accurate the guessed condition, the easier it is to ﬁt action node.
Revised: The more accurate the guessed condition, the easier it is to ﬁt action node.

Original: If the guessed condition happen to be near exact, the sliced teacher’s action tends to be nearly deterministic (i.e., low entropy, line 8, where Θ entropy is a threshold of the entropy).
Revised: If the guessed condition happen to be near exact, the sliced teacher’s action tends to be nearly deterministic (i.e., low entropy, line 8, where Θ entropy is a threshold of the entropy).

Original: Otherwise, if the maximum depth depth MAX is met and the leaf node still is not deterministic (line 17), then it has the option to directly ﬁt symbolic mapping (Cranmer, 2020) (line 20).
Revised: Otherwise, if the maximum depth depth MAX is met and the leaf node still is not deterministic (line 17), then it has the option to directly ﬁt symbolic mapping (Cranmer, 2020) (line 20).

Original: Remark 2: Teaching Behavior Denoising.
Revised: Remark 2: Teaching Behavior Denoising.

Original: An important mechanism in this algorithm is the denoising for the teacher behavior (lines 9/16/22).
Revised: An important mechanism in this algorithm is the denoising for the teacher behavior (lines 9/16/22).

Original: Under some conditions, the teacher model could possibly take noisy or “seemingly random” actions, which make it difﬁcult for the RoundTourMix to ﬁnd a single deterministic action that matches well (line 8 not satisﬁed).
Revised: Under some conditions, the teacher model could possibly take noisy or “seemingly random” actions, which make it difﬁcult for the RoundTourMix to ﬁnd a single deterministic action that matches well (line 8 not satisﬁed).

Original: In these cases, the distillation algorithm has the chance to assign the default action (line 16,22) to denoise it.
Revised: In these cases, the distillation algorithm has the chance to assign the default action (line 16,22) to denoise it.

Original: On other scenarios, the actions subset Y sub under certain total condition have low entropy but are not entirely deterministic, [Figure] single action (line 9), if the entropy is below threshold Θ entropy .
Revised: On other scenarios, the actions subset Y sub under certain total condition have low entropy but are not entirely deterministic, the symbolic distillation can still smooth it into a deterministic single action (line 9), if the entropy is below threshold Θ entropy .

Original: Eventually, with the aid of the last screening stage of RoundTourMix, the symbolic policy taking fewer but more efﬁcient actions are picked as the winner. 
Revised: Eventually, with the aid of the last screening stage of RoundTourMix, the symbolic policy taking fewer but more efﬁcient actions are picked as the winner. [Figure]

Original: Although this smoothing and denoising effect might potentially overlook some subtle behaviors of the teacher agent, we argue that the beneﬁts of introducing such mechanism outweigh the disadvantages.
Revised: of the teacher agent, we argue that the beneﬁts of introducing such mechanism outweigh the disadvantages.

Original: On one hand, symbolic rules can still be encouraged to evolve into more complex trees to capture intricate teacher behaviors, e.g., by setting the depth of the tree controllable by depth MAX , and the richness/complexity of the symbolic building blocks, as deﬁned by the search space.
Revised: On one hand, symbolic rules can still be encouraged to evolve into more complex trees to capture intricate teacher behaviors, e.g., by setting the depth of the tree controllable by depth MAX , and the richness/complexity of the symbolic building blocks, as deﬁned by the search space.

Original: On the other hand, not all environments require intricate manipulations.
Revised: On the other hand, not all environments require intricate manipulations.

Original: For a few environments that challenges on “balancing ability” (no need to involve much reasoning), such as CartPole or LunarLander, it is difﬁcult to distill reliable symbolic policy.
Revised: For a few environments that challenges on “balancing ability” (no need to involve much reasoning), such as CartPole or LunarLander, it is difﬁcult to distill reliable symbolic policy.

Original: However, the visual reasoning tasks dominates the gym retro environments, and in these tasks, the denoised symbolic rules have demonstrated quite unbeatable competence across a host of them, as to be seen in section 5.
Revised: However, the visual reasoning tasks dominates the gym retro environments, and in these tasks, the denoised symbolic rules have demonstrated quite unbeatable competence across a host of them, as to be seen in section 5.

Original:  E XPERIMENTAL S ETTINGS A ND R ESULTS
Revised:  E XPERIMENTAL S ETTINGS A ND R ESULTS

Original: In this section, we select several retro visual RL environments to train CNN policy networks, then distill, verify and interpret the symbolic policy.
Revised: In this section, we select several retro visual RL environments to train CNN policy networks, then distill, verify and interpret the symbolic policy.

Original: Speciﬁcally, the selected environments include:
Revised: Speciﬁcally, the selected environments include:

Original: AdventureIsland2, AdventureIsland3, Pong, CircusCharlie, Airstriker-Genesis, Seaquest, and AstroRoboSasa.
Revised: AdventureIsland2, AdventureIsland3, Pong, CircusCharlie, Airstriker-Genesis, Seaquest, and AstroRoboSasa.

Original: More illustrative visualizations of our experiments can be found in the videos in the supplementary materials, and we sincerely recommend our readers could check them out.
Revised: More illustrative visualizations of our experiments can be found in the videos in the supplementary materials, and we sincerely recommend our readers could check them out.

Original: Performance validation .
Revised: Performance validation .

Original: We ﬁrst compare the performance of the distilled symbolic policy against the RL teacher models.
Revised: We ﬁrst compare the performance of the distilled symbolic policy against the RL teacher models.

Original: Fig.
Revised: Fig.

Original: 4 shows the performance of PPO (Schulman et al., 2017), A2C (Mnih et al., 2016), human player, and the proposed symbolic policy distillation.
Revised: 4 shows the performance of PPO (Schulman et al., 2017), A2C (Mnih et al., 2016), human player, and the proposed symbolic policy distillation.

Original: In the experiments, the symbolic policies are all distilled from the PPO teacher model.
Revised: In the experiments, the symbolic policies are all distilled from the PPO teacher model.

Original: The score of the human player is obtained by a gaming master controlling the agent under the lowered frame rate.
Revised: The score of the human player is obtained by a gaming master controlling the agent under the lowered frame rate.

Original: Due to engineering facilitation considerations, in this work we set the agents in the environments as to only have one life for all.
Revised: Due to engineering facilitation considerations, in this work we set the agents in the environments as to only have one life for all.

Original: As can be seen in the ﬁgure, the distilled policy maintained comparable performance with the teacher model, and for some environments, the distilled policy is even better thanks to the denoising mechanism of symbolic distillation.
Revised: As can be seen in the ﬁgure, the distilled policy maintained comparable performance with the teacher model, and for some environments, the distilled policy is even better thanks to the denoising mechanism of symbolic distillation

Original: Interpretability validation .
Revised: Interpretability validation .

Original: Besides verifying the performance, we also evaluate the interpretability of the regressed policy.
Revised: Besides verifying the performance, we also evaluate the interpretability of the regressed policy.

Original: This requires to check the symbolic operators combinations learned in the tree nodes, re-write them as readable form and analyze.
Revised: This requires to check the symbolic operators combinations learned in the tree nodes, re-write them as readable form and analyze.

Original: On the selected environments, highly interpretable and hierarchical forms of symbolic policy trees are all found in the distilled equations.
Revised: On the selected environments, highly interpretable and hierarchical forms of symbolic policy trees are all found in the distilled equations.

Original: One sample distilled policy for Pong is displayed in Eq. 2, and one sample for CircusCharlie is displayed in Fig.
Revised: One sample distilled policy for Pong is displayed in Eq. 2, and one sample for CircusCharlie is displayed in Fig.

Original: More interpretable distilled policies on other environments are put in appendix B.
Revised: More interpretable distilled policies on other environments are put in appendix C.

Revised: Behavior comparison with the neural network teacher .

Revised: In Fig.

Revised: 4, we plotted the action and the accumulated reward, both for the trained teacher model and the distilled symbolic policy.

Revised: As can be seen in the ﬁgure, a well-trained teacher model make more frequent actions, while still underperform

Original: Evaluation on
Revised: Evaluation on

Original: [Table]
Revised: [Table]

Original: Distilled 
Revised: Distilled Symbolic Policy

Original: [Table] Policy

Original: [Table]
Revised: [Table]

Original: PPO/A2C nor symbolicrules are ﬁne-tuned/modiﬁed
Revised: PPO/A2C nor symbolic are ﬁne- /modiﬁed

Original: Behavior comparison with the neural network teacher . In Fig. 4, we plotted the action and the accumulated reward, both for the trained teacher model and the distilled symbolic policy. As can be seen in the ﬁgure, a well-trained teacher model make more frequent actions, while still underperform the distilled symbolic rule.
Revised: the distilled symbolic rule.

Original: More differences are better shown in the video in the supplementary materials.
Revised: More differences are better shown in the video in the supplementary materials.

Original: The results there shows that in the Pong env, the trained CNN-based teacher model frequently “jitters” up and down even when the pong is very far away from the right racket.
Revised: The results there shows that in the Pong env, the trained CNN-based teacher model frequently “jitters” up and down even when the pong is very far away from the right racket.

Original: In those cases, the pong is usually going to bounce with the wall to change its heading direction, making those states “hardly controllable”.
Revised: In those cases, the pong is usually going to bounce with the wall to change its heading direction, making those states “hardly controllable”.

Original: Due to the near-random behavior of the teacher under these “difﬁcult” states, the symbolic distillation fails to converge to one single action under these cases, and hence mapped default action to them, accidentally generated more efﬁcient rules, and those rules are picked through the last screening step of RoundTourMix.
Revised: Due to the near-random behavior of the teacher under these “difﬁcult” states, the symbolic distillation fails to converge to one single action under these cases, and hence mapped default action to them, accidentally generated more efﬁcient rules, and those rules are picked through the last screening step of RoundTourMix.

Original: Transferablity validation .
Revised: Transferablity validation .

Original: Since the object detection step provide robust environment representations that group different scenarios together, even if the scenes’ pixel-level attributes are markedly different, the distilled symbolic rule should transfer well if the logical speciﬁcations are congruent.
Revised: Since the object detection step provide robust environment representations that group different scenarios together, even if the scenes’ pixel-level attributes are markedly different, the distilled symbolic rule should transfer well if the logical speciﬁcations are congruent.

Original: In order to test the transferability/generalizability of the distilled symbolic rule, we compare the the distilled rules against CNN policies using the AdventureIsland2 and AdventureIsland3 environments (Refer Fig. 5).
Revised: In order to test the transferability/generalizability of the distilled symbolic rule, we compare the the distilled rules against CNN policies using the AdventureIsland2 and AdventureIsland3 environments (Refer Fig. 5).

Original: We ﬁrst train a teacher
Revised: We ﬁrst train a teacher

Original: CNN policy networks on AdventureIsland3 and distill symbolic policy with PPO; we then directly test both the CNN models and the distilled symbolic rule on AdventureIsland2, without tuning the
Revised: CNN policy networks on AdventureIsland3 and distill symbolic policy with PPO; we then directly test both the CNN models and the distilled symbolic rule on AdventureIsland2, without tuning the

Original: CNN policy networks nor changing the symbolic rule.
Revised: CNN policy networks nor changing the symbolic rule.

Original: As can be seen in Table 2, when directly transfered to a new environment with similar underlying rule but slightly different image styles, the CNN models completely fail to match their performances in the original environment, over the checkpoint from the entire training history.
Revised: As can be seen in Table 2, when directly transfered to a new environment with similar underlying rule but slightly different image styles, the CNN models completely fail to match their performances in the original environment, over the checkpoint from the entire training history.

Original: Additionally, ﬁne-tuning the CNN policy network to adapt to the new environment will require another considerable amount of training.
Revised: Additionally, ﬁne-tuning the CNN policy network to adapt to the new environment will require another considerable amount of training.

Original: In contrary, the distilled symbolic rule painlessly transfer to the new environment, with signiﬁcantly fewer performance loss.
Revised: In contrary, the distilled symbolic rule painlessly transfer to the new environment, with signiﬁcantly fewer performance loss.

Original: This is due to that our object detection module dwells on optical ﬂow and object shape, which is robust to color/style changes.
Revised: This is due to that our object detection module dwells on optical ﬂow and object shape, which is robust to color/style changes.

Original: The comparison between the symbolic policy and the CNN based policy network again endorses the better transferability of the symbolic rule.
Revised: The comparison between the symbolic policy and the CNN based policy network again endorses the better transferability of the symbolic rule.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper study the distillation of CNN based reinforcement learning agent into a symbolic policy that dwells on geometric and numerical operands and operators.
Revised: This paper study the distillation of CNN based reinforcement learning agent into a symbolic policy that dwells on geometric and numerical operands and operators.

Original: Thanks to the the denoising and the screening mechanisms in the distillation procedure, the distilled symbolic policy achieves comparable or even better performance than the CNN teacher model.
Revised: Thanks to the the denoising and the screening mechanisms in the distillation procedure, the distilled symbolic policy achieves comparable or even better performance than the CNN teacher model.

Original: Our results point to the new opportunity to make the visual RL more interpretable, reliable and generalizable, by reclaiming a symbolic design with the proposed symbolic policy distillation.
Revised: Our results point to the new opportunity to make the visual RL more interpretable, reliable and generalizable, by reclaiming a symbolic design with the proposed symbolic policy distillation.

Original: Our future work aims for more integrated neural-symbolic solutions for visual RL.
Revised: Our future work aims for more integrated neural-symbolic solutions for visual RL.

Original:  R EPRODUCIBILITY C HECKLIST
Revised:  R EPRODUCIBILITY C HECKLIST

Original: To ensure reproducibility, we use the Machine Learning Reproducibility Checklist v2.0, Apr.2020 (Pineau et al., 2021).
Revised: To ensure reproducibility, we use the Machine Learning Reproducibility Checklist v2.0, Apr.2020 (Pineau et al., 2021).

Original: • For all models and algorithms presented, – A clear description of the mathematical settings, algorithm, and/or model.
Revised: • For all models and algorithms presented, – A clear description of the mathematical settings, algorithm, and/or model.

Original: We clearly describe all of the settings, formulations, and algorithms in Section 4.3.
Revised: We clearly describe all of the settings, formulations, and algorithms in Section 4.3.

Original: – A clear explanation of any assumptions.
Revised: – A clear explanation of any assumptions.

Original: We do not make assumptions.
Revised: We do not make assumptions.

Original: – An analysis of the complexity (time, space, sample size) of any algorithm.
Revised: – An analysis of the complexity (time, space, sample size) of any algorithm.

Original: We do not make the analysis.
Revised: We do not make the analysis.

Original: • For any theoretical claim , – A clear statement of the claim.
Revised: • For any theoretical claim , – A clear statement of the claim.

Original: We do not make theoretical claims.
Revised: We do not make theoretical claims.

Original: – A complete proof of the claim.
Revised: – A complete proof of the claim.

Original: We do not make theoretical claims.
Revised: We do not make theoretical claims.

Original: • For all datasets used, check if you include: – An explanation of any data that were excluded, and all pre-processing step.
Revised: • For all datasets used, check if you include: – An explanation of any data that were excluded, and all pre-processing step.

Original: We did not exclude any data or perform any pre-processing.
Revised: We did not exclude any data or perform any pre-processing.

Original: – A link to a downloadable version of the dataset or simulation environment.
Revised: – A link to a downloadable version of the dataset or simulation environment.

Original: All ROMs used can be downloaded from https://archive.
Revised: All ROMs used can be downloaded from https://archive.

Original: org/details/No-Intro-Collection_2016-01-03_Fixed and http://www.atarimania.com/rom_collection_archive_atari
Revised: org/details/No-Intro-Collection_2016-01-03_Fixed and http://www.atarimania.com/rom_collection_archive_atari

Original: _ 2600_roms.html .
Revised: _ 2600_roms.html .

Original: – For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control.
Revised: – For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control.

Original: We do not collect or release new datasets.
Revised: We do not collect or release new datasets.

Original: • For all reported experimental results , check if you include: – A clear deﬁnition of the speciﬁc measure or statistics used to report results.
Revised: • For all reported experimental results , check if you include: – A clear deﬁnition of the speciﬁc measure or statistics used to report results.

Original: We use the accumulated reward gained in a single episode as our statistical measure.
Revised: We use the accumulated reward gained in a single episode as our statistical measure.

Original: – A description of results with central tendency (e.g. mean) & variation (e.g. error bars).
Revised: – A description of results with central tendency (e.g. mean) & variation (e.g. error bars).

Original: We do not report mean and standard deviation for experiments.
Revised: We do not report mean and standard deviation for experiments.

Original: – The average runtime for each result, or estimated energy cost.
Revised: – The average runtime for each result, or estimated energy cost.

Original: We do not report the running time or energy cost.
Revised: We do not report the running time or energy cost.

Original: – A description of the computing infrastructure used.
Revised: – A description of the computing infrastructure used.

Original: We do not report the computing infrastructure used.
Revised: We do not report the computing infrastructure used.

Revised: A J USTIFICATION FOR K EY N OTES

Revised: In this section we provide justiﬁcations for the architectural and algorithmic designs of our proposed

Original: A D ETAILS OF OPTICAL FLOW BASED OBJECT DETECTION
Revised: RoundTourMix algorithm A. D ETAILS ON THE S EARCH S PACE

Revised: Geometric and Numerical Operator Search Space.

Revised: Precisely, we have a total of 11 different functions in the geometric operator search space and 19 different functions in the numerical operator search space.

Revised: As mentioned in section

Revised: Temporal Tracking and Memory-Attributes.

Revised: We include mechanisms that make our symbolic algorithm aware of the temporal behavior of its entities using memory-attributes.

Revised: For instance, in shooting-based environments, relying just on the current frame gives rise to the agent shooting multiple times at the same enemy if it is not aware of the enemies that it has already launched its bullet at.

Revised: Another memory-attribute in such games arises when the agent needs to remember the target (out of multiple targets) it has locked on to, get in a suitable position and then ﬁre.

Revised: A. M OTIVATIONS BEHIND THE A LGORITHMIC C HOICES

Revised: In this section we break down the Step 2 and Step 3 of RoundTourMix while providing validation for the speciﬁc choices in the same.

Revised: Justiﬁcation for Random Selection.

Revised: Exhaustive search will ensure that excellent performance will be achieved with virtually unlimited hardware and software resources.

Revised: However, as previously mentioned in 4.2, one potential challenge of the symbolic distillation technique is that the geometric representation’s search space is exponentially huge.

Revised: As a result, as compared to exhaustive search, random selection is the only viable option.

Revised: Furthermore, our investigations show that the random search paradigm produces more reliable outcomes over exhaustive searching.

Revised: If we compose symbolic compositions of length 20, the total possible policies are (11 + 19) 20 =

Revised: 30 20 = 3 .

Revised: 48 e 29 which is a gigantic number.

Revised: Furthermore, because there is no ”learning” involved, any exhaustive or brute-force search goes against the spirit of reinforcement learning.

Revised: Traditional reinforcement learning does not use such comprehensive methods because of the vast number of viable policies, as earlier noted.

Revised: Guess and Observe.

Revised: As earlier discussed the geometric search space is very large.

Revised: Hence we randomly guess the 16 different geometric relations to be guessed.

Revised: We then iterate over the image from the teacher behaviour, evaluate the randomly sampled geometric and numerical relations.

Revised: For each frame we create a vector of size 16 of possible logics which are then later on used as the numerical state for symbolic distillation - The set numerical state vectors as extracted from each image constitutes the ofﬂine teacher dataset

Revised: Explanation of Branching Conditions.

Revised: • Search the symbolic policy tree composed of numerical operators • Performed to get suitable candidate policies • We perform the search a total of cnt MAX times • At the start of each search we ﬁrst sample an action node from the set of all unsolved action nodes

Revised: We then slice out the subset of the ofﬂine teacher data that aligns with the guessed branching condition.

Revised: • If the sampled branching condition was a step in the right direction, the entropy of the subset sliced remains at a low value •

Revised: If it does not, it is an indicator that the underlying rule was more complex and the path is to be explored deeper •

Revised: If the tree is already at its maximal depth, we either (A) directly run and ﬁt symbolic regression between the sliced numerical state vs action ofﬂine dataset (B) assign a default action to that action node or (C) prune off the entire sub-tree containing the action node in question • Else if the tree is not yet at its maximal depth and the entropy is high, we either (A) explore deeper by creating a left/right branching condition or (B) assign a default action at the action node in order to denoise it

Revised: B D ETAILS OF OPTICAL FLOW BASED OBJECT DETECTION

Original: Since in most retro environments, the image styles are simple, where the backgrounds are nearly ﬁxed and the objects share similar patterns.
Revised: Since in most retro environments, the image styles are simple, where the backgrounds are nearly ﬁxed and the objects share similar patterns.

Original: Therefore, we take the optical ﬂow as a simpliﬁed object detection module.
Revised: Therefore, we take the optical ﬂow as a simpliﬁed object detection module.

Original: We ﬁrst segment out the object list from template matching, then estimate the velocity using the pretrained FlowNet (Ilg et al., 2017), and get the speed via the average optical ﬂow of detected objects.
Revised: We ﬁrst segment out the object list from template matching, then estimate the velocity using the pretrained FlowNet (Ilg et al., 2017), and get the speed via the average optical ﬂow of detected objects.

Original: The classes of the objects are obtained based on the shape in the predeﬁned templates.
Revised: The classes of the objects are obtained based on the shape in the predeﬁned templates.

Original: B E XAMPLES OF DISTILLED SYMBOLIC POLICY
Revised: C E XAMPLES OF DISTILLED SYMBOLIC POLICY

Original: We check the learned symbolic operators in the tree nodes, and re-write them as interpretable symbolic tree policy.
Revised: We check the learned symbolic operators in the tree nodes, and re-write them as interpretable symbolic tree policy.

Original: The results are plotted in Fig.
Revised: The results are plotted in Fig.

Original: 9, Fig.
Revised: 9, Fig.

Original: 7, Fig 10, Fig 6, Fig 8, and Fig 11.[Figure]
Revised: 7, Fig 10, Fig 6, Fig 8, and Fig 11.

Original: C E XAMPLES OF R ETRO -E NV L IST T HAT O NE -F RAME S YMBOLIC D ISTILLATION C AN W ORK W ELL
Revised: D E XAMPLES OF R ETRO -E NV L IST T HAT O NE -F RAME S YMBOLIC D ISTILLATION C AN W ORK W ELL

Original: After checking the underlying logics and evaluated the possible strategies of gym-retro environments, a plenty of them are found to be solvable by our symbolic policy distillation.
Revised: After checking the underlying logics and evaluated the possible strategies of gym-retro environments, a plenty of them are found to be solvable by our symbolic policy distillation.

Original: Several examples include: 8Eyes-Nes, 1942-Nes, BadDudes-Nes, BioSenshiDanIncreaserTonoTatakai-Nes, BreakThruNes, BubbleBobble-Nes, CodeNameViper-Nes, FrontLine-Nes, GhostsnGoblins-Nes, KidIcarus-Nes,
Revised: Several examples include: 8Eyes-Nes, 1942-Nes, BadDudes-Nes, BioSenshiDanIncreaserTonoTatakai-Nes, BreakThruNes, BubbleBobble-Nes, CodeNameViper-Nes, FrontLine-Nes, GhostsnGoblins-Nes, KidIcarus-Nes,

Original: KidNikiRadicalNinja-Nes, Parodius-Nes, RoboccoWars-Nes, Sansuu5And6NenKeisanGame-Nes,
Revised: KidNikiRadicalNinja-Nes, Parodius-Nes, RoboccoWars-Nes, Sansuu5And6NenKeisanGame-Nes,

Original: Seicross-Nes, SonSon-Nes, Tennis-Atari2600, TwinEagle-Nes, 
Revised: Seicross-Nes, SonSon-Nes, Tennis-Atari2600, TwinEagle-Nes, ViceProjectDoom-Nes, Xexyz-Nes,

Original: [Figure]
Revised: MysteryQuest-Nes. The screenshots for them are displayed in Fig.

Original: D O THER ILLUSTRATIONS
Revised: E O THER ILLUSTRATIONS

Original: The pseudo-code for the algorithm in table 1 are presented in Fig.
Revised: The pseudo-code for the algorithm in table 1 are presented in Fig.

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]