Original: AD
Revised: AD

Original: -VAT
Revised: -VAT

Original: : A N ASYMMETRIC DUELING MECHANISM FOR LEARNING VISUAL ACTIVE TRACKING
Revised: : A N ASYMMETRIC DUELING MECHANISM FOR LEARNING VISUAL ACTIVE TRACKING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Visual active tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.
Revised: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.

Revised: Previous work has shown a Single Agent Reinforcement Learning (SARL) approach, where the tracker can be trained in simulator and performs tracking in real-world scenarios.

Revised: However, during training such an SARL method requires manually specifying the moving path of the target object to be tracked, which can hurt the tracker’s generalization on unseen object moving patterns.

Revised: To learn a robust tracker for

Original: In this paper, we propose a novel  method which adopts an asymmetric dueling mechanism for learning visual active tracking, namely AD-VAT.
Revised: VAT, in this paper we propose a novel Multi-Agent RL (MARL) training method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT.

Original: In AD-VAT, the target and the tracker are mutual opponents, i.e. , the tracker manages to lockup the target, and the target tries to escape from the tracker.In the implementation, both the tracker and the target are approximated by deep networks, and their policies that map environment observations to control actions can be learned via reinforcement learning in an end-to-end manner.
Revised: In ADVAT, both the tracker and the target are approximated by deep networks, and are trained via end to end RL in a dueling/competitive manner: i.e. , the tracker intends to lockup the target, while the target tries to escape from the tracker.

Original: The tracker and the target are asymmetric in observations, network structures and reward functions.
Revised: They are asymmetric in that the target is aware of the tracker, but not vice versa.

Original: Different from the tracker, the target is modeled with a tracker-aware network, i.e. , besides its own observation, the tracker’s observations and actions are also fed as input to the network. In addition, it learns to predict the tracker’s reward as an auxiliary task.
Revised: Speciﬁcally, besides its own observation, the target is fed with the tracker’s observation and action, and learns to predict the tracker’s reward as an auxiliary task.

Original: We argue that such an asymmetric adversarial mechanism is able to learn a stronger target, which vice versa induces a more robust tracker.
Revised: We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker.

Revised: To stabilize the training, we also propose a novel partial zero sum reward for the tracker/target.

Original: The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in trainingthe tracker and  more robust tracking behaviors in different testing scenarios.
Revised: The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios.

Original: For supplementary videos, see: https://www.youtube.
Revised: For supplementary videos, see: https://www.youtube.

Original: com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS
Revised: com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.
Revised: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.

Original: VAT is demanded in many real-world applications such as autonomous vehicle ﬂeet ( e.g. , a slave-vehicle should follow a master-vehicle ahead), service robots and drones ( e.g. , a drone is required to follow a person when recording a video).
Revised: VAT is demanded in many real-world applications such as autonomous vehicle ﬂeet ( e.g. , a slave-vehicle should follow a master-vehicle ahead), service robots and drones ( e.g. , a drone is required to follow a person when recording a video).

Original: To accomplish the VAT task, one typically needs to perform a sequence of tasks such as recognition, localization, motion prediction, and camera control.
Revised: To accomplish the VAT task, one typically needs to perform a sequence of tasks such as recognition, localization, motion prediction, and camera control.

Original: However, conventional visual tracking (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009; Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012) aims to solely propose a 2D bounding box of the target frame by frame, and does not actively take into consideration the camera-control.
Revised: However, conventional visual tracking (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009; Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012) aims to solely propose a 2D bounding box of the target frame by frame, and does not actively take into consideration the camera-control.

Original: Thus, compared to the problem of “passive" tracking,
Revised: Thus, compared to the problem of “passive" tracking,

Original: VAT is more practical and challenging.
Revised: VAT is more practical and challenging.

Original: With the advancement of deep reinforcement learning (Sutton & Barto, 1998;
Revised: With the advancement of deep reinforcement learning (Sutton & Barto, 1998;

Original: Mnih et al., 2015;
Revised: Mnih et al., 2015;

Original: Silver et al., 2016b; Mnih et al., 2016), training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible (Luo et al., 2018).
Revised: Silver et al., 2016b; Mnih et al., 2016), training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible (Luo et al., 2018).

Original: The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network.
Revised: The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network.

Original: Not only can it save the effort in tuning an extra camera controller, but also does it outperform the conventional methods where the passive tracker is equipped with a hand-engineered camera-control module.
Revised: Not only can it save the effort in tuning an extra camera controller, but also does it outperform the conventional methods where the passive tracker is equipped with a hand-engineered camera-control module.

Original: However, the performance of the deep reinforcement learning based tracker is still limited by the training methods.
Revised: However, the performance of the deep reinforcement learning based tracker is still limited by the training methods.

Original: Due to the "trial-and-error" nature of reinforcement learning, it is hard to directly train the tracker in real world.
Revised: Due to the “trial-and-error" nature of reinforcement learning, it is infeasible to directly train the tracker in real world.

Original: Alternatively, virtual environments are always utilized to generate sufﬁcient data for training without tedious human labeling.
Revised: Alternatively, virtual environments are always utilized to generate sufﬁcient data for training without tedious human labeling.

Original: Nevertheless, to deploy the trained tracker in real-world, one has to overcome the gap between virtual and real environments.
Revised: Nevertheless, to deploy the trained tracker in real-world, one has to overcome the gap between virtual and real environments.

Original: One solution can be building numbers of high-ﬁdelity environments (Zhu et al., 2017).
Revised: One solution can be building numbers of high-ﬁdelity environments (Zhu et al., 2017).

Original: However, it is  expensive and tedious to build such environments for VAT.
Revised: However, it is [Figure] expensive and tedious to build such environments for VAT.

Original: Both the visual rendering (illumination, texture,[Figure] etc. ) and the physical properties should be carefully designed to emulate the real world.
Revised: Both the visual rendering (illumination, texture, etc. ) and the physical properties should be carefully designed to emulate the real world.

Original: Suppose the VAT where the target is a pedestrian.
Revised: Suppose we carry out VAT where the target is a pedestrian.

Original: To build the environment, one has to not only model the human’s appearance, but also design physical rules and the pedestrian’s trajectory so that it moves naturally like a human beings.
Revised: To build the environment, one has to not only model the human’s appearance, but also design physical rules and the pedestrian’s trajectory so that it moves naturally like a human beings.

Original: Recently, Luo et al. (2018) tried to overcome the virtual-real gap by applying the so -called environment augmentation technique.
Revised: Recently, (Luo et al., 2018) tried to overcome the virtual-real gap by applying the so-called environment augmentation technique.

Original: They diversify the visual appearance by changing the placement of the background objects and by ﬂipping left-right the screen frame.
Revised: They diversify the visual appearance by changing the placement of the background objects and by ﬂipping left-right the screen frame.

Original: However, they neglect another important factor, that is, the trajectory of the targetobject for VAT task.
Revised: However, they neglect another important factor, that is, the motion of the target for VAT task.

Original: Intuitively, the complexity and diversity of the target motion in training will impact the generalization of the tracker.
Revised: Intuitively, the complexity and diversity of the target motion in training will impact the generalization of the tracker.

Original: For example, if the target only moves forward during training, the tracker may over ﬁt to move straightly and fail to track other motion patterns, like a sharp turn.
Revised: For example, if the target only moves forward during training, the tracker may over ﬁt to move straightly and fail to track other motion patterns, like a sharp turn.

Original: In this work, we propose an Asymmetric Dueling mechanism for learning Visual Active Tracking (AD-VAT) (see Fig.1) where the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during training. The two agents are different in observation, network architecture and goal, yet are competitive, hence the name “asymmetric dueling".
Revised: In this work, we propose a novel adversarial Multi-Agent RL for learning VAT, refereed to as AD-VAT ( Asymmetric Dueling mechanism for learning Visual Active Tracking ). In the mechanism, the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during competition (See Fig.1).

Original: The tracker manages to follow the target.
Revised: As the training proceeds, the environments of AD-VAT naturally compose a curriculum, because the tracker is more likely to compete with a target with the appropriate difﬁculty level when both agents are becoming stronger simultaneously.

Original: Meanwhile, thetarget tries to escape from the tracker. We shape a reward structure to guide the learning of the two agents.
Revised: When exploring the escape policy, the target consequently generates various trajectories to train the tracker.

Original: Different from the zero-sum reward structure (Bansal et al., 2017; Pinto et al., 2017), we adopt a novel reward referred to as “partial zero-sum”, which only encourages the tracker-target competition when they are reasonably close and penalizes when they are too far-away.
Revised: By the dueling/adversarial mechanism, the target is encouraged to discover the weakness of the tracker more often, which could serve as a kind of “weakness-ﬁnding" that makes the tracker be more robust.

Original: We ﬁnd empirically that the partial zero-sum stabilizes the training.
Revised: However, in practice, using heuristic MARL method for training VAT is unstable and slow to converge. To address these issues, we derive two components in AD-VAT: the “partial zero-sum reward” and the “tracker-aware network" for target.

Original: If the zero-sum were adopted, the training would become highly unstable in the sense that  the target quickly learns to be far away from the tracker while the tracker has no chance to see the target any more and henceforth gets plateaus during training.
Revised: The “partial zero-sum” is designed to avoid an observed phenomenon, that when the target quickly learns to be far away from the tracker while the tracker has no chance to see the target any more and henceforth gets plateaus during training.

Original: When exploring the escape policy, the target consequently generates various trajectories of its actions. Such escape trajectories could be regarded as “hard examples" for the tracker, which should be beneﬁcial for the learning of the tracker. As the training proceeds, the environments of AD-VAT naturally form a curriculum, because the tracker is more likely to compete with a target with the appropriate difﬁculty level when both agents are becoming stronger simultaneously. Thus, it improves the sample-efﬁciency of learning the tracker.
Revised: We adopt a novel reward referred to as “partial zero-sum”, which only encourages the tracker-target competition when they are reasonably close and penalizes when they are too far-away.

Original: To learn the optimal policy to escape, we model the target with a tracker-aware network, i.e. , besides its own observation, the observation and actions of the tracker’s are also fed to the escaping network.
Revised: To learn the optimal policy to escape, we model the target with a “tracker-aware network", i.e. , besides its own observation, the observation and actions of the tracker are also fed to the escaping network.

Original: In addition,it learns topredict the reward of the tracker as an auxiliary task, which could help the target better shape a  representation about the tracker to learn the optimal policy to escape.
Revised: In addition, to shape a better representation about the tracker, we add an auxiliary task for the target, learning to predict the immediate reward of the tracker.

Original: We argue that such an “asymmetric dueling" mechanism is able to learn a stronger target, which vice versa yields a more robust tracker ultimately.
Revised: We argue that such an “asymmetric dueling" mechanism is able to learn a stronger target, which vice versa yields a more robust tracker ultimately.

Original: The experiment is conducted in various 2D and 3D environments for further studying AD-VAT.
Revised: The experiment is conducted in various 2D and 3D environments for further studying AD-VAT.

Original: The 2D environment is a matrix map where obstacles are randomly placed.
Revised: The 2D environment is a matrix map where obstacles are randomly placed.

Original: In the 2D environments, we evaluate and quantify the effectiveness of our approach in an ideal condition, lacking of noise in observation and action.
Revised: In the 2D environments, we evaluate and quantify the effectiveness of our approach in an ideal condition, lacking of noise in observation and action.

Original: We also conduct an ablation study to show the effectiveness of the two important components: “partial zero-sum reward" and “tracker-aware network".
Revised: We also conduct an ablation study to show the effectiveness of the two important components: “partial zero-sum reward" and “tracker-aware network".

Original: The 3D environments are built on Unreal Engine 4, a popular game engine for building high-ﬁdelity environments.
Revised: The 3D environments are built on Unreal Engine 4, a popular game engine for building high-ﬁdelity environments.

Original: We choose a large room for training, where the texture of the background/players and the illumination are randomized.
Revised: We choose a large room for training, where the texture of the background/players and the illumination are randomized.

Original: Three realistic scenarios built by artists are used for further evaluating the generalization.
Revised: Three realistic scenarios built by artists are used for further evaluating the robustness.

Original: In the 3D environments, we further demonstrate that the tracker trained in AD-VAT is capable of generalizing to high-ﬁdelity environments even it is trained in a simple environment.
Revised: In the 3D environments, we further demonstrate that the tracker trained in AD-VAT is capable of generalizing to high-ﬁdelity environments even it is trained in a simple environment.

Original: The contributions of our work can be summarized as follows: (1)We propose an asymmetric dueling mechanism for learning visual active tracking (AD-VAT).
Revised: The contributions of our work can be summarized as follows: • Unlike the Single Agent Reinforcement Learning (SARL) approach (Luo et al., 2018), we propose a novel Multi Agent Reinforcement Learning (MARL) method for VAT task, i.e. , the Asymmetric Dueling mechanism (AD-VAT).

Original: Within the proposed AD-VAT,  target generates diverse trajectories when competing with the tracker, ultimately driving the learning of the tracker; (2) We shape a  partial zero-sum reward structure for AD-VAT. The reward structure makes the training efﬁciently; (3) We develop a tracker-aware network for building an effective target agent.
Revised: In AD-VAT, the target learns to generate diverse trajectories when competing with the tracker, which in turn helps train a more robust tracker. • We provide two techniques to guarantee an efﬁcient yet effective AD-VAT. A partial zero-sum reward structure, which signiﬁcantly stabilizes the training.

Original: This yields better escaping policy and consequently better tracking policy.
Revised: A tracker-aware network for the target, which yields better escaping policy and consequently better tracking policy.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Active Object Tracking.
Revised: Active Object Tracking.

Original: As described above that, active object tracking deals with object tracking and camera control at the same time.
Revised: As described above that, active object tracking deals with object tracking and camera control at the same time.

Original: This problem attracts less attention compared with traditional object tracking (or visual object tracking)
Revised: This problem attracts less attention compared with traditional object tracking (or visual object tracking)

Original: (Wu et al., 2013).
Revised: (Wu et al., 2013).

Original: In general, this problem could be addressed in a two-step manner or in a end-to-end manner.
Revised: In general, this problem could be addressed in a two-step manner or in a end-to-end manner.

Original: In the two-step solution, traditional object tracking and camera control are conducted sequentially to obtain tracking results and manipulate camera.
Revised: In the two-step solution, traditional object tracking and camera control are conducted sequentially to obtain tracking results and manipulate camera.

Original: Great progress has been achieved in traditional object tracking in recent decades (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009;
Revised: Great progress has been achieved in traditional object tracking in recent decades (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009;

Original: Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012).
Revised: Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012).

Original: Thus one can utilize mature visual tracking algorithms (Henriques et al., 2015; Ma et al., 2015; Choi et al., 2017) to accomplish the passive tracking task.
Revised: Thus one can utilize mature visual tracking algorithms (Henriques et al., 2015; Ma et al., 2015; Choi et al., 2017) to accomplish the passive tracking task.

Original: According to the tracking results, camera control module could be developed to actively follow a target.
Revised: According to the tracking results, camera control module could be developed to actively follow a target.

Original: For example, in (Denzler &
Revised: For example, in (Denzler &

Original: Paulus, 1994) a two-stage method is proposed to handle robot control by motion detection and motion tracking.
Revised: Paulus, 1994) a two-stage method is proposed to handle robot control by motion detection and motion tracking.

Original: Kim et al.
Revised: Kim et al.

Original: (Kim et al., 2005) detect moving objects and track them using an active camera with pan/tilt/zoom.
Revised: (Kim et al., 2005) detect moving objects and track them using an active camera with pan/tilt/zoom.

Original: In (Hong et al., 2018), two modules, a perception module and a control policy module, are learned separately to train an agent accomplishing both  obstacle avoidance task and a target following task.
Revised: In (Hong et al., 2018), two modules, a perception module and a control policy module, are learned separately to train an agent accomplishing both an obstacle avoidance task and a target following task.

Original: Admittedly, tracking algorithms has been successful while still not perfect.
Revised: Admittedly, tracking algorithms have been successful while still not perfect.

Original: Camera control based on tracking results is challenging due to factors such as that the correspondence between image space and camera parameter space is unknown.
Revised: Camera control based on tracking results is challenging due to factors such as that the correspondence between image space and camera parameter space is unknown.

Original: Additionally, joint tuning of visual tracking and camera control is expensive and encounters trial-and-errors in real world.
Revised: Additionally, joint tuning of visual tracking and camera control is expensive and encounters trial-and-errors in real world.

Original: In end-to-end methods, direct mapping between raw input frame and camera action is established.
Revised: In end-to-end methods, direct mapping between raw input frame and camera action is established.

Original: Thus the intermediate visual tracking results are not necessarily required.
Revised: Thus the intermediate visual tracking results are not necessarily required.

Original: For instance, by reinforcement learning, camera is controlled by signal outputted from a Conv-LSTM network given raw input frame (Luo et al., 2018).
Revised: For instance, by reinforcement learning, camera is controlled by signal outputted from a Conv-LSTM network given raw input frame (Luo et al., 2018).

Original: This end-to-end solution veriﬁes the effectiveness, while not efﬁcient enough to solve this problem.
Revised: This end-to-end solution veriﬁes the effectiveness, while not efﬁcient enough to solve this problem.

Original: To improve the generalization potential, environment augmentation is conducted in this work.
Revised: To improve the generalization potential, environment augmentation is conducted in this work.

Original: However, the target itself, like the path, motion pattern is ﬁxed in their augmented environments, which is believed to limit the performance.
Revised: However, the target itself, like the path, motion pattern is ﬁxed in their augmented environments, which is believed to limit the performance.

Original: This inspires us to resort to the idea of dueling in this paper, i.e. , the target learns to get rid of the tracker by itself, guiding the learning of the tracker.
Revised: This inspires us to resort to the idea of dueling in this paper, i.e. , the target learns to get rid of the tracker by itself, guiding the learning of the tracker.

Original: Self-play method for reinforcement learning.
Revised: Self-play method for reinforcement learning.

Original: In terms of game theory, our setting is a 2-player non-cooperative extensive game with partial observation (Osborne & Rubinstein, 1994).
Revised: In terms of game theory, our setting is a two-player non-cooperative extensive game with partial observation (Osborne & Rubinstein, 1994).

Original: When the reward is zero-sum, some self-play algorithm with a speciﬁc value function ( e.g. , Counterfactual
Revised: When the reward is zero-sum, some self-play algorithm with a speciﬁc value function ( e.g. , Counterfactual

Original: Regret, abbreviated as CFR (Zinkevich et al., 2008)) is proven to approximate the Nash Equilibrium, which means no player can obtain more rewards by unilaterally altering its policy, and henceforth produces very strong policy.
Revised: Regret, abbreviated as CFR (Zinkevich et al., 2008)) is proven to approximate the Nash Equilibrium, which means no player can obtain more rewards by unilaterally altering its policy, and henceforth produces very strong policy.

Original: CFR and its variants ( e.g. , CFR+ (Tammelin, 2014), Monte Carlo
Revised: CFR and its variants ( e.g. , CFR+ (Tammelin, 2014), Monte Carlo

Original: CFR (Burch et al., 2012)) are shown to be successful in several applications.
Revised: CFR (Burch et al., 2012)) are shown to be successful in several applications.

Original: For instance, the poker
Revised: For instance, the poker

Original: AI Libratus defeats professional human players in 2-player unlimited Texas Hold’em (Brown &
Revised: AI Libratus defeats professional human players in two-player unlimited Texas Hold’em (Brown &

Original: Sandholm, 2017), where CFR is utilized in combination of other careful devised techniques.
Revised: Sandholm, 2017), where CFR is utilized in combination of other careful devised techniques.

Original: In speciﬁc, the observation sequence is formalized as a game tree, where similar observations/actions are merged (called abstraction) and sub-games are solved efﬁciently.
Revised: In speciﬁc, the observation sequence is formalized as a game tree, where similar observations/actions are merged (called abstraction) and sub-games are solved efﬁciently.

Original: However, the abstraction and sub-game solving technique cannot be applied to our task in a straightforward way.
Revised: However, the abstraction and sub-game solving technique cannot be applied to our task in a straightforward way.

Original: In our VAT task, the observation is screen pixels, for which we need a more efﬁcient visual feature representation, preferably learned automatically with, e.g. , a deep ConvNet.
Revised: In our VAT task, the observation is screen pixels, for which we need a more efﬁcient visual feature representation, preferably learned automatically with, e.g. , a deep ConvNet.

Original: There are attempts to combine CFR with deep nets (Jin et al., 2017; Lanctot et al., 2017; Moravˇcík et al., 2017), however, they lose the theoretical guarantee in regards of ﬁnding the Nash Equilibrium, despite they show encouraging empirical results.
Revised: There are attempts to combine CFR with deep nets (Jin et al., 2017; Lanctot et al., 2017; Moravˇcík et al., 2017). However, they lose the theoretical guarantee in regard of ﬁnding the Nash Equilibrium, despite they show encouraging empirical results.

Original: On the other hand, there is a body of work that combines self-play with deep net to train a policy in either partial or perfect observation, e.g. , playing board game (Tesauro, 1995; Silver et al., 2016a; 2017b;a), playing video game (ope; Jaderberg et al., 2018), robot controlling (Bansal et al., 2017;
Revised: On the other hand, there is a body of work that combines self-play with deep net to train a policy in either partial or perfect observation, e.g. , playing board game (Tesauro, 1995; Silver et al., 2016a; 2017b;a), playing video game (ope; Jaderberg et al., 2018), robot controlling (Bansal et al., 2017;

Original: Pinto et al., 2017), navigation in maze (Sukhbaatar et al., 2017), collision avoidance in autonomous driving (Behzadan & Munir, 2018), etc.
Revised: Pinto et al., 2017), navigation in maze (Sukhbaatar et al., 2017), collision avoidance in autonomous driving (Behzadan & Munir, 2018).

Original: The methods just utilize the conventional value functions in the RL literature, and also shown good empirical results.
Revised: The methods just utilize the conventional value functions in the RL literature, and also show good empirical results.

Original: Despite lacking theoretical guarantee of Nash Equilibrium ﬁnding, several techniques, in particular the domain randomization and the opponent pool sampling, are suggested to ensure the policy diversity and the training stabilization.
Revised: Despite lacking theoretical guarantee of Nash Equilibrium ﬁnding, several techniques, in particular the domain randomization and the opponent pool sampling, are suggested to ensure the policy diversity and the training stabilization.

Original: Noting this, we stick to the conventional value function when performing the self-play training in this work.
Revised: Noting this, we stick to the conventional value function when performing the self-play training in this work.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce our proposed method: Asymmetric Dueling mechanism for learning
Revised: In this section, we introduce our proposed method: Asymmetric Dueling mechanism for learning

Original: Visual Active Tracking (AD-VAT) .
Revised: Visual Active Tracking (AD-VAT) .

Original: At ﬁrst the proposed is formulated as  multi-agent system.
Revised: At ﬁrst the proposed is formulated as a multi-agent system.

Original: Then we illustrate the two key components: the reward structure and a tracker-aware model for the target.
Revised: Then we illustrate the two key components: the reward structure and a tracker-aware model for the target.

Original: 3. F ORMULATION
Revised: 3. F ORMULATION

Original: For the notations of our two-agent settings, let superscript α denote the tracker and superscript β denote the target.
Revised: For the notations of our two-agent settings, let superscript α denote the tracker and superscript β denote the target.

Original: The policy of the tracker, π α ( a α t | o α t ) , is a distribution over tracker action a α t , and conditioned on tracker observation o αt , where the subscript t denotes time step.
Revised: The policy of the tracker, π α ( a αt | o αt ) , is a distribution over tracker action a αt , and conditioned on tracker observation o αt , where the subscript t denotes time step.

Original: When the policy takes as function approximator a Neural Network with parameter θ α , we further write it as
Revised: When the policy takes as function approximator a Neural Network with parameter θ α , we further write it as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Likewise, the policy of the target can be written as
Revised: Likewise, the policy of the target can be written as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The tracker intends to maximize its expected return
Revised: The tracker intends to maximize its expected return

Original: [Equation]
Revised: [Equation]

Original: by learning the parameter θ α , where T denotes the horizon length of an episode and r αt is the immediate reward of the tracker at time step t .
Revised: by learning the parameter θ α , where T denotes the horizon length of an episode and r αt is the immediate reward of the tracker at time step t .

Original: In contrast, the target tries to maximize
Revised: In contrast, the target tries to maximize

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: by learning θ β .
Revised: by learning θ β .

Original: 3. R EWARD S TRUCTURE
Revised: 3. R EWARD S TRUCTURE

Original: The conventional adversarial methods (Bansal et al., 2017; Pinto et al., 2017) usually formulate the policy learning as a zero-sum game.
Revised: The conventional adversarial methods (Bansal et al., 2017; Pinto et al., 2017) usually formulate the policy learning as a zero-sum game.

Original: In the zero-sum game, the sum of the reward of each agent is always 0 , r αt + r βt = 0 .
Revised: In the zero-sum game, the sum of the reward of each agent is always 0 , r αt + r βt = 0 .

Original: However, such kind of formulation is not suitable for AD-VAT.
Revised: However, such kind of formulation is not suitable for AD-VAT.

Original: Considering a case that, when the two opponents are too far to observe each other, their taken actions can hardly inﬂuence the observation of their opponents directly.
Revised: Considering a case that, when the two opponents are too far to observe each other, their taken actions can hardly inﬂuence the observation of their opponents directly.

Original: In this case, the latent MDP transition is usually meaningless and ineffective for improving the skill level of the agent.
Revised: In this case, the latent MDP transition is usually meaningless and ineffective for improving the skill level of the agent.

Original: So constraining the competition in the observable range would make the learning more efﬁcient.
Revised: So constraining the competition in the observable range would make the learning more efﬁcient.

Original: Motivated by this, we shape a partial zero-sum reward structure, which utilize the zero-sum reward only when the target is observed by tracker, but gives penalties to each agent when they are far.
Revised: Motivated by this, we shape a partial zero-sum reward structure, which utilize the zero-sum reward only when the target is observed by tracker, but gives penalties to each agent when they are far.

Original: In the following, we will introduce the details of the partial zero-sum reward structure for visual active tracking.
Revised: In the following, we will introduce the details of the partial zero-sum reward structure for visual active tracking.

Original: Reward for tracker.
Revised: Reward for tracker.

Original: The reward for tracker is similar to that in (Luo et al., 2018), composing of a positive constant and an error penalty term.
Revised: The reward for tracker is similar to that in (Luo et al., 2018), composing of a positive constant and an error penalty term.

Original: However, we do not take the orientation difference between the target and the tracker into consideration.
Revised: However, we do not take the orientation difference between the target and the tracker into consideration.

Original: Moreover, we measure the relative position error based on a polar coordinate system in the 3D environment, considering the model of the camera observation.
Revised: Moreover, we measure the relative position error based on a polar coordinate system in the 3D environment, considering the model of the camera observation.

Original: With a slight abuse of notation, we can now write the reward function as
Revised: With a slight abuse of notation, we can now write the reward function as

Original: [Equation]
Revised: [Equation]

Original: here A > 0 , ζ > 0 are tuning parameters, specially A = 1 . 0 , ζ = 2 . 0 .
Revised: here A > 0 , ζ > 0 are tuning parameters, specially A = 1 . 0 , ζ = 2 . 0 .

Original: P β and P exp represents the target object’s position and the expected position.
Revised: P β and P exp represent the target object’s position and the expected position.

Original: Note that P is the relative coordinates in the tracker-centric coordinate system.
Revised: Note that P is the relative coordinates in the tracker-centric coordinate system.

Original: Each term of the coordinate is normalized regarding the range of the observation R .
Revised: Each term of the coordinate is normalized regarding the range of the observation R .

Original: Speciﬁcally, P is plane position ( x, y ) in the 2D environment, but is under a polar coordinates ( ρ, θ ) in 3D environment.
Revised: Speciﬁcally, P is plane position ( x, y ) in the 2D environment, but is under a polar coordinates ( ρ, θ ) in 3D environment.

Original: ρ is the distance between tracker and target, θ is the relative angle to the front of the camera.
Revised: ρ is the distance between tracker and target, θ is the relative angle to the front of the camera.

Original: Besides, the reward is clipped to be in the range of [ − A, A ] to avoid over punishment when the object is far away from the expected position.
Revised: Besides, the reward is clipped to be in the range of [ − A, A ] to avoid over punishment when the object is far away from the expected position.

Original: Reward for target.
Revised: Reward for target.

Original: The reward for the target object is closely related to reward of the tracker, written as below:
Revised: The reward for the target object is closely related to reward of the tracker, written as below:

Original: [Equation]
Revised: [Equation]

Original: where r α is the reward of the tracker as deﬁned in Eq.
Revised: where r α is the reward of the tracker as deﬁned in Eq.

Original: 5, ξ > 0 is a tuning parameter controlling the factor of the penalty term, and P far =
Revised: 5, ξ > 0 is a tuning parameter controlling the factor of the penalty term, and P far =

Original: (1 , 1) , representing the farthest point to tracker within the observation.
Revised: (1 , 1) , representing the farthest point to tracker within the observation.

Original: The reward is also clipped  r t is smaller than − A .
Revised: The reward is also clipped when r t is smaller than − A .

Original: The insight behind this formulation is that, we do not encourage the target to escape from the tracker as far as possible.
Revised: The insight behind this formulation is that, we do not encourage the target to escape from the tracker as far as possible.

Original: Alternatively, by applying this reward function, the optimal policy for the target we expect should be escaping and disappearing from the observation range of the tracker but keeping close to the tracker.
Revised: Alternatively, by applying this reward function, the optimal policy for the target we expect should be escaping and disappearing from the observation range of the tracker but keeping close to the tracker.

Original: In Appendix A we visualize the reward structure given in Eq.
Revised: In Appendix A we visualize the reward structure given in Eq.

Original: (5) and (6) for a better understanding.
Revised: (5) and (6) for a better understanding.

Original: 3. T RACKER - AWARE T ARGET
Revised: 3. T RACKER - AWARE T ARGET

Revised: By tracker-awareness, the target would be “stronger” than the tracker, as it knows what the tracker knows.

Original: The “tracker-aware" idea is inspired by an ancient Chinese proverb, “know the enemy, know yourself, and in every battle you will be victorious", from the masterpiece Sun Tzu on the Art of War Tzu (2008).
Revised: This idea manifests an ancient Chinese proverb, “know the enemy, know yourself, and in every battle you will be victorious", from the masterpiece Sun Tzu on the Art of War Sun (2008).

Original: The conventional adversary usually uses only its own observation Bansal et al.
Revised: The conventional adversary usually uses only its own observation Bansal et al.

Original: (2017) or shares the same observation as the opponent Pinto et al.
Revised: (2017) or shares the same observation as the opponent Pinto et al.

Original: Recall the target policy π β ( a β | o β ; θ β ) written as in Eq.
Revised: Recall the target policy π β ( a βt | o βt ; θ β ) written as in Eq.

Original: However, the imperfect/partial o β observation seriously degrades the performance of the adversary.
Revised: However, the imperfect/partial o βt observation seriously degrades the performance of the adversary.

Original: Thus, we propose a “tracker-aware" model for the target.
Revised: Thus, we propose a “tracker-aware" model for the target.

Original: Besides the target’s own observation, we additionally feed the observation and action from the tracker into the target network, in order to enrich the input information of the target.
Revised: Besides the target’s own observation, we additionally feed the observation and action from the tracker into the target network, in order to enrich the input information of the target.

Original: Moreover, we add an auxiliary task, which predicts the immediate reward of the tracker, see the TRP module in Fig.
Revised: Moreover, we add an auxiliary task, which predicts the immediate reward of the tracker, see the TRP module in Fig.

Original: This auxiliary task can be seen as a kind of “opponent modeling”, and alleviate the difﬁculty in its own policy learning.
Revised: This auxiliary task can be seen as a kind of “opponent modeling”, and alleviate the difﬁculty in its own policy learning.

Original: By doing so, we can write the output heads of such a “tracker-aware" policy network as π β (cid:16) a βt , ˆ r αt | o αt , a αt , o βt  (cid:17) in a slight abuse of notations, where ˆ r αt is the predicted immediate reward for the tracker and o αt , a αt are respectively the observation and the action of the tracker.
Revised: By doing so, we can write the output heads of such a “tracker-aware" policy network as π β (cid:16) a βt , ˆ r αt | o αt , a αt , o βt ; θ β (cid:17) in a slight abuse of notations, where ˆ r αt is the predicted immediate reward for the tracker and o αt , a αt are respectively the observation and the action of the tracker.

Revised: Empirical results show that the tracker-aware target yields more diversiﬁed escaping policy and ﬁnally helps producing a more robust tracker.

Revised: Note that we cannot apply the trick for a tracker, as the tracker has to use its own observation during testing/deployment.

Original:  E XPERIMENTS 4. E NVIRONMENTS
Revised:  E XPERIMENTS 4. E NVIRONMENTS

Original: 2D Environments.
Revised: 2D Environments.

Original: In the 2D Environment, maps are represented by a 80 × 80 matrix, wheredenotes free space, 1 denotes an obstacle, 2 denotes the tracker, and 4 denotes the target.
Revised: In the 2D Environment, maps are represented by a 80 × 80 matrix, wheredenotes free space, 1 denotes an obstacle, 2 denotes the tracker, and 4 denotes the target.

Original: We generate the maps of two patterns, “random maze" and “random block" (see examples in the top row of Fig.2).
Revised: We generate the maps of two patterns, “random maze" and “random block" (see examples in the top row of Fig.2).

Original: We use maps of block for training and maps of maze for testing.
Revised: We use maps of block for training and maps of maze for testing.

Original: For both the tracker and the target, the observation is a matrix of size 13 × 13 around the agent.
Revised: For both the tracker and the target, the observation is a matrix of size 13 × 13 around the agent.

Original: The tracker aims to place the target in the center of the observation.
Revised: The tracker aims to place the target in the center of the observation.

Original: During each episode, the tracker starts from a free space in the map randomly, and the target starts around the tracker in a 3 × 3 tracker-centric matrix.
Revised: During each episode, the tracker starts from a free space in the map randomly, and the target starts around the tracker in a 3 × 3 tracker-centric matrix.

Original: The 2D environments provide ideal conditions due to lack of observation or motion noise.
Revised: The 2D environments provide ideal conditions due to lack of observation or motion noise.

Original: The experiments in the 2D environments are dedicated to evaluate and quantify the effectiveness of our approach in ideal conditions.
Revised: The experiments in the 2D environments are dedicated to evaluate and quantify the effectiveness of our approach in ideal conditions.

Original: 3D Environments.
Revised: 3D Environments.

Original: The 3D environments are more similar to the real-world setting in observation and action.
Revised: The 3D environments are more similar to the real-world setting in observation and action.

Original: In the 3D environments, the observation is an image of the ﬁrst-person view of 
Revised: In the 3D environments, the observation is an image of the ﬁrst-person view of the world as seen by the agent.

Revised: The possible actions are move-forward, move-backward, turn-left, turn-right, turn-left-and-move-forward, turn-right-and-move-forward , and no-op .

Original: UnrealEnv is a photo-realistic and ﬂexible simulation environment built on the Unreal Engine.
Revised: UnrealEnv is a photo-realistic and ﬂexible simulation environment built on the Unreal Engine.

Original: We employ UnrealCV
Revised: We employ UnrealCV

Original: (Qiu et al., 2017), which provides convenient APIs, along with a wrapper (Zhonget al., 2017) compatible with OpenAI Gym (Brockman et al., 2016), for interactions between RL
Revised: (Qiu et al., 2017), which provides convenient APIs, along with a wrapper (Zhong

Original: [Figure]
Revised: [Figure]

Original:  algorithms and the UnrealEnv.
Revised: et al., 2017) compatible with OpenAI Gym (Brockman et al., 2016), for interactions between RL algorithms and the UnrealEnv.

Original: For training, we built a Domain Randomized Room(DR Room) with two controllable players (target and tracker).
Revised: For training, we built a Domain Randomized Room(DR Room) with two controllable players (target and tracker).

Original: The domain randomization techniques used could help agents learn better feature representation in terms of visual observation.
Revised: The domain randomization techniques used could help agents learn better feature representation in terms of visual observation.

Original: In testing, we focus on the transferring ability of the tracker to different unseen environments.
Revised: In testing, we focus on the transferring ability of the tracker to different unseen environments.

Original: We use three realistic scenarios,
Revised: We use three realistic scenarios,

Original: Urban City, Snow Village and Parking Lot , to mimic real-world scenes for evaluating.
Revised: Urban City, Snow Village and Parking Lot , to mimic real-world scenes for evaluating.

Original: The bottom row in Fig.2 shows the snapshots of the four 3D environments used.
Revised: The bottom row in Fig.2 shows the snapshots of the four 3D environments used.

Original: Details of the four environments are: (1) DR Room is a plain room of ﬂoor and walls only, but the textures and illumination conditions are randomized.
Revised: Details of the four environments are: (1) DR Room is a plain room of ﬂoor and walls only, but the textures and illumination conditions are randomized.

Original: For the textures, we randomly choose pictures from a texture dataset (Kylberg, 2011) and place them on the surface of walls, ﬂoor, and the players.
Revised: For the textures, we randomly choose pictures from a texture dataset (Kylberg, 2011) and place them on the surface of walls, ﬂoor, and the players.

Original: For the illumination condition, we randomize the intensity and color of each light source as well as each position, orientation.
Revised: For the illumination condition, we randomize the intensity and color of each light source as well as each position, orientation.

Original: (2) Urban City is a street view of urban city, including well modelled buildings, streets, trees and transportation facilities with high ﬁdelity.
Revised: (2) Urban City is a street view of urban city, including well modelled buildings, streets, trees and transportation facilities with high ﬁdelity.

Original: Besides, there are some puddles on the road, reﬂecting the buildings.
Revised: Besides, there are some puddles on the road, reﬂecting the buildings.

Original: (3) Snow Village consists of bumpy snowﬁelds with several trees, bushes and some cabins.
Revised: (3) Snow Village consists of bumpy snowﬁelds with several trees, bushes and some cabins.

Original: Speciﬁcally, the bumpy snowﬁeld is challenging for tracking, because it frequently causes the situation that agent observation is not in the same height.
Revised: Speciﬁcally, the bumpy snowﬁeld is challenging for tracking, because it frequently causes the situation that agent observation is not in the same height.

Original: (4) Parking Lot is an underground parking lot with complex illumination condition.
Revised: (4) Parking Lot is an underground parking lot with complex illumination condition.

Original: The lack of light source makes the illumination uneven, i.e. , some places are bright but the others are dark.
Revised: The lack of light source makes the illumination uneven, i.e. , some places are bright but the others are dark.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We provide two kinds of base target agent to randomly generate trajectories as baselines to compare,
Revised: We provide two kinds of base target agent to randomly generate trajectories as baselines to compare,

Original: Rambler (Ram) and Navigator (Nav) .
Revised: Rambler (Ram) and Navigator (Nav) .

Original: Agent Ram walks randomly without any purpose like a man.
Revised: Agent Ram walks randomly without any purpose like a man.

Original: Technically, it randomly samples actions from the action space and keeps executing the action n times, where n is also a random integer in the range of (1 , 10) .
Revised: Technically, it randomly samples actions from the action space and keeps executing the action n times, where n is also a random integer in the range of (1 , 10) .

Original: Agent Nav is a navigator, planning the shortest path to a speciﬁc goal in use of the Astar algorithm (Hart et al., 1968).
Revised: Agent Nav is a navigator, planning the shortest path to a speciﬁc goal in use of the Astar algorithm (Hart et al., 1968).

Original: Thus, it could navigate to most of free space in the map.
Revised: Thus, it could navigate to most of free space in the map.

Original: To randomize the trajectories, the goal coordinate and the initial coordinate are randomly sampled from the free space.
Revised: To randomize the trajectories, the goal coordinate and the initial coordinate are randomly sampled from the free space.

Original: In most of case, Ram prefers to walk around a local area repeatedly.
Revised: In most of case, Ram prefers to walk around a local area repeatedly.

Original: In contrast, Nav would like to explore the map globally, shown as the yellow trajectories in Fig.
Revised: In contrast, Nav would like to explore the map globally, shown as the yellow trajectories in Fig.

Original: Thus we regard trajectories from the Ram as easier cases, and trajectories from Nav as more difﬁcult cases for tracker.
Revised: Thus we regard trajectories from the Ram as easier cases, and trajectories from Nav as more difﬁcult cases for tracker.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Each agent is trained by A3C (Mnih et al., 2016), a commonly used reinforcement learning algorithm.
Revised: Each agent is trained by A3C (Mnih et al., 2016), a commonly used reinforcement learning algorithm.

Original: Multiple workers are running in parallel when training.
Revised: Multiple workers are running in parallel when training.

Original: Speciﬁcally, 16 workers are used in the 2D experiment, and 4 workers are used in the 3D experiment.
Revised: Speciﬁcally, 16 workers are used in the 2D experiment, and 4 workers are used in the 3D experiment.

Original: Network Architecture.
Revised: Network Architecture.

Original: For the tracker, we follow the end-to-end Conv-LSTM network architecture as (Luo et al., 2018).
Revised: For the tracker, we follow the end-to-end Conv-LSTM network architecture as (Luo et al., 2018).

Original: Differently, the Conv-Net in this paper is deeper, and the input color images are pre-processed, transformed to gray image and the pixel values are scaled to [ − 1 , 1] .
Revised: Differently, the Conv-Net in this paper is deeper, and the input color images are pre-processed, transformed to gray image and the pixel values are scaled to [ − 1 , 1] .

Original: we also develop the Conv-LSTM network architecture for the target, but different in the input and output, shown as
Revised: we also develop the Conv-LSTM network architecture for the target, but different in the input and output, shown as

Original: Fig.1.
Revised: Fig.1.

Original: The network parameters are updated with a shared Adam optimizer.
Revised: The network parameters are updated with a shared Adam optimizer.

Original: Hyper parameters.
Revised: Hyper Parameters.

Original: For the tracker, the learning rates δ α 0 and δ α 1 (cid:48) in 2D and 3D environments are 0.001 and 0.0001, respectively.
Revised: For the tracker, the learning rates δ α 0 and δ α 1 (cid:48) in 2D and 3D environments are 0.001 and 0.0001, respectively.

Original: The reward discount factor γ = 0 .
Revised: The reward discount factor γ = 0 .

Original: 9 , generalized advantage estimate parameter τ = 1 . 00 , and regularizer factor for tracker λ α = 0 .
Revised: 9 , generalized advantage estimate parameter τ = 1 . 00 , and regularizer factor for tracker λ α = 0 .

Original: The parameter updating frequency n is 20, and the maximum global iteration for training is 150 K. Comparing to the tracker, a higher regularizer factor is used for encouraging the target to explore, λ β 0 = 0 .
Revised: The parameter updating frequency n is 20, and the maximum global iteration for training is 150 K. Comparing to the tracker, a higher regularizer factor is used for encouraging the target to explore, λ β 0 = 0 .

Original: 2 in 2D and λ β 1 = 0 .
Revised: 2 in 2D and λ β 1 = 0 .

Original: in 3D.
Revised: in 3D.

Original: The more exploration taken by target, the more diverse the generated trajectories are.
Revised: The more exploration taken by target, the more diverse the generated trajectories are.

Original: It is useful for the learning of the tracker.
Revised: It is useful for the learning of the tracker.

Original: Validation is performed in parallel and the best validation network model is applied to report performance in testing environments.
Revised: Validation is performed in parallel and the best validation network model is applied to report performance in testing environments.

Original: Note that the validation environment is of the same settings as training, except that the target is controlled by a Nav agent.
Revised: Note that the validation environment is of the same settings as training, except that the target is controlled by a Nav agent.

Original: Compared with the
Revised: Compared with the

Original: Ram agents, the Nav agent is more challenging, thus is more suitable for validation.
Revised: Ram agent, the Nav agent is more challenging, thus is more suitable for validation.

Original: Metric.
Revised: Metric.

Original: Two metrics are employed for the experiments.
Revised: Two metrics are employed for the experiments.

Original: Speciﬁcally, Accumulated Reward (AR) and
Revised: Speciﬁcally, Accumulated Reward (AR) and

Original: Episode Length (EL) of each episode are calculated for quantitative evaluation.
Revised: Episode Length (EL) of each episode are calculated for quantitative evaluation.

Original: AR is a comprehensive metric, representing the tracker’s capability about precision and robustness.
Revised: AR is a comprehensive metric, representing the tracker’s capability about precision and robustness.

Original: It is effected by the immediate reward and the episode length.
Revised: It is effected by the immediate reward and the episode length.

Original: Immediate reward measures the goodness of tracking, and
Revised: Immediate reward measures the goodness of tracking, and

Original: EL roughly measures the duration of good tracking.
Revised: EL roughly measures the duration of good tracking.

Original: Because the episode is done when the tracker lost for continuous 10 steps or reach the max episode length.
Revised: Because the episode is done when the tracker lost for continuous 10 steps or reach the max episode length.

Original: 4. R ESULTS ON THE 2D ENVIRONMENT
Revised: 4. R ESULTS ON THE 2D ENVIRONMENT

Original: We quantitatively evaluate the performance of our approach, comparing to the two baselines.
Revised: We quantitatively evaluate the performance of our approach, comparing to the two baselines.

Original: Furthermore, we conduct an ablation study to show the effect of thetracker-aware model and partial zero-sum reward.
Revised: Furthermore, we conduct an ablation study to show the effect of the partial zero-sum reward and tracker-aware model.

Original: Quantitative Evaluation.
Revised: Quantitative Evaluation.

Original: We test the active tracker trained with different target agents in four testing settings, showing the effectiveness of AD-VAT .
Revised: We test the active tracker trained with different target agents in four testing settings, showing the effectiveness of AD-VAT .

Original: Considering the random seed of the environments, we conduct 100 runs in each and report the mean and standard deviation of AR and EL, shown in
Revised: Considering the random seed of the environments, we conduct 100 runs in each and report the mean and standard deviation of AR and EL, shown in

Original: Table 1.
Revised: Table 1.

Original: The max episode length is 500, so the upper bound of EL is 500.
Revised: The max episode length is 500, so the upper bound of EL is 500.

Original: Thus, when EL equals to 500, we could infer that the tracker performs perfectly, without losing the target.
Revised: Thus, when EL equals to 500, we could infer that the tracker performs perfectly, without losing the target.

Original: We note that, at the beginning of the learning, the adversarial target usually walks randomly around the start point, performing similar policy as Ram .
Revised: We note that, at the beginning of the learning, the adversarial target usually walks randomly around the start point, performing similar policy as Ram .

Original: Such target is easier to be found and observed, even though the tracker is in exploration.
Revised: Such target is easier to be found and observed, even though the tracker is in exploration.

Original: Thus, the tracker could warm up faster.
Revised: Thus, the tracker could warm up faster.

Original: With the growth of the tracker, the target gradually explores other motion pattern, which could further reinforce the tracker.
Revised: With the growth of the tracker, the target gradually explores other motion patterns, which could further reinforce the tracker.

Original: Such a learning process is close to the curriculum learning, but the curriculum is automatically produced by the target via adversarial reinforcement learning.
Revised: Such a learning process is close to the curriculum learning, but the curriculum is automatically produced by the target via adversarial reinforcement learning.

Original: We also report the learning curve as the mean of cumulative rewards in the validation environment, shown as the left sub-ﬁgure in Fig 3.
Revised: We also report the learning curve as the mean of cumulative rewards in the validation environment, shown as the left sub-ﬁgure in Fig 3.

Original: It consistently shows the advantage of the proposed AD-VAT .
Revised: It consistently shows the advantage of the proposed AD-VAT .

Original: Ablation Study.
Revised: Ablation Study.

Original: In Section 3, we introduced two components to implement AD-VAT : “partial zerosum" reward structure (PZR) and tracker-aware model (TAM) for target.
Revised: In Section 3, we introduced two components to implement AD-VAT : “partial zerosum" reward structure (PZR) and tracker-aware model (TAM) for target.

Original: These two components are important as they inﬂuence the natural curriculum for the tracker.
Revised: These two components are important as they inﬂuence the natural curriculum for the tracker.

Original: Thus, we report an ablation study result to show the effectiveness of these two components, shown in Fig.
Revised: Thus, we report an ablation study result to show the effectiveness of these two components, shown in Fig.

Original: The naive method is an intuitive idea that target only uses its own observation with auxiliary task, guided by a zero-sum reward.
Revised: The naive method is an intuitive idea that target only uses its own observation with auxiliary task, guided by a zero-sum reward.

Original: As shown in the right sub-ﬁgure of Fig.
Revised: As shown in the right sub-ﬁgure of Fig.

Original: 3, using each component separately could improve 
Revised: 3, using each component separately could improve the sample-efﬁciency, comparing to the naive method. Besides, PZR contributes to the improvement

Original: Environment
Revised: Environment

Original: [Table]
Revised: [Table]

Original: Map
Revised: Map

Original: Target
Revised: Target

Revised: Ram

Original: [Table]
Revised: [Table]

Original: Maze
Revised: Maze

Original: Ram
Revised: [Figure] 350 ±287 ± [Table]

Original: [Table]
Revised: [Table]

Revised: Nav

Revised: [Table]

Original: Block
Revised: Block

Original:  [Table]
Revised: [Figure] [Table] [Table] [Table]

Revised: Nav

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: the sample-efﬁciency, comparing to the naive method.
Revised: [Table] [Table] [Table]

Original: Besides, PZR contributes to the improvement of the tracking performance more signiﬁcant than TAM.
Revised: [Table] [Table] [Figure] of the tracking performance more signiﬁcant than TAM.

Original: Moreover, when combining PZR and TAM, both sample-efﬁciency and the tracking performance are signiﬁcantly boosted, comparing to the other three settings.
Revised: Moreover, when combining PZR and TAM, both sample-efﬁciency and the tracking performance are signiﬁcantly boosted, comparing to the other three settings.

Original: 4. R ESULTS ON THE 3D ENVIRONMENT
Revised: 4. R ESULTS ON THE 3D ENVIRONMENT

Original: In the 3D UnrealEnv, we test the effectiveness of the model in unseen realistic scenarios, showing the transfer potential in real-world scenario.
Revised: In the 3D UnrealEnv, we test the effectiveness of the model in unseen realistic scenarios, showing the transfer potential in real-world scenario.

Original: We train the three models with different target models ( Ram,
Revised: We train the three models with different target models ( Ram,

Original: Nav , and AD-VAT ) in the DR Room.
Revised: Nav , and AD-VAT ) in the DR Room.

Original: And then, we directly run the three trackers in the validation and testing environments, 100 episodes for each, and report AR and EL in Fig.
Revised: And then, we directly run the three trackers in the validation and testing environments, 100 episodes for each, and report AR and EL in Fig.

Original: The result in DR Room demonstrates again that even though the target model is unseen for our
Revised: The result in DR Room demonstrates again that even though the target model is unseen for our

Original: AD-VAT tracker, it still outperforms the others.
Revised: AD-VAT tracker, it still outperforms the others.

Original: Because the DR Room for validation is of the same settings as the training of Nav tracker.
Revised: Because the DR Room for validation is of the same settings as the training of Nav tracker.

Original: The results in UrbanCity show that the three models are able to transfer to realistic environment.
Revised: The results in UrbanCity show that the three models are able to transfer to realistic environment.

Original: We believe that the domain randomization method and the Conv-LSTM network endow the trackers the ability of transferring.
Revised: We believe that the domain randomization method and the Conv-LSTM network endow the trackers the ability of transferring.

Original: However, the performance of the two baselines degrades more compared with our method, in other environments.
Revised: However, the performance of the two baselines degrades more compared with our method, in other environments.

Original: So we think that the two baseline methods are not sufﬁciently effective when transferring to challenging environments.
Revised: So we suspect that the two baseline methods are not sufﬁciently effective when transferring to challenging environments.

Original: Training with AD-VAT could signiﬁcantly improve the capability of the tracker in challenging environments.
Revised: Training with AD-VAT could signiﬁcantly improve the capability of the tracker in challenging environments.

Original: We infer that the target in AD-VAT could explore the environment more actively to discover more challenging cases.
Revised: We infer that the target in AD-VAT could explore the environment more actively to discover more challenging cases.

Original: For example, in DR Room, the target would prefer to move close to the wall that is similar to itself to fool the tracker(see Fig. 5).
Revised: For example, in DR Room, the target would prefer to move close to the wall that is similar to itself to fool the tracker (see Fig. 5).

Original: By competing with the target, the tracker consequently becomes stronger.
Revised: By competing with the target, the tracker consequently becomes stronger.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have proposed a mechanismAD-VAT for visual active tracking.
Revised: In this paper, we have proposed an asymmetric dueling mechanism for visual active tracking (ADVAT).

Original: Within AD-VAT, agents of tracker and target are learned in an adversarial manner.
Revised: Within AD-VAT, agents of tracker and target are learned in an adversarial manner.

Original: With the design of the “partial zerosum" reward structure and the tracker-aware target model, the reinforced active tracker outperforms baseline methods.
Revised: With the design of the “partial zero-sum" reward structure and the tracker-aware target model, the reinforced active tracker outperforms baseline methods.

Original: Experiments including ablation study in both 2D and 3D environments verify the effectiveness of the proposed mechanism.
Revised: Experiments including ablation study in both 2D and 3D environments verify the effectiveness of the proposed mechanism.

Original: A V ISUALIZING P ARTIAL Z ERO - SUM R EWARD
Revised: A V ISUALIZING P ARTIAL Z ERO - SUM R EWARD

Original: To help better understand the reward structure given in Eq.
Revised: To help better understand the reward structure given in Eq.

Original: (5) and (6), we visualize the sum r α + r β as heatmap in x − y plane.
Revised: (5) and (6), we visualize the sum r α + r β as heatmap in x − y plane.

Original: See Fig. 6.
Revised: See Fig. 6.

Original: For the 2D experiment, the observations for both the tracker and target are bird-views.
Revised: For the 2D experiment, the observations for both the tracker and target are bird-views.

Original: We want to penalize that the target gets too far away from the tracker.
Revised: We want to penalize that the target gets too far away from the tracker.

Original: Therefore, the zero-sum area is a circle (Fig. 6, Left), where the tracker is in the centre.
Revised: Therefore, the zero-sum area is a circle (Fig. 6, Left), where the tracker is in the centre.

Original: With the increasing of the distance, the penalty term in r β (see Eq. 6) starts taking effect on the sum.
Revised: With the increasing of the distance, the penalty term in r β (see Eq. 6) starts taking effect on the sum.

Original: It causes the sum to reduce gradually until the target reaches the dark area, where r β = − A .
Revised: It causes the sum to reduce gradually until the target reaches the dark area, where r β = − A .

Original: For the 3D experiment, the observations for both the tracker and target are front-views.
Revised: For the 3D experiment, the observations for both the tracker and target are front-views.

Original: We want to penalize that the target gets too far away from the tracker or that the target cannot be seen by the tracker.
Revised: We want to penalize that the target gets too far away from the tracker or that the target cannot be seen by the tracker.

Original: Thus, the zero-sum area is a sector area (Fig. 6, Right), which approximately ﬁts the Field of View (FoV) of the tracker’s camera.
Revised: Thus, the zero-sum area is a sector area (Fig. 6, Right), which approximately ﬁts the Field of View (FoV) of the tracker’s camera.

Original: Note that the FoV in our experiment is 90 degree.
Revised: Note that the FoV in our experiment is 90 degree.

Original: Both the relative angle θ and distance ρ contribute to the penalty term in r β .
Revised: Both the relative angle θ and distance ρ contribute to the penalty term in r β .

Original: Thus the sum decreases like a divergence sector.
Revised: Thus the sum decreases like a divergence sector.

Original: B V ISUALIZING THE T RAINING P ROCESS
Revised: B V ISUALIZING THE T RAINING P ROCESS

Original: For a better understanding of the training process, we record trajectories of the target and the tracker during different training stages.
Revised: For a better understanding of the training process, we record trajectories of the target and the tracker during different training stages.

Original: Speciﬁcally, we have 6 stages, ranging from early training to late training.
Revised: Speciﬁcally, we have 6 stages, ranging from early training to late training.

Original: For each stage we record 100 episodes.
Revised: For each stage we record 100 episodes.

Original: In Fig.
Revised: In Fig.

Original: 7, we plot the target position distribution, instead of the trajectory itself.
Revised: 7, we plot the target position distribution, instead of the trajectory itself.

Original: For ease of visualization, we adopt a relative coordinate system for the target position when drawing the distribution, because the start locations for both the tracker and the target are random upon each episode.
Revised: For ease of visualization, we adopt a relative coordinate system for the target position when drawing the distribution, because the start locations for both the tracker and the target are random upon each episode.

Original: In Fig.
Revised: In Fig.

Original: 7, the distributions are in a start point -centric coordinate system, while in Fig.
Revised: 7, the distributions are in a start point -centric coordinate system, while in Fig.

Original: 8 the distributions are in a tracker -centric coordinate system.
Revised: 8 the distributions are in a tracker -centric coordinate system.

Original: At early training stages (See the left of Fig. 7), AD-VAT and Ram generate similar trajectories, which are randomly walking around the start point.
Revised: At early training stages (See the left of Fig. 7), AD-VAT and Ram generate similar trajectories, which are randomly walking around the start point.

Original: In contrast, for Nav method the target usually goes along a straight line to the goal position, causing the tracker to get lost quickly at beginning.
Revised: In contrast, for Nav method the target usually goes along a straight line to the goal position, causing the tracker to get lost quickly at beginning.

Original: The randomly walking trajectories help the tracker observe the target appearing in various positions, and henceforth sample more diverse experiences.
Revised: The randomly walking trajectories help the tracker observe the target appearing in various positions, and henceforth sample more diverse experiences.

Original: As a result, a better exploration is achieved during the early training stage, which is not the case for the Nav method.
Revised: As a result, a better exploration is achieved during the early training stage, which is not the case for the Nav method.

Original: With the evolution of the tracker during training, the target will gradually seek more difﬁcult cases to defeat the tracker.
Revised: With the evolution of the tracker during training, the target will gradually seek more difﬁcult cases to defeat the tracker.

Original: In this regard, the target for both Ram and Nav usually explore possible directions uniformly.
Revised: In this regard, the target for both Ram and Nav usually explore possible directions uniformly.

Original: The minor difference is that the Nav tends to explore the map globally, while the Ram is in local (See the right of Fig. 7).
Revised: The minor difference is that the Nav tends to explore the map globally, while the Ram is in local (See the right of Fig. 7).

Original: As for our AD-VAT method, however, the reinforced target could adapt to the capability of the tracker, resulting to different direction choosing patters at different stage.
Revised: As for our AD-VAT method, however, the reinforced target could adapt to the capability of the tracker, resulting to different direction choosing patters at different stage.

Original: For example, the target tends to move bottom-right at the third stage, but top-left at the fourth stage.
Revised: For example, the target tends to move bottom-right at the third stage, but top-left at the fourth stage.

Original: See
Revised: See

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Fig.
Revised: Fig.

Original: Besides, it seems that the reinforced target could balance the two exploration modes of Nav and Ram naturally.
Revised: Besides, it seems that the reinforced target could balance the two exploration modes of Nav and Ram naturally.

Original: Sometimes it explores the map, and sometimes it duels with the tracker locally.
Revised: Sometimes it explores the map, and sometimes it duels with the tracker locally.

Original: By the dueling mechanism, the target could ﬁnd the weakness of the tracker more often (See the right of Fig.
Revised: By the dueling mechanism, the target could ﬁnd the weakness of the tracker more often (See the right of Fig.

Original: 8), which seems to serve as a kind of importance sampling that enhances the tracker efﬁciently during training.
Revised: 8), which seems to serve as a kind of importance sampling that enhances the tracker efﬁciently during training.

Original: Such a "weakness-ﬁnding" seems absent for the Nav and Ram algorithms.
Revised: Such a "weakness-ﬁnding" seems absent for the Nav and Ram algorithms.

Original: C P ASSIVELY T EST T RACKER ON R EAL -W ORLD V IDEO C LIPS
Revised: C P ASSIVELY T EST T RACKER ON R EAL -W ORLD V IDEO C LIPS

Original: To demonstrate the capability of our tracker in real-world scenarios, we conduct a qualitative evaluation as Luo et al.
Revised: To demonstrate the capability of our tracker in real-world scenarios, we conduct a qualitative evaluation as Luo et al.

Original: In this evaluation, we feed the video clips from VOT dataset (Kristan et al., 2016) to the tracker and observe the network output actions.
Revised: In this evaluation, we feed the video clips from VOT dataset (Kristan et al., 2016) to the tracker and observe the network output actions.

Original: Note that the tracker could not really control the camera movement, hence we regard the test as “passively".
Revised: Note that the tracker could not really control the camera movement, hence we regard the test as “passively".

Original: However, the tracker’s output action is expected to be sensitive to the position and the scale of the target in the image.
Revised: However, the tracker’s output action is expected to be sensitive to the position and the scale of the target in the image.

Original: For example, when the target appears in the right (left) side, the tracker tends to turn right (left), trying to ﬁctitiously move the camera to “place” the target in the image center.
Revised: For example, when the target appears in the right (left) side, the tracker tends to turn right (left), trying to ﬁctitiously move the camera to “place” the target in the image center.

Original: By visualizing whether the output action is consistent with the position and scale of the target at each frame, we are able to demonstrate the potential of transferring the tracking ability to real-world.
Revised: By visualizing whether the output action is consistent with the position and scale of the target at each frame, we are able to demonstrate the potential of transferring the tracking ability to real-world.

Original: We plot three “Action" maps, shown in Fig.
Revised: We plot three “Action" maps, shown in Fig.

Original: 9, Fig. 10, and Fig.
Revised: 9, Fig. 10, and Fig.

Original: 11, respectively.
Revised: 11, respectively.

Original: Note that the meaning of each axis is the same as (Luo et al., 2018) except that we normalize the values for better understanding.
Revised: Note that the meaning of each axis is the same as (Luo et al., 2018) except that we normalize the values for better understanding.

Original: In details, the horizontal axis indicates the normalized x-axis position of the target in the image, with a positive (negative) value meaning that a target is in the right (left) side.
Revised: In details, the horizontal axis indicates the normalized x-axis position of the target in the image, with a positive (negative) value meaning that a target is in the right (left) side.

Original: The vertical axis indicates the normalized size of the target, i.e. , the area of the ground truth bounding box.
Revised: The vertical axis indicates the normalized size of the target, i.e. , the area of the ground truth bounding box.

Original: We use seven marks to represent the seven actions respectively, as shown in the legend.
Revised: We use seven marks to represent the seven actions respectively, as shown in the legend.

Original: More results on other VOT videos are available at: https://youtu.be/jv-5HVg_Sf4 .
Revised: More results on other VOT videos are available at: https://youtu.be/jv-5HVg_Sf4 .