Original: AD-VAT: A N ASYMMETRIC DUELING MECHANISM FOR LEARNING VISUAL ACTIVE TRACKING
Revised: AD-VAT: A N ASYMMETRIC DUELING MECHANISM FOR LEARNING VISUAL ACTIVE TRACKING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.
Revised: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.

Original: Previous work has shown a Single Agent Reinforcement Learning (SARL) approach, where the tracker can be trained in a simulator  and performs tracking in real-world scenarios.
Revised: Previous work has shown that the tracker can be trained in a simulator via reinforcement learning and deployed in real-world scenarios.

Original: However, during training, suchSARL method requires manually specifying the moving path of the target object to be tracked, which can hurt the tracker’s generalization on unseen object moving patterns.
Revised: However, during training, such method requires manually specifying the moving path of the target object to be tracked, which cannot ensure the tracker’s generalization on unseen object moving patterns

Original: To learn a robust tracker for VAT, in this paper we propose a novel Multi-Agent RL(MARL) training method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT.
Revised: To learn a robust tracker for VAT, in this paper, we propose a novel adversarial RL method which adopts an Asymmetric Dueling mechanism , referred to as AD -VAT

Original: In AD-VAT, both the tracker and the target are approximated by deep networks and are trained viaend to end RL in a dueling/competitive manner: i e. , the tracker intends to lockup the target while the target tries to escape from the tracker.
Revised: In AD-VAT , both the tracker and the target are approximated by end-to-end neural networks and are trained via RL in a dueling/competitive manner: i e. , the tracker intends to lockup the target, while the target tries to escape from the tracker.

Original: They are asymmetric in that the target is aware of the tracker, but not vice versa.
Revised: They are asymmetric in that the target is aware of the tracker but not vice versa.

Original: Speciﬁcally, besides its own observation, the target is fed with the tracker’s observation and action and learns to predict the tracker’s reward as an auxiliary task.
Revised: Speciﬁcally, besides its own observation, the target is fed with the tracker’s observation and action and learns to predict the tracker’s reward as an auxiliary task.

Original: We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker.
Revised: We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker.

Original: To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target.
Revised: To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target.

Original: The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios
Revised: The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios.

Original: For supplementary videos, see: https://www.youtube.com/playlist?
Revised: For supplementary videos, see: https://www.youtube.

Original: list= PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS
Revised: com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.
Revised: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.

Original: VAT is demanded in many real-world applications such as autonomous vehicle ﬂeet ( e.g. , a slave-vehicle should follow a master-vehicle ahead), service robots and drones ( e.g. , a drone is required to follow a person when recording a video).
Revised: VAT is demanded in many real-world applications such as autonomous vehicle ﬂeet ( e.g. , a slave-vehicle should follow a master-vehicle ahead), service robots and drones ( e.g. , a drone is required to follow a person when recording a video).

Original: To accomplish the VAT task, one typically needs to perform a sequence of tasks such as recognition, localization, motion prediction, and camera control.
Revised: To accomplish the VAT task, one typically needs to perform a sequence of tasks such as recognition, localization, motion prediction, and camera control.

Original: However, conventional visual tracking (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009; Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012) aims to solely propose a 2D bounding box of the target frame by frame, and does not actively take into consideration the camera-control.
Revised: However, conventional visual tracking (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009; Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012) aims to solely propose a 2D bounding box of the target frame by frame, and does not actively take into consideration the control of camera.

Original: Thus, compared to the problem of “passive" tracking,
Revised: Thus, compared to the problem of “passive” tracking,

Original: VAT is more practical and challenging.
Revised: VAT is more practical and challenging.

Original: With the advancement of deep reinforcement learning (Sutton & Barto, 1998;
Revised: With the advancement of deep reinforcement learning (Sutton & Barto, 1998;

Original: Mnih et al., 2015;
Revised: Mnih et al., 2015;

Original: Silver et al., 2016b; Mnih et al., 2016), training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible (Luo et al., 2018).
Revised: Silver et al., 2016; Mnih et al., 2016), training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible (Luo et al., 2018; Luo et al., 2019).

Original: The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network.
Revised: The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network.

Original: Not only can it save the effort in tuning an extra camera controller, but also does it outperform the conventional methods where the passive tracker is equipped with a hand-engineered camera-control module.
Revised: Such an end-to-end approach could save the effort of tuning an extra camera controller. Meanwhile, it also [Figure] outperforms the conventional methods where the passive tracker is equipped with a hand-engineered camera controller.

Original: However, the performance of the deep reinforcement learning based tracker is still limited by the training methods.
Revised: However, the performance of the deep reinforcement learning based tracker is still limited by the training methods.

Original: Due to the “trial-and-error" nature of reinforcement learning, it is infeasible to directly train the tracker in  real world.
Revised: Due to the “trial-and-error” nature of reinforcement learning, it is infeasible to directly train the tracker in the real world.

Original: Alternatively, virtual environments are always utilized to generate sufﬁcient data for training without tedious human labeling.
Revised: Alternatively, virtual environments are always utilized to generate sufﬁcient data for training without tedious human labeling.

Original: Nevertheless, to deploy the trained tracker in real-world, one has to overcome the gap between virtual and real environments.
Revised: Nevertheless, to deploy the trained tracker in the real world, one has to overcome the virtual-to-real gap.

Original: One solution can be building numbers of high-ﬁdelity environments (Zhu et al., 2017).
Revised: One solution can be building numbers of high-ﬁdelity environments (Zhu et al., 2017).

Original: However, it is[Figure] expensive and tedious to build such environments for VAT.
Revised: However, it is expensive and tedious to build such environments for VAT.

Original: Both the visual rendering (illumination, texture, etc. ) and the physical properties should be carefully designed to emulate the real world.
Revised: Both the visual rendering (illumination, texture, etc. ) and the physical properties should be carefully designed to emulate the real world.

Original: Suppose we carry out VAT where the target is a pedestrian.
Revised: Suppose we carry out VAT where the target is a pedestrian.

Original: To build the environment, one has to not only model the human’s appearance, but also design physical rules and the pedestrian’s trajectory so that it moves naturally like a human beings.
Revised: To build the environment, one has to not only model the human’s appearance, but also design physical rules and the pedestrian’s trajectory so that it moves naturally like a human beings.

Original: Recently, (Luo et al., 2018) tried to overcome the virtual-real gap by applying the so-called environment augmentation technique.
Revised: Recently, (Luo et al., 2018) tried to overcome the virtual-to-real gap by applying the so-called environment augmentation technique.

Original: They diversify the visual appearance by changing the placement of the background objects and by ﬂipping left-right the screen frame.
Revised: They diversify the visual appearance by changing the placement of the background objects and by ﬂipping left-right the screen frame.

Original: However, they neglect another important factor, that is, the motion of the target for VAT task.
Revised: However, they neglect another important factor, that is, the motion of the target for VAT task.

Original: Intuitively, the complexity and diversity of the target motion in training will impact the generalization of the  tracker.
Revised: Intuitively, the complexity and diversity of the target motion in training will impact the generalization of the data-driven tracker.

Original: For example, if the target only moves forward during training, the tracker may over ﬁt to move straightly and fail to track other motion patterns, like a sharp turn.
Revised: For example, if the target only moves forward during training, the tracker may over ﬁt to move straightly and fail to track other motion patterns, like a sharp turn.

Original: In this work, we propose a novel adversarialMulti-Agent RL  for learning VAT, refereed to as AD-VAT ( Asymmetric Dueling mechanism for learning Visual Active Tracking ).
Revised: In this work, we propose a novel adversarial RL method for learning VAT, refereed to as AD-VAT ( Asymmetric Dueling mechanism for learning Visual Active Tracking ).

Original: In the mechanism, the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during competition (See Fig.1).
Revised: In the mechanism, the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during competition (See Fig. 1).

Original: As the training proceeds, the environments of AD-VAT naturally compose a curriculum, because the tracker is more likely to compete with a target with the appropriate difﬁculty level when both agents are becoming stronger simultaneously.
Revised: As the training proceeds, the environments of AD-VAT naturally compose a curriculum, because the tracker is more likely to compete with a target with the appropriate difﬁculty level when both agents are becoming stronger simultaneously.

Original: When exploring the escape policy, the target consequently generates various trajectories to train the tracker.
Revised: When exploring the escape policy, the target consequently generates various trajectories to train the tracker.

Original: By the dueling/adversarial mechanism, the target is encouraged to discover the weakness of the tracker more often, which could serve as a kind of “weakness-ﬁnding" that makes the trackerbe more robust.
Revised: By the dueling/adversarial mechanism, the target is encouraged to discover the weakness of the tracker more often, which could serve as a kind of “weakness-ﬁnding” that makes the tracker more robust.

Original: However, in practice, using heuristic MARL method for training VAT is unstable and slow to converge.
Revised: However, in practice, using heuristic adversarial RL method for training VAT is unstable and slow to converge.

Original: To address these issues, we derive two components in AD-VAT: the “partial zero-sum reward” and the “tracker-aware network" for target. 
Revised: To address these issues, we derive two components in AD-VAT: partial zero-sum reward(PZR) and tracker-aware model(TAM) for target. PZR is a hybrid reward structure.

Original: The “partial zero-sum” is designed to avoid an observed phenomenon, that when the target quickly learns to be far away from the tracker while the tracker has no chance to see the target any more and henceforth gets plateaus during training.We adopt a novel reward referred to as “partial zero-sum”, which only encourages the tracker-target competition when they are reasonably close and penalizes when they are too far-away.
Revised: It encourages a zero-sum tracker-target competition in the near range, where the target is close to the expected position to tracker; whereas, beyond the zero-sum zone, it is a non-zero-sum game, in which the target is penalized for running too far to track. Such reward structure is inspired by an observed phenomenon, that when the target quickly learns to be far away from the tracker while the tracker has no chance to see the target once more and henceforth gets plateaus during training.

Original: To learn the optimal policy to escape, we model the target with a “tracker-aware network", i.e. , besides its own observation, the observation and actions of the tracker are also fed to the escaping network.
Revised: To learn the optimal policy to escape, we model the target with a “tracker-aware network”, i.e. , besides its own observation, the observation and actions of the tracker are also fed to the escaping network.

Original: In addition, to shape a better representation about the tracker, we add an auxiliary task for the target, learning to predict the immediate reward of the tracker.
Revised: In addition, to shape a better representation about the tracker, we add an auxiliary task for the target, learning to predict the immediate reward of the tracker.

Original: We argue that such an “asymmetric dueling" mechanism is able to learn a stronger target, which vice versa yields a more robust tracker ultimately.
Revised: We argue that such an “asymmetric dueling” mechanism is able to learn a stronger target, which vice versa yields a more robust tracker ultimately.

Original: The experiment is conducted in various 2D and 3D environments for further studying AD-VAT.
Revised: The experiment is conducted in various 2D and 3D environments for further studying AD-VAT.

Original: The 2D environment is a matrix map where obstacles are randomly placed.
Revised: The 2D environment is a matrix map where obstacles are randomly placed.

Original: In the 2D environments, we evaluate and quantify the effectiveness of our approach in an ideal condition, lacking of noise in observation and action.
Revised: In the 2D environments, we evaluate and quantify the effectiveness of our approach in an ideal condition, free from noise in observation and action.

Original: We also conduct an ablation study to show the effectiveness of the two important components: “partial zero-sum reward" and “tracker-aware network".
Revised: We also conduct an ablation study to show the effectiveness of the two important components, “partial zero-sum reward” and “tracker-aware network”.

Original: The 3D environments are built on Unreal Engine 4, a popular game engine for building high-ﬁdelity environments.
Revised: The 3D environments are built on Unreal Engine 4, a popular game engine for building high-ﬁdelity environments.

Original: We choose a large room for training, where the texture of the background/players and the illumination are randomized.
Revised: We choose a large room for training, where the texture of the background/players and the illumination are randomized.

Original: Three realistic scenarios built by artists are used for further evaluating the robustness.
Revised: Three realistic scenarios built by artists are used for further evaluating the robustness.

Original: In the 3D environments, we further demonstrate that the tracker trained in AD-VAT is capable of generalizing to high-ﬁdelity environments even it is trained in a simple environment.
Revised: In the 3D environments, we further demonstrate that the tracker trained in AD-VAT is capable of generalizing to high-ﬁdelity environments even it is trained in a simple environment.

Original: The contributions of our work can be summarized as follows:
Revised: The contributions of our work can be summarized as follows:

Original: • Unlike the Single Agent Reinforcement Learning (SARL) approach (Luo et al., 2018), we propose a novel Multi Agent Reinforcement Learning(MARL) method for VAT task, i.e. , the Asymmetric Dueling mechanism (AD-VAT).
Revised: • We propose a novel Adversarial Reinforcement Learning method for VAT task, i.e. , the Asymmetric Dueling mechanism (AD-VAT).

Original: In AD-VAT, the target learns to generate diverse trajectories when competing with the tracker, which in turn helps train a more robust tracker.
Revised: In AD-VAT, the target learns to generate diverse trajectories when competing with the tracker, which in turn helps train a more robust tracker.

Original: • We provide two techniques to guarantee an efﬁcient yet effective AD-VAT.
Revised: • We provide two techniques to guarantee an efﬁcient yet effective AD-VAT.

Original: A partial zero-sum reward structure, which signiﬁcantly stabilizes the training.
Revised: A partial zero-sum reward structure, which signiﬁcantly stabilizes the training.

Original: A tracker-aware network for the target, which yields better escaping policy and consequently better tracking policy.
Revised: A tracker-aware network for the target, which yields better escaping policy and consequently better tracking policy.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Active Object Tracking.
Revised: Active Object Tracking.

Original: As described above that, active object tracking deals with object tracking and camera control at the same time.
Revised: As described above that, active object tracking deals with object tracking and camera control at the same time.

Original: This problem attracts less attention compared with traditional object tracking (or visual object tracking)
Revised: This problem attracts less attention compared with traditional object tracking (or visual object tracking)

Original: (Wu et al., 2013).
Revised: (Wu et al., 2013).

Original: In general, this problem could be addressed in a two-step manner or in a end-to-end manner.
Revised: In general, this problem could be addressed in a two-step manner or in an end-to-end manner.

Original: In the two-step solution, traditional object tracking and camera control are conducted sequentially to obtain tracking results and manipulate camera.
Revised: In the two-step solution, traditional object tracking and camera control are conducted sequentially to obtain tracking results and manipulate camera.

Original: Great progress has been achieved in traditional object tracking in recent decades (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009;
Revised: Great progress has been achieved in traditional object tracking in recent decades (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009;

Original: Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012).
Revised: Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012).

Original: Thus one can utilize mature visual tracking algorithms (Henriques et al., 2015; Ma et al., 2015; Choi et al., 2017) to accomplish the passive tracking task.
Revised: Thus one can utilize mature visual tracking algorithms (Henriques et al., 2015; Ma et al., 2015; Choi et al., 2017) to accomplish the passive tracking task.

Original: According to the tracking results, camera control module could be developed to actively follow a target.
Revised: According to the tracking results, camera control module could be developed to actively follow a target.

Original: For example, in (Denzler &
Revised: For example, in (Denzler &

Original: Paulus, 1994) a two-stage method is proposed to handle robot control by motion detection and motion tracking.
Revised: Paulus, 1994) a two-stage method is proposed to handle robot control by motion detection and motion tracking.

Original: Kim et al.
Revised: Kim et al.

Original: (Kim et al., 2005) detect moving objects and track them using an active camera with pan/tilt/zoom.
Revised: (Kim et al., 2005) detect moving objects and track them using an active camera with pan/tilt/zoom.

Original: In (Hong et al., 2018), two modules, a perception module and a control policy module, are learned separately to train an agent accomplishing both an obstacle avoidance task and a target following task.
Revised: In (Hong et al., 2018), two modules, a perception module and a control policy module, are learned separately to train an agent accomplishing both an obstacle avoidance task and a target following task.

Original: Admittedly, tracking algorithms have been successful while still not perfect.
Revised: Admittedly, tracking algorithms have been successful while still not perfect.

Original: Camera control based on tracking results is challenging due to factors such as that the correspondence between image space and camera parameter space is unknown.
Revised: Camera control based on tracking results is challenging due to factors such as that the correspondence between image space and camera parameter space is unknown.

Original: Additionally, joint tuning of visual tracking and camera control is expensive and encounters trial-and-errors in real world.
Revised: Additionally, joint tuning of visual tracking and camera control is expensive and encounters trial-and-errors in real world.

Original: In end-to-end methods, direct mapping between raw input frame and camera action is established.
Revised: In end-to-end methods, direct mapping between raw input frame and camera action is established.

Original: Thus the intermediate visual tracking results are not necessarily required.
Revised: Thus the intermediate visual tracking results are not necessarily required.

Original: For instance, by reinforcement learning, camera is controlled by signal outputted from a Conv-LSTM network given raw input frame (Luo et al., 2018).
Revised: For instance, by reinforcement learning, camera is controlled by signal outputted from a Conv-LSTM network given raw input frame (Luo et al., 2018; Luo et al., 2019).

Original: This end-to-end solution veriﬁes the effectiveness, while not efﬁcient enough to solve this problem.
Revised: This end-to-end solution veriﬁes the effectiveness, while not efﬁcient enough to solve this problem.

Original: To improve the generalization potential, environment augmentation is conducted in this work.
Revised: To improve the generalization potential, environment augmentation is conducted in this work.

Original: However, the target itself, like the path, motion pattern is ﬁxed in their augmented environments, which is believed to limit the performance.
Revised: However, the target itself, like the path, motion pattern is ﬁxed in their augmented environments, which is believed to limit the performance.

Original: This inspires us to resort to the idea of dueling in this paper, i.e. , the target learns to get rid of the tracker by itself, guiding the learning of the tracker.
Revised: This inspires us to resort to the idea of dueling in this paper, i.e. , the target learns to get rid of the tracker by itself, guiding the learning of the tracker.

Original: Self-play method for reinforcement learning.
Revised: Adversarial Reinforcement Learning.

Original: In terms of game theory, our setting is  a two-player non-cooperative extensive game with partial observation (Osborne & Rubinstein, 1994).
Revised: Using an adversarial framework to improve the RL agent’s robustness is not a new concept. In (Huang et al., 2017), they add nonrandom adversarial noise to state input for the purpose of altering or misdirecting policies.

Original: When the reward is zero-sum, some self-play algorithm with a speciﬁc value function ( e.g. , Counterfactual
Revised: Mandlekar et al.

Original: Regret, abbreviated as CFR (Zinkevich et al., 2008)) is proven to approximate the Nash Equilibrium, which means no player can obtain more rewards by unilaterally altering its policy, and henceforth produces very strong policy.
Revised: (Mandlekar et al., 2017) use adversarial examples to actively choose perturbations during training in physical domains. In (Pinto et al., 2017), an adversary is introduced to apply adversarial force during training. It produces an agent whose policy is robust to a wide range of noise at testing time.

Original: CFR and its variants ( e.g. , CFR+ (Tammelin, 2014), Monte Carlo CFR (Burch et al., 2012)) are shown to be successful in several applications. For instance, the poker
Revised: Sukhbaatar et al. (Sukhbaatar et al., 2017) pit a virtual agent Alice against another agent Bob, where Alice creates steadily more difﬁcult challenges for Bob to complete.

Original: AI Libratus defeats professional human players in two-player unlimited Texas Hold’em (Brown & Sandholm, 2017), where CFR is utilized in combination of other careful devised techniques.
Revised: In (Held et al., 2017), they optimize a generator network via adversarial training to make an agent automatically produce tasks that are always at the appropriate level of difﬁculty for the agent.

Original: In speciﬁc, the observation sequence is formalized as a game tree, where similar observations/actions are merged (called abstraction) and sub-games are solved efﬁciently.
Revised: Roughly speaking, in these proposed methods the adversary is viewed as a ghost/virtual player, which is unseen and could only challenge the protagonist by adding noise in the observation (Huang et al., 2017), action (Pinto et al., 2017) and the system dynamic s (Mandlekar et al., 2017), or by generating the goal/initial position for navigation task (Held et al., 2017; Sukhbaatar et al., 2017).

Original: However, the abstraction and sub-game solving technique cannot be applied to our task in  a straightforward way.
Revised: In this paper, we design a two-agent no-cooperative game for VAT task, where the tracker intends to lockup the target, while the target tries to escape from the tracker. Unlike the aforementioned previous work, the adversary (target to be tracked) in AD-VAT is a physical player, which could fully control the movement of the target at any time step. In this paper, such two-physical-player-competition is referred to as “dueling”.

Original: In our VAT task, the observation is screen pixels, for which we need a more efﬁcient visual feature representation, preferably learned automatically with, e.g. , a deep ConvNet.
Revised: We argue that such a fully controllable opponent could bring to the protagonist more challenges during training and henceforth produces a more robust visual tracker.

Original: There are attempts to combine CFR with deep nets (Jin et al., 2017; Lanctot et al.,  2017; Moravˇcík et al., 2017).
Revised: The proposed approach is also related to self-play (Silver et al., 2016; 2017; Bansal et al., 2017).

Original: However, they lose the theoretical guarantee in regard of ﬁnding the Nash Equilibrium, despite they show encouraging empirical results. On the other hand, there is a body of work that combines self-play with deep net to train a policy in either partial or perfect observation, e.g. , playing board game (Tesauro, 1995; Silver et al., 2016a; 2017b;a), playing video game (ope; Jaderberg et al., 2018), robot controlling (Bansal et al., 2017;
Revised: In self-play, two physical players usually compete for the same goal, with symmetric observation and action space. Usually, both players share the same model so as to ensure each agent seeing an environmental dynamics at appropriate difﬁculty level.

Original: Pinto et al., 2017), navigation in maze (Sukhbaatar et al., 2017), collision avoidance in autonomous driving (Behzadan & Munir, 2018).
Revised: It is thus viewed as a means to automatically generate learning curriculum (Bengio et al., 2009; Kumar et al., 2010).

Original: The methods just utilize the conventional value functions in the RL literature, and also show good empirical results. Despite lacking theoretical guarantee of Nash
Revised: Our setting is substantively different from standard self-play. In AD-VAT, the two players are asymmetric in observation and task.

Original: Equilibrium ﬁnding, several techniques, in particular the domain randomization and  the opponent pool sampling, are suggested to ensure the policy diversity and the training stabilization. Noting this, we stick to the conventional value function when performing the self-play training in this work.
Revised: The target observes more, and is equipped with additional auxiliary task for its adversarial policy. Thus, the target and tracker could not share the same model while learning in AD-VAT, which would make the learning become unstable. To address the issue, we propose two techniques which stabilize the training, as described in Sec. 3.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce our proposed method: Asymmetric Dueling mechanism for learning
Revised: In this section, we introduce our proposed method: Asymmetric Dueling mechanism for learning

Original: Visual Active Tracking (AD-VAT) .
Revised: Visual Active Tracking (AD-VAT) .

Original: At ﬁrst the proposed  is formulated as a multi-agent system.
Revised: At ﬁrst the proposed method is formulated as a two-player game.

Original: Then we illustrate the two key components: the reward structure and a tracker-aware model for the target.
Revised: Then we illustrate the two key components in AD-VAT: partial zero-sum reward structure and a tracker-aware model for the target.

Original: 3. F ORMULATION
Revised: 3. F ORMULATION

Revised: We adopt the Partial Observable Two-Agent Game settings (Srinivasan et al., 2018), which extends the Markov Game (Littman, 1994) to partial observation.

Original: For the notations of our two-agent settings, let superscript α denote the tracker  and superscript β denote the target.
Revised: For the notations of our two-agent game, let superscript α denote the tracker (agent 1) and superscript β denote the target (agent 2).

Revised: The game is governed by the tuple < S , O α , O β , A α , A β , r α , r β , P > , where S , O , A , r, P denote state space, observation space, action space, reward function and environment state transition probability, respectively.

Revised: Let subscript t ∈ { 1 , 2 , ... } denote the time step.

Revised: In the case of partial observation, we have the observation o α t

Revised: =

Original: The policy of the tracker, π α  ( a αt | o αt ) , is a distribution over tracker action a αt ,and conditioned on tracker observation o αt , where the subscript t denotes time step.
Revised: o α t ( s t , s t − 1 , o t − 1 ) , where o t , o t − 1 ∈ O , s t , s t − 1 ∈ S . It reduces to o αt = s t in case of full observation. The counterpart notation o βt is deﬁned likewise.

Original: When the policy takes as function approximator a Neural Network with parameter θ α , we further write it as
Revised: When the two agents take simultaneous actions a αt ∈ A α , a βt ∈ A β , the updated state s t +1 is drawn from the environment state transition probability P ( ·| s t , a αt , a βt ) . Meanwhile, the two agents receive rewards r αt =

Revised: r αt ( s t , a αt ) , r βt = r

Original: [Equation]
Revised: [Equation]

Revised: ( s t , a βt ) .

Revised: The policy of the tracker, π α ( a αt | o αt ) , is a distribution over tracker action a αt conditioned on its observation o αt .

Revised: We rely on model free independent Reinforcement

Revised: Learning to learn π α .

Revised: Speciﬁcally, the policy takes as function approximator a Neural Network with parameter θ α , written as

Original: [Equation]
Revised: [Equation]

Original: Likewise, the policy of the target can be written as
Revised: Likewise, the policy of the target can be written as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The tracker intends to maximize its expected return
Revised: The tracker intends to maximize its expected return

Original: [Equation]
Revised: [Equation]

Original: by learning the parameter θ α , where T denotes the horizon length of an episode and r αt is the immediate reward of the tracker at time step t .
Revised: by learning the parameter θ α , where T denotes the horizon length of an episode and r αt is the immediate reward of the tracker at time step t .

Original: In contrast, the target tries to maximize
Revised: In contrast, the target tries to maximize

Original: [Equation]
Revised: [Equation]

Original: by learning θ β .
Revised: by learning θ β .

Original: 3. R EWARD S TRUCTURE
Revised: 3. R EWARD S TRUCTURE

Original: The conventional adversarial methods (Bansal et al., 2017; Pinto et al., 2017) usually formulate the policy learning as a zero-sum game.
Revised: The conventional adversarial methods (Bansal et al., 2017; Pinto et al., 2017) usually formulate the policy learning as a zero-sum game.

Original: In the zero-sum game, the sum of the reward of each agent is always 0 ,  r αt + r βt = 0 .
Revised: In the zero-sum game, the sum of the reward of each agent is always 0 , e.g. , r αt + r βt = 0 .

Original: However, such kind of formulation is not suitable for AD-VAT.
Revised: However, such kind of formulation is not suitable for VAT.

Original: Considering a case that, when the two opponents are too far to observe each other, their taken actions can hardly inﬂuence the observation of their opponents directly.
Revised: Considering a case that, when the two opponents are too far to observe each other, their taken actions can hardly inﬂuence the observation of their opponents directly under the partial observable game.

Original: In this case, the latent MDP transition is usually meaningless and ineffective for improving the skill level of the agent.
Revised: In this case, the sampled experiences are usually meaningless and ineffective for improving the skill level of the agent.

Original: So constraining the competition in the observable range would make the learning more efﬁcient.
Revised: So constraining the competition in the observable range would make the learning more efﬁcient.

Original: Motivated by this, we shape a partial zero-sum reward structure, which utilize the zero-sum reward only when the target is observed by  tracker, but gives penalties to each agent when they are far.
Revised: Motivated by this, we shape a partial zero-sum reward structure, which utilizes the zero-sum reward only when the target is observed by the tracker, but gives penalties to each agent when they are far.

Original: In the following, we will introduce the details of the partial zero-sum reward structure for visual active tracking.
Revised: In the following, we will introduce the details of the partial zero-sum reward structure for visual active tracking.

Original: Reward for tracker.
Revised: Reward for tracker.

Original: The reward for tracker is similar to that in (Luo et al., 2018), composing of a positive constant and an error penalty term.
Revised: The reward for tracker is similar to that in (Luo et al., 2018), composing of a positive constant and an error penalty term.

Original: However, we do not take the orientation difference between the target and the tracker into consideration.
Revised: However, we do not take the orientation discrepancy between the target and the tracker into consideration.

Original: Moreover, we measure the relative position error based on a polar coordinate system in the 3D environment, considering the model of the camera observation.
Revised: Moreover, we measure the relative position error based on a polar coordinate system in the 3D environment, considering the model of the camera observation.

Original: With a slight abuse of notation, we can now write the reward function as
Revised: With a slight abuse of notation, we can now write the reward function as

Original: [Equation]
Revised: [Equation]

Original:  here A > 0 , ζ > 0 are tuning parameters, specially A = 1 . 0 , ζ = 2 . 0 .
Revised: ) here A > 0 , ζ > 0 are tuning parameters, specially A = 1 . 0 , ζ = 2 . 0 .

Original: P β and P exp represent the target object’s position and the expected position.
Revised: P β and P exp represent the target object’s position and the expected position.

Original: Note that P is the relative coordinates in the tracker-centric coordinate system.
Revised: Note that P is the relative coordinates in the tracker-centric coordinate system.

Original: Each term of the coordinate is normalized regarding the range of the observation R .
Revised: Each term of the coordinate is normalized regarding the range of the observation R .

Original: Speciﬁcally, P is plane position ( x, y ) in the 2D environment, but isunder a polar coordinates ( ρ, θ )  in 3D environment.
Revised: Speciﬁcally, P is plane position ( x, y ) in the 2D environment, but is ( ρ, θ ) under a polar coordinates in 3D environment.

Original: ρ is the distance between  tracker and target, θ is the relative angle to the front of the camera.
Revised: ρ is the distance between the tracker and target, θ is the relative angle to the front of the camera.

Original: Besides, the reward is clipped to be in the range of [ − A, A ] to avoid over punishment when the object is far away from the expected position.
Revised: Besides, the reward is clipped to be in the range of [ − A, A ] to avoid over punishment when the object is far away from the expected position.

Original: Reward for target.
Revised: Reward for target.

Original: The reward for the target object is closely related to  reward of the tracker, written as below:
Revised: The reward for the target object is closely related to the reward of the tracker, written as:

Original: [Equation]
Revised: [Equation]

Revised: if tracker could observe the target

Revised: [Equation]

Original: where r α is the reward of the tracker as deﬁned in Eq.
Revised: where r α is the reward of the tracker as deﬁned in Eq.

Original: 5, ξ > 0 is a tuning parameter controlling the factor of the penalty term, and P far = (1 , 1) , representing the farthest point to tracker within the observation.
Revised: (5), ξ > 0 is a tuning parameter controlling the factor of the penalty term, and P far represents the farthest point to tracker within the observation.

Original: The reward is also clipped when r t is smaller than − A .
Revised: The reward is also clipped when r t is smaller than − A .

Original: The insight behind this formulation is that, we do not encourage the target to escape from the tracker as far as possible.
Revised: The insight behind this formulation is that, we do not encourage the target to escape from the tracker as far as possible.

Original: Alternatively, by applying this reward function, the optimal policy for the target we expect should be escaping and disappearing from the observation range of the tracker but keeping close to the tracker.
Revised: Alternatively, by applying this reward function, the optimal policy for the target we expect should be escaping and disappearing from the observation range of the tracker but keeping close to the tracker.

Original: In Appendix A we visualize the reward structure given in Eq.
Revised: In Appendix A we visualize the r α + r β given in Eq.

Original: (5) and (6) for a better understanding.
Revised: (5) and (6) for a better understanding.

Original: 3. T RACKER - AWARE T ARGET
Revised: 3. T RACKER - AWARE T ARGET

Original: By tracker-awareness, the target would be “stronger” than the tracker, as it knows what the tracker knows.
Revised: By tracker-awareness, the target would be “stronger” than the tracker, as it knows what the tracker knows.

Original: This idea manifests an ancient Chinese proverb, “know the enemy, know yourself, and in every battle you will be victorious", from the masterpiece Sun Tzu on the Art of War Sun (2008).
Revised: This idea manifests an ancient Chinese proverb, “know the enemy, know yourself, and in every battle you will be victorious”, from the masterpiece Sun Tzu on the Art of War Sun (2008).

Original: The conventional adversary usually uses only its own observation Bansal et al.
Revised: The conventional adversary usually uses only its own observation Bansal et al.

Original: (2017) or shares the same observation as the opponent Pinto et al.
Revised: (2017) or shares the same observation as the protagonist Pinto et al.

Original: Recall the target policy π β ( a βt | o βt ; θ β ) written as in Eq.
Revised: Recall the target policy π β ( a βt | o βt ; θ β ) written as in Eq.

Original: However, the imperfect/partial o βt observation seriously degrades the performance of the adversary.
Revised: However, the imperfect/partial o βt observation seriously degrades the performance of the adversary.

Original: Thus, we propose a “tracker-aware" model for the target.
Revised: Thus, we propose a “tracker-aware” model for the target.

Original: Besides the target’s own observation, we additionally feed the observation and action from the tracker into the target network, in order to enrich the input information of the target.
Revised: Besides the target’s own observation, we additionally feed the observation and action from the tracker into the target network, in order to enrich the input information of the target.

Original: Moreover, we add an auxiliary task, which predicts the immediate reward of the tracker, see the TRP module in Fig. 
Revised: Moreover, we add an auxiliary task, which predicts the immediate reward of the tracker (see the TRP module in Fig. 1).

Original: This auxiliary task can be seen as a kind of “opponent modeling”, and alleviate the difﬁculty in its own policy learning.
Revised: This auxiliary task can be treated as a kind of “opponent modeling”, and alleviate the difﬁculty in its own policy learning.

Original: By doing so, we can write the output heads of such a “tracker-aware" policynetwork as π β (cid:16) a βt , ˆ r αt | o αt , a αt , o βt ; θ β (cid:17) in a slight abuse of notations, where ˆ r αt is the predicted immediate reward for the tracker and o αt , a αt are respectively the observation and the action of the tracker.
Revised: By doing so, we can write the output heads of such a “tracker-aware” policy as π β (cid:16) a βt , ˆ r αt | o αt , a αt , o βt ; θ β (cid:17) with a slight abuse of notations, where ˆ r αt is the predicted immediate reward for the tracker and o αt , a αt are respectively the observation and the action of the tracker.

Original: Empirical results show that the tracker-aware target yields more diversiﬁed escaping policy and ﬁnally helps producing a more robust tracker.
Revised: Empirical results show that the tracker-aware target yields more diversiﬁed escaping policy and ﬁnally helps producing a more robust tracker.

Original: Note that we cannot apply the trick for a tracker, as the tracker has to use its own observation during testing/deployment.  
Revised: Note that we cannot apply the trick for a tracker, as the tracker has to use its own observation during testing/deployment.  E XPERIMENTS

Revised: The following experiments explore our approach for the VAT task from 2D to 3D environments.

Original: E XPERIMENTS 4. E NVIRONMENTS
Revised: 4. E NVIRONMENTS

Original: 2D Environments.
Revised: 2D Environments.

Revised: Although 2D environments exhibit unreality to some extent, they are ideal for evaluating and quantifying the effectiveness of each method, sidestepping the uncontrolled noise in observation and action.

Original: In the 2D Environment, maps are represented by a 80 × 80 matrix, wheredenotes free space, 1 denotes an obstacle, 2 denotes the tracker, and 4 denotes the target.
Revised: In the 2D Environment, maps are represented by a 80 × 80 matrix, where 0 denotes free space, 1 denotes an obstacle, 2 denotes the tracker, and 4 denotes the target.

Original: We  generate the maps of two patterns, “random maze" and “random block" (see examples in the top row of Fig.2).
Revised: We randomly generate the maps of two patterns, “maze” and “block” (see examples in the top row of Fig.

Original: We use  mapsof block for training and maps of maze for testing.
Revised: We use the “block” maps for training and both kinds of maps for testing.

Original: For both the tracker and the target, the observation  is a matrix of size 13 × 13 around the agent.
Revised: The observation of each agent is a matrix of size 13 × 13 around the agent.

Original: The tracker aims to place the target in the center of the observation.
Revised: The tracker’s goal is placing the target as close to the center of the observed matrix as possible.

Original: During each episode, the tracker starts from a free space in the map randomly, and the target starts around the tracker in a 3 × 3 tracker-centric matrix.
Revised: During each episode, the tracker starts from a free space in the map randomly, and the target starts around the tracker in a 3 × 3 tracker-centric matrix.

Original: The 2D environments provide ideal conditions due to lack of observation or motion noise.
Revised: At each step, the agent could take an action to move toward one of four directions.

Original: The experiments in the 2D environments are dedicated to evaluate and quantify the effectiveness of our approach in ideal conditions.
Revised: The experiments in the 2D environments are dedicated to evaluate and quantify the effectiveness of our approach in ideal conditions.

Original: 3D Environments.
Revised: 3D Environments.

Original: The 3D environments are more similar to  the real-world setting in observation and action. In the 3D environments, the observation is an image of the ﬁrst-person view of the world as seen by the agent. The possible actions are move-forward, move-backward, turn-left, turn-right, turn-left-and-move-forward, turn-right-and-move-forward , and no-op .
Revised: The 3D environments show high ﬁdelity, aiming to mimic the real-world active tracking scenarios.

Original: UnrealEnv is a photo-realistic and ﬂexible simulation environment built on the Unreal Engine.
Revised: The 3D environments are built on the Unreal Engine, which and could ﬂexibly simulate a photo-realistic world.

Original: We employ UnrealCV
Revised: We employ UnrealCV

Original: (Qiu et al., 2017), which provides convenientAPIs, along with a wrapper (Zhong [Figure]
Revised: (Qiu et al., 2017), which provides convenient

Original:  et al., 2017) compatible with OpenAI Gym (Brockman et al., 2016), for interactions between RL algorithms and the UnrealEnv.
Revised: APIs, along with a wrapper (Zhong et al., 2017) compatible with OpenAI Gym (Brockman et al., 2016), for interactions between RL algorithms and the environment.

Revised: The observation is an image of the ﬁrst-person view of the world as seen by the agent.

Revised: The actions space is discrete with seven candidate motions, move-forward, move-backward, turn-left, turn-right, turn-left-and-move-forward, turn-right-and-move-forward , and no-op .

Original: For training, we built a Domain Randomized Room(DR Room) with two controllable players (target and tracker).
Revised: For training, we build a Domain Randomized Room(DR Room) with two controllable players (target and tracker).

Original: The domain randomization techniquesused could help agents learn better feature representation in terms of visual observation.
Revised: The domain randomization techniques could help agents learn better feature representation in terms of visual observation.

Original: In testing, we focus on the transferring ability of the tracker to different unseen environments.
Revised: In testing, we focus on the transferring ability of the tracker to different unseen environments.

Original: We use three realistic scenarios, Urban City, Snow Village and Parking Lot , to mimic real-world scenes for evaluating.
Revised: We use three realistic scenarios, Urban City, Snow Village and Parking Lot , to mimic real-world scenes for evaluating.

Original: The bottom row in Fig.2 shows the snapshots of the four 3D environments used.
Revised: The bottom row in Fig. 2 shows the snapshots of the four 3D environments used.

Original: Details of the four environments are: (1) DR Room is a plain room  of ﬂoor and walls only, but the textures and illumination conditions are randomized.
Revised: Details of the four environments are: • DR Room is a plain room comprised of ﬂoor and walls only, but the textures and illumination conditions are randomized.

Original: For the textures, we randomly choose pictures from a texture dataset (Kylberg, 2011) and place them on the surface of  walls, ﬂoor, andthe players.
Revised: For the textures, we randomly choose pictures from a texture dataset (Kylberg, 2011) and place them on the surface of the walls, ﬂoor, and players.

Original: For the illumination condition, we randomize the intensity and color of each light source as well as each position, orientation.
Revised: For the illumination condition, we randomize the intensity and color of each light source as well as each position, orientation.

Original: (2) Urban City is a  street view of  urban city, including well modelled buildings, streets, trees and transportation facilities with high ﬁdelity.
Revised: • Urban City is a high-ﬁdelity street view of an urban city, including well-modeled buildings, streets, trees and transportation facilities.

Original: Besides, there are some puddles on the road, reﬂecting the  buildings.
Revised: Besides, there are some puddles on the road, reﬂecting the objects and buildings.

Original: (3) Snow Village consists of bumpy snowﬁelds with several trees, bushes and some cabins.
Revised: • Snow Village consists of bumpy snowﬁelds with several trees, bushes and some cabins.

Original: Speciﬁcally, the bumpy snowﬁeld is challenging for tracking, because it frequently causes the situation that agent observation is not in the same height.
Revised: Occasionally, the target will be occluded by trees and bushes, and the tracker will be distracted by the snowﬂake and halo.

Original: (4) Parking Lot is an underground parking lot with complex illumination condition.
Revised: • Parking Lot is an underground parking lot with complex illumination condition.

Original: The lack of light source makes the illumination uneven, i.e. , some places are bright but the others are dark.
Revised: The lack of light source makes the illumination uneven, i.e. , some places are bright but the others are dark.

Revised: Besides, the pillars may occlude the target to

Revised: [Figure]

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We provide two kinds of base target agent to randomly generate trajectories as baselines to compare, Rambler (Ram) and Navigator (Nav) .
Revised: We provide two kinds of base target agent to randomly generate trajectories as baselines to compare with, Rambler (Ram) and Navigator (Nav) .

Original: Agent Ram walks randomly without any purpose like a man.
Revised: Agent Ram walks randomly without any purpose like a man.

Original: Technically, it randomly samples actions from the action space and keeps executing the action n times, where n is also a random integer in the range of (1 , 10) .
Revised: Technically, it randomly samples actions from the action space and keeps executing the action n times, where n is also a random integer in the range of (1 , 10) .

Original: Agent Nav is a navigator, planning the shortest path to a speciﬁc goal in use of the Astar algorithm (Hart et al., 1968).
Revised: Agent Nav is a navigator, which plans the shortest path to a speciﬁc goal.

Original: Thus, it could navigate to most of free space in the map.
Revised: Thus, it could navigate to most of free space in the map.

Original: To randomize the trajectories, the goal coordinate and the initial coordinate are randomly sampled from the free space.
Revised: To randomize the trajectories, the goal coordinate and the initial coordinate are randomly sampled.

Original: In most of case, Ram prefers to walk around a local area repeatedly.
Revised: In most of case, Ram prefers to walk around a local area repeatedly.

Original: In contrast, Nav would like to explore the map globally, shown as the yellow trajectories in Fig.
Revised: In contrast, Nav would like to explore the map globally, shown as the yellow trajectories in Fig.

Original: Thus we regard trajectories from the Ram as easier cases, and trajectories from Nav as more difﬁcult cases for tracker.
Revised: Thus we regard trajectories from the Ram as easier cases, and trajectories from Nav as more difﬁcult cases for tracker.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Each agent is trained by A3C (Mnih et al., 2016), a commonly used reinforcement learning algorithm.
Revised: Each agent is trained by A3C (Mnih et al., 2016), a commonly used reinforcement learning algorithm.

Revised: The code for A3C is based on a pytorch implementation (Grifﬁs).

Original: Multiple workers are running in parallel when training.
Revised: Multiple workers are running in parallel when training.

Original: Speciﬁcally, 16 workers are used in the 2D experiment, and 4 workers are used in the 3D experiment.
Revised: Speciﬁcally, 16 workers are used in the 2D experiment, and 4 workers are used in the 3D experiment.

Original: Network Architecture.
Revised: Network Architecture.

Original: For the tracker, we follow the end-to-end Conv-LSTM network architecture as (Luo et al., 2018).
Revised: For the tracker, we follow the end-to-end Conv-LSTM network architecture as (Luo et al., 2018).

Revised: Differently, there is no fully-connected layer between the Conv-Net and the

Original: Differently, the Conv-Net in this paper is deeper, and  the input color images arepre-processed, transformed to gray image and the pixel values are scaled to [ − 1 , 1] .
Revised: LSTM-Net in this paper. The Conv-Net is a two-layer CNN for the 2D experiments and four-layer CNN for the 3D experiments. In the 3D experiments, the input color images are transformed to gray image and the pixel values are scaled to [ − 1 , 1] .

Original: we also develop the  Conv-LSTM network architecture for the target, but different in the input and output, shown as Fig.1.
Revised: we also develop the same Conv-LSTM network architecture for the target, but different in the input and output, shown as Fig.

Original: The network parameters are updated with a shared Adam optimizer.
Revised: The network parameters are updated with a shared Adam optimizer.

Original: Hyper Parameters.
Revised: Hyper Parameters.

Original: For the tracker, the learning rates δ α 0 and δ α 1(cid:48) in 2D and 3D environments are 0.001 and 0.0001, respectively.
Revised: For the tracker, the learning rates δ α 0 and δ α 1 in 2D and 3D environments are 0.001 and 0.0001, respectively.

Original: The reward discount factor γ = 0 .
Revised: The reward discount factor γ = 0 .

Original: 9 , generalized advantage estimate parameter τ = 1 . 00 , and regularizer factor for tracker λ α = 0 .
Revised: 9 , generalized advantage estimate parameter τ = 1 . 00 , and regularizer factor for tracker λ α = 0 .

Original: The parameter updating frequency n is 20, and the maximum global iteration for training is 150 K. Comparing to the tracker, a higher regularizer factor is used for encouraging the target to explore, λ β 0 = 0 .
Revised: The parameter updating frequency n is 20, and the maximum global iteration for training is 150 K. Comparing to the tracker, a higher regularizer factor is used for encouraging the target to explore, λ β 0 = 0 .

Original: 2 in 2D and λ β 1 = 0 .
Revised: 2 in 2D and λ β 1 = 0 .

Original: in 3D.
Revised: in 3D.

Original: The more exploration taken by target, the more diverse the generated trajectories are.
Revised: The more exploration taken by target, the more diverse the generated trajectories are.

Original: It is useful for the learning of the tracker.
Revised: It is useful for the learning of the tracker.

Original: Validation is performed in parallel and the best validation network model is applied to report performance in testing environments.
Revised: Validation is performed in parallel and the best validation network model is applied to report performance in testing environments.

Original: Note that the validation environment is of the same settings as training, except that the target is controlled by a Nav agent.
Revised: Note that the validation environment is of the same settings as training, except that the target is controlled by a Nav agent.

Original: Compared with the
Revised: Compared with the

Original: Ram agent, the Nav agent is more challenging, thus is more suitable for validation.
Revised: Ram agent, the Nav agent is more challenging, thus is more suitable for validation.

Original: Metric.
Revised: Metric.

Original: Two metrics are employed for the experiments.
Revised: Two metrics are employed for the experiments.

Original: Speciﬁcally, Accumulated Reward (AR) and
Revised: Speciﬁcally, Accumulated Reward (AR) and

Original: Episode Length (EL) of each episode are calculated for quantitative evaluation.
Revised: Episode Length (EL) of each episode are calculated for quantitative evaluation.

Original: AR is a comprehensive metric, representing the tracker’s capability about precision and robustness.
Revised: AR is a comprehensive metric, representing the tracker’s capability about precision and robustness.

Original: It is effected by the immediate reward and the episode length.
Revised: It is effected by the immediate reward and the episode length.

Original: Immediate reward measures the goodness of tracking, and
Revised: Immediate reward measures the goodness of tracking, and

Original: EL roughly measures the duration of good tracking.
Revised: EL roughly measures the duration of good tracking.

Original: Because the episode is done when the tracker lost for continuous 10 steps or reach the max episode length. 
Revised: Because the episode is terminated when the tracker loses the target for continuous 10 steps or reaches the max episode length. [Table]

Original: 4. R ESULTS ON THE 2D ENVIRONMENT
Revised: 4. R ESULTS ON THE 2D ENVIRONMENT

Original: We quantitatively evaluate the performance of our approach, comparing to the two baselines.
Revised: We quantitatively evaluate the performance of our approach, comparing to the two baselines.

Original: Furthermore, we conduct an ablation study to show the effect of the partial zero-sum reward and tracker-aware model.
Revised: Furthermore, we conduct an ablation study to show the effectiveness of the partial zero-sum reward and tracker-aware model.

Original: Quantitative Evaluation.
Revised: Quantitative Evaluation.

Original: We test the active tracker trained with different target agents in four testing settings, showing the effectiveness of AD-VAT .
Revised: We test the active tracker trained with different target agents in four testing settings, showing the effectiveness of AD-VAT .

Original: Considering the random seed of the environments, we conduct 100 runs in each and report the mean and standard deviation of AR and EL, shown in
Revised: Considering the random seed of the environments, we conduct 100 runs in each and report the mean and standard deviation of AR and EL, shown in

Original: Table 1.
Revised: Table 1.

Original: The max episode length is 500, so the upper bound of EL is 500.
Revised: The max episode length is 500, so the upper bound of EL is 500.

Original: Thus, when EL equals to 500, we could infer that the tracker performs perfectly, without losing the target.
Revised: Thus, when EL equals to 500, we could infer that the tracker performs perfectly, without losing the target.

Original: We note that, at the beginning of the learning, the adversarial target usually walks randomly around the start point, performing similar policy as Ram .
Revised: We note that, at the beginning of the learning, the adversarial target usually walks randomly around the start point, performing similar policy as Ram .

Original: Such target is easier to be found and observed, even though the tracker is in exploration.
Revised: Such target is easier to be found and observed, even though the tracker is in exploration.

Original: Thus, the tracker could warm up faster.
Revised: Thus, the tracker could warm up faster.

Original: With the growth of the tracker, the target gradually explores other motion patterns, which could further reinforce the tracker.
Revised: With the growth of the tracker, the target gradually explores other motion patterns, which could further reinforce the tracker.

Original: Such a learning process is close to the curriculum learning, but the curriculum is automatically produced by the target via adversarial reinforcement learning.
Revised: Such a learning process is close to the curriculum learning, but the curriculum is automatically produced by the target via adversarial reinforcement learning.

Original: We also report the learning curve as the mean of cumulative rewards in the validation environment, shown as the left sub-ﬁgure in Fig 3.
Revised: We also report the learning curve as the mean of cumulative rewards in the validation environment, shown as the left sub-ﬁgure in Fig.

Original: It consistently shows the advantage of the proposed AD-VAT .
Revised: It consistently shows the advantage of the proposed AD-VAT .

Original: Ablation Study.
Revised: Ablation Study.

Original: In Section 3, we introduced two components to implement AD-VAT : “partial zerosum" rewardstructure (PZR) and tracker-aware model (TAM) for target.
Revised: In Section 3, we introduced two components to implement AD-VAT : partial zerosum reward (PZR) and tracker-aware model (TAM) for target.

Original: These two components are important as they inﬂuence the natural curriculum for the tracker.
Revised: These two components are important as they inﬂuence the natural curriculum for the tracker.

Original: Thus, we report an ablation study result to show the effectiveness of these two components, shown in Fig.
Revised: Thus, we report an ablation study result to show the effectiveness of these two components, shown in Fig.

Original: The naive method is an intuitive idea that target only uses its own observation with auxiliary task, guided by a zero-sum reward.
Revised: The naive method is an intuitive idea that target only uses its own observation with auxiliary task, guided by a zero-sum reward.

Original: As shown in the right sub-ﬁgure of Fig.
Revised: As shown in the right sub-ﬁgure of Fig.

Original: 3, using each component separately could improve the sample-efﬁciency, comparing to the naive method.
Revised: 3, using each component separately could improve the sample-efﬁciency, comparing to the naive method.

Original: Besides, PZR contributes to the improvement Environment [Table] Map
Revised: Besides, PZR contributes to the improvement of the tracking performance more signiﬁcant than TAM.

Original: Target

Original: [Table]

Original: Maze

Original: [Table]

Original: [Table]

Original: Block

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Average

Original: [Table]

Original: [Table] [Figure] of the tracking performance more signiﬁcant than TAM. Moreover, when combining PZR and TAM, both sample-efﬁciency and the tracking performance are signiﬁcantly boosted, comparing to the other three settings.
Revised: Moreover, when combining PZR and TAM, both sample-efﬁciency and the tracking performance are signiﬁcantly boosted, comparing to the other three settings.

Original: 4. R ESULTS ON THE 3D ENVIRONMENT
Revised: 4. R ESULTS ON THE 3D ENVIRONMENT

Original: In the 3D UnrealEnv, we test the effectiveness of the model in unseen realistic scenarios, showing the transfer potential in real-world scenario.
Revised: In the 3D experiments, we test the generalization of the model in unseen realistic scenarios, showing the transfer potential in real-world scenarios.

Original: We trainthe three models with different target models ( Ram,
Revised: We train three models with different target models ( Ram,

Original: Nav , and AD-VAT ) in the DR Room.
Revised: Nav , and AD-VAT ) in the DR Room.

Original: And then, we directly run the three trackers in the validation and testing environments, 100 episodes for each, and report AR and EL in Fig.
Revised: And then, we directly run the three trackers in the validation and testing environments, 100 episodes for each, and report AR and EL in Fig.

Original: The result in DR Room demonstrates again that even though the targetmodel is unseen for our AD-VAT tracker, it still outperforms the others.
Revised: The result in DR Room demonstrates again that even though the target is unseen for our AD-VAT tracker, it still outperforms the others.

Original: Because the DR Room for validation is of the same settings as the training of Nav tracker.
Revised: Note that the DR Room for validation is of the same settings as the training of Nav tracker.

Original: The results in UrbanCity show that the three models are able to transfer to realistic environment.
Revised: The results in the other environments show that the three models are able to transfer to realistic environment.

Original: We believe that the domain randomization method and the Conv-LSTM network endow the trackers the ability of transferring.
Revised: We believe that the domain randomization method and the Conv-LSTM network endow the trackers the ability of transferring.

Original: However, the  performance  of the two baselines degrades more compared with our method, in other environments. So we suspect that the two baseline methods are not sufﬁciently effective when transferring to challenging environments. Training with AD-VAT could signiﬁcantly improve the capability of the tracker in  challenging environments.
Revised: However, the tracker’s performance is also highly related to the behavior of the targets during training, especially in complex environments ( Snow Village and Parking Lot ). Compared with the two baselines, the adversarial behavior of the target could signiﬁcantly improve the capability of the tracker in these challenging environments.

Original: We infer that the target in AD-VAT could explore the environment more actively to discover more challenging cases.
Revised: We infer that the target in AD-VAT could explore the environment more acvtively to discover more difﬁcult cases.

Original: For example, in DR Room, the target would prefer to move close to the wall that is similar to itself to fool the tracker (see Fig. 5).
Revised: For example, in DR Room, the target would prefer to move close to the wall that is similar to itself to fool the tracker (see Fig. 5).

Original: By competing with the target, the tracker consequently becomes stronger.
Revised: By competing with the target, the tracker consequently becomes stronger.

Revised: In Appendix.

Revised: C, we further evaluate the capability of our tracker in the real-world video clips, qualitatively.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have proposed an asymmetric dueling mechanism for visual active tracking (ADVAT).
Revised: In this paper, we have proposed an asymmetric dueling mechanism for visual active tracking (ADVAT).

Original: Within AD-VAT, agents of tracker and target are learned in an adversarial manner.
Revised: Within AD-VAT, agents of tracker and target are learned in an adversarial manner.

Original: With the design of the “partial zero-sum" reward structure andthe tracker-awaretarget model, the reinforced active tracker outperforms baseline methods.
Revised: With the design of the partial zero-sum reward structure and tracker-aware model, the reinforced active tracker outperforms baseline methods.

Original: Experiments including ablation study in both 2D and 3D environments verify the effectiveness of the proposed mechanism.  
Revised: Experiments including ablation study in both 2D and 3D environments verify the effectiveness of the proposed mechanism.  A CKNOWLEDGMENTS

Revised: The authors would like to thank Weichao Qiu for his help in extending UnrealCV.

Revised: Fangwei Zhong,

Revised: Tingyun Yan and Yizhou Wang were supported in part by the following grants: NSFC-61625201,

Revised: NSFC-61527804, Tencent AI Lab Rhino-Bird Focused Research Program

Original:  A V ISUALIZING P ARTIAL Z ERO - SUM R EWARD
Revised: No.JR201851, Qualcomm University Collaborative Research Program. A V ISUALIZING P ARTIAL Z ERO - SUM R EWARD

Original: To help better understand the reward structure given in Eq.
Revised: To help better understand the reward structure given in Eq.

Original: (5) and (6), we visualize the sum r α + r β as heatmap in x − y plane.
Revised: (5) and (6), we visualize the sum r α + r β as heatmap in x − y plane.

Original: See Fig. 6.
Revised: See Fig. 6.

Original: For the 2D experiment, the observations for both the tracker and target are bird-views.
Revised: For the 2D experiment, the observations for both the tracker and target are bird-views.

Original: We want to penalize that the target gets too far away from the tracker.
Revised: We want to penalize that the target gets too far away from the tracker.

Original: Therefore, the zero-sum area is a circle (Fig. 6, Left), where the tracker is in the centre.
Revised: Therefore, the zero-sum area is a circle (Fig. 6, Left), where the tracker is in the centre.

Original: With the increasing of the distance, the penalty term in r β (see Eq. 6) starts taking effect on the sum.
Revised: With the increasing of the distance, the penalty term in r β (see Eq. (6)) starts taking effect on the sum.

Original: It causes the sum to reduce gradually until the target reaches the dark area, where r β = − A .
Revised: It causes the sum to reduce gradually until the target reaches the dark area, where r β = − A .

Original: For the 3D experiment, the observations for both the tracker and target are front-views.
Revised: For the 3D experiment, the observations for both the tracker and target are front-views.

Original: We want to penalize that the target gets too far away from the tracker or that the target cannot be seen by the tracker.
Revised: We want to penalize that the target gets too far away from the tracker or that the target cannot be seen by the tracker.

Original: Thus, the zero-sum area is a sector area (Fig. 6, Right), which approximately ﬁts the Field of View (FoV) of the tracker’s camera.
Revised: Thus, the zero-sum area is a sector area (Fig. 6, Right), which approximately ﬁts the Field of View (FoV) of the tracker’s camera.

Original: Note that the FoV in our experiment is 90 degree.
Revised: Note that the FoV in our experiment is 90 degree.

Original: Both the relative angle θ and distance ρ contribute to the penalty term in r β .
Revised: Both the relative angle θ and distance ρ contribute to the penalty term in r β .

Original: Thus the sum decreases like a divergence sector.
Revised: Thus the sum decreases like a divergence sector.

Original: B V ISUALIZING THE T RAINING P ROCESS
Revised: B V ISUALIZING THE T RAINING P ROCESS

Original: For a better understanding of the training process, we record trajectories of the target and the tracker during different training stages.
Revised: For a better understanding of the training process, we record trajectories of the target and the tracker during different training stages.

Original: Speciﬁcally, we have 6 stages, ranging from early training to late training.
Revised: Speciﬁcally, we have 6 stages, ranging from early training to late training.

Original: For each stage we record 100 episodes.
Revised: For each stage we record 100 episodes.

Original: In Fig.
Revised: In Fig.

Original: 7, we plot the target position distribution, instead of the trajectory itself.
Revised: 7, we plot the target position distribution, instead of the trajectory itself.

Original: For ease of visualization, we adopt a relative coordinate system for the target position when drawing the distribution, because the start locations for both the tracker and the target are random upon each episode.
Revised: For ease of visualization, we adopt a relative coordinate system for the target position when drawing the distribution, because the start locations for both the tracker and the target are random upon each episode.

Original: In Fig.
Revised: In Fig.

Original: 7, the distributions are in a start point -centric coordinate system, while in Fig.
Revised: 7, the distributions are in a start point -centric coordinate system, while in Fig.

Original: 8 the distributions are in a tracker -centric coordinate system.
Revised: 8 the distributions are in a tracker -centric coordinate system.

Original: At early training stages (See the left of Fig. 7), AD-VAT and Ram generate similar trajectories, which are randomly walking around the start point.
Revised: At early training stages (See the left of Fig. 7), AD-VAT and Ram generate similar trajectories, which are randomly walking around the start point.

Original: In contrast, for Nav method the target usually goes along a straight line to the goal position, causing the tracker to get lost quickly at beginning.
Revised: In contrast, for Nav method the target usually goes along a straight line to the goal position, causing the tracker to get lost quickly at beginning.

Original: The randomly walking trajectories help the tracker observe the target appearing in various positions, and henceforth sample more diverse experiences.
Revised: The randomly walking trajectories help the tracker observe the target appearing in various positions, and henceforth sample more diverse experiences.

Original: As a result, a better exploration is achieved during the early training stage, which is not the case for the Nav method.
Revised: As a result, a better exploration is achieved during the early training stage, which is not the case for the Nav method.

Original: With the evolution of the tracker during training, the target will gradually seek more difﬁcult cases to defeat the tracker.
Revised: With the evolution of the tracker during training, the target will gradually seek more difﬁcult cases to defeat the tracker.

Original: In this regard, the target for both Ram and Nav usually explore possible directions uniformly.
Revised: In this regard, the target for both Ram and Nav usually explore possible directions uniformly.

Original: The minor difference is that the Nav tends to explore the map globally, while the Ram is in local (See the right of Fig. 7).
Revised: The minor difference is that the Nav tends to explore the map globally, while the Ram is in local (See the right of Fig. 7).

Original: As for our AD-VAT method, however, the reinforced target could adapt to the capability of the tracker, resulting to different direction choosing patters at different stage.
Revised: As for our AD-VAT method, however, the reinforced target could adapt to the capability of the tracker, resulting to different direction choosing patters at different stage.

Original: For example, the target tends to move bottom-right at the third stage, but top-left at the fourth stage.
Revised: For example, the target tends to move bottom-right at the third stage, but top-left at the fourth stage.

Original: See
Revised: See

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Fig.
Revised: Fig.

Original: Besides, it seems that the reinforced target could balance the two exploration modes of Nav and Ram naturally.
Revised: Besides, it seems that the reinforced target could balance the two exploration modes of Nav and Ram naturally.

Original: Sometimes it explores the map, and sometimes it duels with the tracker locally.
Revised: Sometimes it explores the map, and sometimes it duels with the tracker locally.

Original: By the dueling mechanism, the target could ﬁnd the weakness of the tracker more often (See the right of Fig.
Revised: By the dueling mechanism, the target could ﬁnd the weakness of the tracker more often (See the right of Fig.

Original: 8), which seems to serve as a kind of importance sampling that enhances the tracker efﬁciently during training.
Revised: 8), which seems to serve as a kind of importance sampling that enhances the tracker efﬁciently during training.

Original: Such a "weakness-ﬁnding" seems absent for the Nav and Ram algorithms.
Revised: Such a “weakness-ﬁnding” seems absent for the Nav and Ram algorithms.

Original: CP ASSIVELY T EST T RACKER ON R EAL -W ORLD V IDEO C LIPS
Revised: C T ESTING T RACKER ON R EAL -W ORLD V IDEO C LIPS

Original: To demonstrate the capability of our tracker in real-world scenarios, we conduct a qualitative evaluation as Luo et al.
Revised: To demonstrate the capability of our tracker in real-world scenarios, we conduct a qualitative evaluation as Luo et al.

Original: In this evaluation, we feed the video clips from VOT dataset (Kristan et al., 2016) to the tracker and observe the network output actions.
Revised: In this evaluation, we feed the video clips from VOT dataset (Kristan et al., 2016) to the tracker and observe the network output actions.

Original: Note that the tracker could not really control the camera movement, hence we regard the test as “passively".
Revised: Note that the tracker could not really control the camera movement, hence we regard the test as “passively”.

Original: However, the tracker’s output action is expected to be sensitive to the position and the scale of the target in the image.
Revised: However, the tracker’s output action is expected to be sensitive to the position and the scale of the target in the image.

Original: For example, when the target appears in the right (left) side, the tracker tends to turn right (left), trying to ﬁctitiously move the camera to “place” the target in the image center.
Revised: For example, when the target appears in the right (left) side, the tracker tends to turn right (left), trying to ﬁctitiously move the camera to “place” the target in the image center.

Original: By visualizing whether the output action is consistent with the position and scale of the target at each frame, we are able to demonstrate the potential of transferring the tracking ability to real-world.
Revised: By visualizing whether the output action is consistent with the position and scale of the target at each frame, we are able to demonstrate the potential of transferring the tracking ability to real-world.

Original: We plot three “Action" maps, shown in Fig.
Revised: We plot three “Action” maps, shown in Fig.

Original: 9, Fig. 10, and Fig.
Revised: 9, Fig. 10, and Fig.

Original: 11, respectively.
Revised: 11, respectively.

Original: Note that the meaning of each axis is the same as (Luo et al., 2018) except that we normalize the values for better understanding.
Revised: Note that the meaning of each axis is the same as (Luo et al., 2018) except that we normalize the values for better understanding.

Original: In details, the horizontal axis indicates the normalized x-axis position of the target in the image, with a positive (negative) value meaning that a target is in the right (left) side.
Revised: In details, the horizontal axis indicates the normalized x-axis position of the target in the image, with a positive (negative) value meaning that a target is in the right (left) side.

Original: The vertical axis indicates the normalized size of the target, i.e. , the area of the ground truth bounding box.
Revised: The vertical axis indicates the normalized size of the target, i.e. , the area of the ground truth bounding box.

Original: We use seven marks to represent the seven actions respectively, as shown in the legend.
Revised: We use seven marks to represent the seven actions respectively, as shown in the legend.

Original: More results on other VOT videos are available at: https://youtu.be/jv-5HVg_Sf4 .
Revised: More results on other VOT videos are available at: https://youtu.be/jv-5HVg_Sf4 .