Original: AD-VAT: A N ASYMMETRIC DUELING MECHANISM FOR LEARNING VISUAL ACTIVE TRACKING
Revised: AD-VAT: A N ASYMMETRIC DUELING MECHANISM FOR LEARNING VISUAL ACTIVE TRACKING

Original: A BSTRACT
Revised: A BSTRACT

Original: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.
Revised: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.

Original: Previous work has shown that the tracker can be trained in a simulator via reinforcement learning and deployed in real-world scenarios.
Revised: Previous work has shown that the tracker can be trained in a simulator via reinforcement learning and deployed in real-world scenarios.

Original: However, during training, such method requires manually specifying the moving path of the target object to be tracked, which cannot ensure the tracker’s generalization on unseen object moving patterns
Revised: However, during training, such method requires manually specifying the moving path of the target object to be tracked, which cannot ensure the tracker’s generalization on unseen object moving patterns.

Original: To learn a robust tracker for VAT, in this paper, we propose a novel adversarial RL method which adopts an Asymmetric Dueling mechanism , referred to as AD -VAT
Revised: To learn a robust tracker for VAT, in this paper, we propose a novel adversarial RL method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT.

Original: In AD-VAT
Revised: In AD

Original: , both the tracker and the target are approximated by end-to-end neural networks  and are trained via RL in a dueling/competitive manner: i e. , the tracker intends to lockup the target, while the target tries to escape from the tracker.
Revised: VAT, both the tracker and the target are approximated by end to end neural networks , and are trained via RL in a dueling competitive manner: i.e . , the tracker intends to lockup the target, while the target tries to escape from the tracker

Original: They are asymmetric in that the target is aware of the tracker but not vice versa.
Revised: They are asymmetric in that the target is aware of the tracker but not vice versa.

Original: Speciﬁcally, besides its own observation, the target is fed with the tracker’s observation and action and learns to predict the tracker’s reward as an auxiliary task.
Revised: Speciﬁ cally besides its own observation, the target is fed with the tracker’s observation and action and learns to predict the tracker’s reward as an auxiliary task.

Original: We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker.
Revised: We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker.

Original: To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target.
Revised: To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target.

Original: The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios.
Revised: The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios.

Original: For supplementary videos, see: https://www.youtube.
Revised: For supplementary videos, see: https://www.youtube.

Original: com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS
Revised: com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.
Revised: Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations.

Original: VAT is demanded in many real-world applications such as autonomous vehicle ﬂeet ( e.g. , a slave-vehicle should follow a master-vehicle ahead), service robots and drones ( e.g. , a drone is required to follow a person when recording a video).
Revised: VAT is demanded in many real-world applications such as autonomous vehicle ﬂeet ( e.g. , a slave-vehicle should follow a master-vehicle ahead), service robots and drones ( e.g. , a drone is required to follow a person when recording a video).

Original: To accomplish the VAT task, one typically needs to perform a sequence of tasks such as recognition, localization, motion prediction, and camera control.
Revised: To accomplish the VAT task, one typically needs to perform a sequence of tasks such as recognition, localization, motion prediction, and camera control.

Original: However, conventional visual tracking (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009; Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012) aims to solely propose a 2D bounding box of the target frame by frame, and does not actively take into consideration the control of camera.
Revised: However, conventional visual tracking (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009; Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012) aims to solely propose a 2D bounding box of the target frame by frame, and does not actively take into consideration the control of camera.

Original: Thus, compared to the problem of “passive” tracking,
Revised: Thus, compared to the problem of “passive” tracking,

Original: VAT is more practical and challenging.
Revised: VAT is more practical and challenging.

Original: With the advancement of deep reinforcement learning (Sutton & Barto, 1998;
Revised: With the advancement of deep reinforcement learning (Sutton & Barto, 1998;

Original: Mnih et al., 2015;
Revised: Mnih et al., 2015;

Original: Silver et al., 2016; Mnih et al., 2016), training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible (Luo et al., 2018;
Revised: Silver et al., 2016; Mnih et al., 2016), training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible (Luo et al., 2018;

Original: Luo et al., 2019).
Revised: Luo et al., 2019).

Original: The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network.
Revised: The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network.

Original: Such an end-to-end approach could save the effort of tuning an extra camera controller.
Revised: Such an end-to-end approach could save the effort of tuning an extra camera controller.

Original: Meanwhile, it also 
Revised: Meanwhile, it also outperforms the conventional methods where the passive tracker is equipped with a hand-engineered camera controller.

Original: [Figure]
Revised: [Figure]

Original: outperforms the conventional methods where the passive tracker is equipped with a hand-engineered camera controller.

Original: However, the performance of the deep reinforcement learning based tracker is still limited by the training methods.
Revised: However, the performance of the deep reinforcement learning based tracker is still limited by the training methods.

Original: Due to the “trial-and-error” nature of reinforcement learning, it is infeasible to directly train the tracker in the real world.
Revised: Due to the “trial-and-error” nature of reinforcement learning, it is infeasible to directly train the tracker in the real world.

Original: Alternatively, virtual environments are always utilized to generate sufﬁcient data for training without tedious human labeling.
Revised: Alternatively, virtual environments are always utilized to generate sufﬁcient data for training without tedious human labeling.

Original: Nevertheless, to deploy the trained tracker in the real world, one has to overcome the virtual-to-real gap.
Revised: Nevertheless, to deploy the trained tracker in the real world, one has to overcome the virtual-to-real gap.

Original: One solution can be building numbers of high-ﬁdelity environments (Zhu et al., 2017).
Revised: One solution can be building numbers of high-ﬁdelity environments (Zhu et al., 2017).

Original: However, it is expensive and tedious to build such environments for VAT.
Revised: However, it is expensive and tedious to build such environments for VAT.

Original: Both the visual rendering (illumination, texture, etc. ) and the physical properties should be carefully designed to emulate the real world.
Revised: Both the visual rendering (illumination, texture, etc. ) and the physical properties should be carefully designed to emulate the real world.

Original: Suppose we carry out VAT where the target is a pedestrian.
Revised: Suppose we carry out VAT where the target is a pedestrian.

Original: To build the environment, one has to not only model the human’s appearance, but also design physical rules and the pedestrian’s trajectory so that it moves naturally like a human beings.
Revised: To build the environment, one has to not only model the human’s appearance, but also design physical rules and the pedestrian’s trajectory so that it moves naturally like a human beings.

Original: Recently, (Luo et al., 2018) tried to overcome the virtual-to-real gap by applying the so-called environment augmentation technique.
Revised: Recently, (Luo et al., 2018) tried to overcome the virtual-to-real gap by applying the so-called environment augmentation technique.

Original: They diversify the visual appearance by changing the placement of the background objects and by ﬂipping left-right the screen frame.
Revised: They diversify the visual appearance by changing the placement of the background objects and by ﬂipping left-right the screen frame.

Original: However, they neglect another important factor, that is, the motion of the target for VAT task.
Revised: However, they neglect another important factor, that is, the motion of the target for VAT task.

Original: Intuitively, the complexity and diversity of the target motion in training will impact the generalization of the data-driven tracker.
Revised: Intuitively, the complexity and diversity of the target motion in training will impact the generalization of the data-driven tracker.

Original: For example, if the target only moves forward during training, the tracker may over ﬁt to move straightly and fail to track other motion patterns, like a sharp turn.
Revised: For example, if the target only moves forward during training, the tracker may over ﬁt to move straightly and fail to track other motion patterns, like a sharp turn.

Original: In this work, we propose a novel adversarial RL method for learning VAT, refereed to as AD-VAT ( Asymmetric Dueling mechanism for learning Visual Active Tracking ).
Revised: In this work, we propose a novel adversarial RL method for learning VAT, refereed to as AD-VAT ( Asymmetric Dueling mechanism for learning Visual Active Tracking ).

Original: In the mechanism, the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during competition (See Fig. 1).
Revised: In the mechanism, the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during competition (See Fig. 1).

Original: As the training proceeds, the environments of AD-VAT naturally compose a curriculum, because the tracker is more likely to compete with a target with the appropriate difﬁculty level when both agents are becoming stronger simultaneously.
Revised: As the training proceeds, the environments of AD-VAT naturally compose a curriculum, because the tracker is more likely to compete with a target with the appropriate difﬁculty level when both agents are becoming stronger simultaneously.

Original: When exploring the escape policy, the target consequently generates various trajectories to train the tracker.
Revised: When exploring the escape policy, the target consequently generates various trajectories to train the tracker.

Original: By the dueling/adversarial mechanism, the target is encouraged to discover the weakness of the tracker more often, which could serve as a kind of “weakness-ﬁnding” that makes the tracker more robust.
Revised: By the dueling/adversarial mechanism, the target is encouraged to discover the weakness of the tracker more often, which could serve as a kind of “weakness-ﬁnding” that makes the tracker more robust.

Original: However, in practice, using heuristic adversarial RL method for training VAT is unstable and slow to converge.
Revised: However, in practice, using heuristic adversarial RL method for training VAT is unstable and slow to converge.

Original: To address these issues, we derive two components in AD-VAT: partial zero-sum reward(PZR) and tracker-aware model(TAM) for target.
Revised: To address these issues, we derive two components in AD-VAT: partial zero-sum reward(PZR) and tracker-aware model(TAM) for target.

Original: PZR is a hybrid reward structure.
Revised: PZR is a hybrid reward structure.

Original: It encourages a zero-sum tracker-target competition in the near range, where the target is close to the expected position to tracker; whereas, beyond the zero-sum zone, it is a non-zero-sum game, in which the target is penalized for running too far to track.
Revised: It encourages a zero-sum tracker-target competition in the near range, where the target is close to the expected position to tracker; whereas, beyond the zero-sum zone, it is a non-zero-sum game, in which the target is penalized for running too far to track.

Original: Such reward structure is inspired by an observed phenomenon, that when the target quickly learns to be far away from the tracker while the tracker has no chance to see the target once more and henceforth gets plateaus during training.
Revised: Such reward structure is inspired by an observed phenomenon, that when the target quickly learns to be far away from the tracker while the tracker has no chance to see the target once more and henceforth gets plateaus during training.

Original: To learn the optimal policy to escape, we model the target with a “tracker-aware network”, i.e. , besides its own observation, the observation and actions of the tracker are also fed to the escaping network.
Revised: To learn the optimal policy to escape, we model the target with a “tracker-aware network”, i.e. , besides its own observation, the observation and actions of the tracker are also fed to the escaping network.

Original: In addition, to shape a better representation about the tracker, we add an auxiliary task for the target, learning to predict the immediate reward of the tracker.
Revised: In addition, to shape a better representation about the tracker, we add an auxiliary task for the target, learning to predict the immediate reward of the tracker.

Original: We argue that such an “asymmetric dueling” mechanism is able to learn a stronger target, which vice versa yields a more robust tracker ultimately.
Revised: We argue that such an “asymmetric dueling” mechanism is able to learn a stronger target, which vice versa yields a more robust tracker ultimately.

Original: The experiment is conducted in various 2D and 3D environments for further studying AD-VAT.
Revised: The experiment is conducted in various 2D and 3D environments for further studying AD-VAT.

Original: The 2D environment is a matrix map where obstacles are randomly placed.
Revised: The 2D environment is a matrix map where obstacles are randomly placed

Original: In the 2D environments, we evaluate and quantify the effectiveness of our approach in an ideal condition, free from noise in observation and action.
Revised: In the 2D environments , we evaluate and quantify the effectiveness of our approach in an ideal condition free from noise in observation and action

Original: We also conduct an ablation study to show the effectiveness of the two important components, “partial zero-sum reward” and “tracker-aware network”.
Revised: We also conduct an ablation study to show the effectiveness of the two important components “partial zero-sum reward” and “tracker-aware network”.

Original: The 3D environments are built on Unreal Engine4, a popular game engine for building high-ﬁdelity environments.
Revised: The 3D environments are built on Unreal Engine a popular game engine for building high-ﬁdelity environments.

Original: We choose a large room for training, where the texture of the background/players and the illumination are randomized.
Revised: We choose a large room for training where the texture of the background/players and the illumination are randomized.

Original: Three realistic scenarios built by artists are used for further evaluating the robustness.
Revised: Three realistic scenarios built by artists are used for further evaluating the robustness.

Original: In the 3D environments, we further demonstrate that the tracker trained in AD-VAT is capable of generalizing to high-ﬁdelity environments even it is trained in a simple environment.
Revised: In the 3D environments, we further demonstrate that the tracker trained in AD-VAT is capable of generalizing to high-ﬁdelity environments even it is trained in a simple environment.

Original: The contributions of our work can be summarized as follows:
Revised: The contributions of our work can be summarized as follows:

Original: • We propose a novel Adversarial Reinforcement Learning method for VAT task, i.e. , the Asymmetric Dueling mechanism (AD-VAT).
Revised: • We propose a novel Adversarial Reinforcement Learning method for VAT task, i.e. , the Asymmetric Dueling mechanism (AD-VAT).

Original: In AD-VAT, the target learns to generate diverse trajectories when competing with the tracker, which in turn helps train a more robust tracker.
Revised: In AD-VAT, the target learns to generate diverse trajectories when competing with the tracker, which in turn helps train a more robust tracker.

Original: • We provide two techniques to guarantee an efﬁcient yet effective AD-VAT.
Revised: • We provide two techniques to guarantee an efﬁcient yet effective AD-VAT.

Original: A partial zero-sum reward structure, which signiﬁcantly stabilizes the training.
Revised: A partial zero-sum reward structure, which signiﬁcantly stabilizes the training.

Original: A tracker-aware network for the target, which yields better escaping policy and consequently better tracking policy.
Revised: A tracker-aware network for the target, which yields better escaping policy and consequently better tracking policy.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Active Object Tracking.
Revised: Active Object Tracking.

Original: As described above that, active object tracking deals with object tracking and camera control at the same time.
Revised: As described above that, active object tracking deals with object tracking and camera control at the same time.

Original: This problem attracts less attention compared with traditional object tracking (or visual object tracking)
Revised: This problem attracts less attention compared with traditional object tracking (or visual object tracking)

Original: (Wu et al., 2013).
Revised: (Wu et al., 2013).

Original: In general, this problem could be addressed in a two-step manner or in an end-to-end manner.
Revised: In general, this problem could be addressed in a two-step manner or in an end-to-end manner.

Original: In the two-step solution, traditional object tracking and camera control are conducted sequentially to obtain tracking results and manipulate camera.
Revised: In the two-step solution, traditional object tracking and camera control are conducted sequentially to obtain tracking results and manipulate camera.

Original: Great progress has been achieved in traditional object tracking in recent decades (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009;
Revised: Great progress has been achieved in traditional object tracking in recent decades (Babenko et al., 2009; Ross et al., 2008; Mei & Ling, 2009;

Original: Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012).
Revised: Hu et al., 2012; Bolme et al., 2010; Kalal et al., 2012).

Original: Thus one can utilize mature visual tracking algorithms (Henriques et al., 2015; Ma et al., 2015; Choi et al., 2017) to accomplish the passive tracking task.
Revised: Thus one can utilize mature visual tracking algorithms (Henriques et al., 2015; Ma et al., 2015; Choi et al., 2017) to accomplish the passive tracking task.

Original: According to the tracking results, camera control module could be developed to actively follow a target.
Revised: According to the tracking results, camera control module could be developed to actively follow a target.

Original: For example, in (Denzler &
Revised: For example, in (Denzler &

Original: Paulus, 1994) a two-stage method is proposed to handle robot control by motion detection and motion tracking.
Revised: Paulus, 1994) a two-stage method is proposed to handle robot control by motion detection and motion tracking.

Original: Kim et al.
Revised: Kim et al.

Original: (Kim et al., 2005) detect moving objects and track them using an active camera with pan/tilt/zoom.
Revised: (Kim et al., 2005) detect moving objects and track them using an active camera with pan/tilt/zoom.

Original: In (Hong et al., 2018), two modules, a perception module and a control policy module, are learned separately to train an agent accomplishing both an obstacle avoidance task and a target following task.
Revised: In (Hong et al., 2018), two modules, a perception module and a control policy module, are learned separately to train an agent accomplishing both an obstacle avoidance task and a target following task.

Original: Admittedly, tracking algorithms have been successful while still not perfect.
Revised: Admittedly, tracking algorithms have been successful while still not perfect.

Original: Camera control based on tracking results is challenging due to factors such as that the correspondence between image space and camera parameter space is unknown.
Revised: Camera control based on tracking results is challenging due to factors such as that the correspondence between image space and camera parameter space is unknown.

Original: Additionally, joint tuning of visual tracking and camera control is expensive and encounters trial-and-errors in real world.
Revised: Additionally, joint tuning of visual tracking and camera control is expensive and encounters trial-and-errors in real world.

Original: In end-to-end methods, direct mapping between raw input frame and camera action is established.
Revised: In end-to-end methods, direct mapping between raw input frame and camera action is established.

Original: Thus the intermediate visual tracking results are not necessarily required.
Revised: Thus the intermediate visual tracking results are not necessarily required.

Original: For instance, by reinforcement learning, camera is controlled by signal outputted from a Conv-LSTM network given raw input frame (Luo et al., 2018;
Revised: For instance, by reinforcement learning, camera is controlled by signal outputted from a Conv-LSTM network given raw input frame (Luo et al., 2018;

Original: Luo et al., 2019).
Revised: Luo et al., 2019).

Original: This end-to-end solution veriﬁes the effectiveness, while not efﬁcient enough to solve this problem.
Revised: This end-to-end solution veriﬁes the effectiveness, while not efﬁcient enough to solve this problem.

Original: To improve the generalization potential, environment augmentation is conducted in this work.
Revised: To improve the generalization potential, environment augmentation is conducted in this work.

Original: However, the target itself, like the path, motion pattern is ﬁxed in their augmented environments, which is believed to limit the performance.
Revised: However, the target itself, like the path, motion pattern is ﬁxed in their augmented environments, which is believed to limit the performance.

Original: This inspires us to resort to the idea of dueling in this paper, i.e. , the target learns to get rid of the tracker by itself, guiding the learning of the tracker.
Revised: This inspires us to resort to the idea of dueling in this paper, i.e. , the target learns to get rid of the tracker by itself, guiding the learning of the tracker.

Original: Adversarial Reinforcement Learning.
Revised: Adversarial Reinforcement Learning.

Original: Using an adversarial framework to improve the RL agent’s robustness is not a new concept.
Revised: Using an adversarial framework to improve the RL agent’s robustness is not a new concept.

Original: In (Huang et al., 2017), they add nonrandom adversarial noise to state input for the purpose of altering or misdirecting policies.
Revised: In (Huang et al., 2017), they add nonrandom adversarial noise to state input for the purpose of altering or misdirecting policies.

Original: Mandlekar et al.
Revised: Mandlekar et al.

Original: (Mandlekar et al., 2017) use adversarial examples to actively choose perturbations during training in physical domains.
Revised: (Mandlekar et al., 2017) use adversarial examples to actively choose perturbations during training in physical domains.

Original: In (Pinto et al., 2017), an adversary is introduced to apply adversarial force during training.
Revised: In (Pinto et al., 2017), an adversary is introduced to apply adversarial force during training.

Original: It produces an agent whose policy is robust to a wide range of noise at testing time.
Revised: It produces an agent whose policy is robust to a wide range of noise at testing time.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: (Sukhbaatar et al., 2017) pit a virtual agent Alice against another agent Bob, where Alice creates steadily more difﬁcult challenges for Bob to complete.
Revised: (Sukhbaatar et al., 2017) pit a virtual agent Alice against another agent Bob, where Alice creates steadily more difﬁcult challenges for Bob to complete.

Original: In (Held et al., 2017), they optimize a generator network via adversarial training to make an agent automatically produce tasks that are always at the appropriate level of difﬁculty for the agent.
Revised: In (Held et al., 2017), they optimize a generator network via adversarial training to make an agent automatically produce tasks that are always at the appropriate level of difﬁculty for the agent.

Original: Roughly speaking, in these proposed methods the adversary is viewed as a ghost/virtual player, which is unseen and could only challenge the protagonist by adding noise in the observation (Huang et al., 2017), action (Pinto et al., 2017) and the system dynamics (Mandlekar et al., 2017), or by generating the goal/initial position for navigation task (Held et al., 2017; Sukhbaatar et al., 2017).
Revised: Roughly speaking, in these proposed methods the adversary is viewed as a ghost/virtual player, which is unseen and could only challenge the protagonist by adding noise in the observation (Huang et al., 2017), action (Pinto et al., 2017) and the system dynamics (Mandlekar et al., 2017), or by generating the goal/initial position for navigation task (Held et al., 2017; Sukhbaatar et al., 2017).

Original: In this paper, we design a two-agent no-cooperative game for VAT task, where the tracker intends to lockup the target, while the target tries to escape from the tracker.
Revised: In this paper, we design a two-agent no-cooperative game for VAT task, where the tracker intends to lockup the target, while the target tries to escape from the tracker.

Original: Unlike the aforementioned previous work, the adversary (target to be tracked) in
Revised: Unlike the aforementioned previous work, the adversary (target to be tracked) in

Original: AD-VAT is a physical player, which could fully control the movement of the target at any time step.
Revised: AD-VAT is a physical player, which could fully control the movement of the target at any time step.

Original: In this paper, such two-physical-player-competition is referred to as “dueling”.
Revised: In this paper, such two-physical-player-competition is referred to as “dueling”.

Original: We argue that such a fully controllable opponent could bring to the protagonist more challenges during training and henceforth produces a more robust visual tracker.
Revised: We argue that such a fully controllable opponent could bring to the protagonist more challenges during training and henceforth produces a more robust visual tracker.

Original: The proposed approach is also related to self-play (Silver et al., 2016; 2017; Bansal et al., 2017).
Revised: The proposed approach is also related to self-play (Silver et al., 2016; 2017; Bansal et al., 2017).

Original: In self-play, two physical players usually compete for the same goal, with symmetric observation and action space.
Revised: In self-play, two physical players usually compete for the same goal, with symmetric observation and action space.

Original: Usually, both players share the same model so as to ensure each agent seeing an environmental dynamics at appropriate difﬁculty level.
Revised: Usually, both players share the same model so as to ensure each agent seeing an environmental dynamics at appropriate difﬁculty level.

Original: It is thus viewed as a means to automatically generate learning curriculum (Bengio et al., 2009; Kumar et al., 2010).
Revised: It is thus viewed as a means to automatically generate learning curriculum (Bengio et al., 2009; Kumar et al., 2010).

Original: Our setting is substantively different from standard self-play.
Revised: Our setting is substantively different from standard self-play.

Original: In AD-VAT, the two players are asymmetric in observation and task.
Revised: In AD-VAT, the two players are asymmetric in observation and task.

Original: The target observes more, and is equipped with additional auxiliary task for its adversarial policy.
Revised: The target observes more, and is equipped with additional auxiliary task for its adversarial policy.

Original: Thus, the target and tracker could not share the same model while learning in AD-VAT, which would make the learning become unstable.
Revised: Thus, the target and tracker could not share the same model while learning in AD-VAT, which would make the learning become unstable.

Original: To address the issue, we propose two techniques which stabilize the training, as described in Sec. 3.
Revised: To address the issue, we propose two techniques which stabilize the training, as described in Sec. 3.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce our proposed method: Asymmetric Dueling mechanism for learning
Revised: In this section, we introduce our proposed method: Asymmetric Dueling mechanism for learning

Original: Visual Active Tracking (AD-VAT) .
Revised: Visual Active Tracking (AD-VAT) .

Original: At ﬁrst the proposed method is formulated as a two-player game.
Revised: At ﬁrst the proposed method is formulated as a two-player game.

Original: Then we illustrate the two key components in AD-VAT: partial zero-sum reward structure and a tracker-aware model for the target.
Revised: Then we illustrate the two key components in AD-VAT: partial zero-sum reward structure and a tracker-aware model for the target.

Original: 3. F ORMULATION
Revised: 3. F ORMULATION

Original: We adopt the Partial Observable Two-Agent Game settings (Srinivasan et al., 2018), which extends the Markov Game (Littman, 1994) to partial observation.
Revised: We adopt the Partial Observable Two-Agent Game settings (Srinivasan et al., 2018), which extends the Markov Game (Littman, 1994) to partial observation.

Original: For the notations of our two-agent game, let superscript α denote the tracker (agent 1) and superscript β denote the target (agent 2).
Revised: For the notations of our two-agent game, let subscript 1 denote the tracker (agent 1) and subscript 2 denote the target (agent 2).

Original: The game is governed by the tuple < S , O α , O β , A α , A β , r α , r β , P > , where S , O , A , r, P denote state space, observation space, action space, reward function and environment state transition probability, respectively.
Revised: The game is governed by the tuple < S , O 1 , O 2 , A 1 , A 2 , r 1 , r 2 , P > , where S , O , A , r, P denote state space, observation space, action space, reward function and environment state transition probability, respectively.

Original: Let subscript t ∈ { 1 , 2 , ... } denote the time step.
Revised: Let subscript t ∈ { 1 , 2 , ... } denote the time step.

Original: In the case of partial observation, we have the observation o α,t
Revised: In the case of partial observation, we have the observation o 1 ,t

Original: = o α,t ( s t , s t − 1 , o t − 1 ) , where o t , o t − 1 ∈ O , s t , s t − 1 ∈ S .
Revised: = o 1 ,t ( s t , s t − 1 , o t − 1 ) , where o t , o t − 1 ∈ O , s t , s t − 1 ∈ S .

Original: It reduces to o α,t =
Revised: It reduces to o 1 ,t =

Original: s t in case of full observation.
Revised: s t in case of full observation.

Original: The counterpart notation o β,t is deﬁned likewise.
Revised: The counterpart notation o 2 ,t is deﬁned likewise.

Original: When the two agents take simultaneous actions a α,t ∈
Revised: When the two agents take simultaneous actions a 1 ,t ∈

Original: A α , a β,t ∈
Revised: A 1 , a 2 ,t ∈

Original: A β , the updated state s t +1 is drawn from the environment state transition probability P ( ·| s t , a α,t , a β,t ) .
Revised: A 2 , the updated state s t +1 is drawn from the environment state transition probability P ( ·| s t , a 1 ,t , a 2 ,t ) .

Original: Meanwhile, the two agents receive rewards r α,t = r α,t ( s t , a α,t ) , r β,t = r β,t ( s t , a β,t ) .
Revised: Meanwhile, the two agents receive rewards r 1 ,t = r 1 ,t ( s t , a 1 ,t ) , r 2 ,t = r 2 ,t ( s t , a 2 ,t ) .

Original: The policy of the tracker, π α ( a α,t | o α,t ) , is a distribution over tracker action a α,t conditioned on its observation o α,t .
Revised: The policy of the tracker, π 1 ( a 1 ,t | o 1 ,t ) , is a distribution over tracker action a 1 ,t conditioned on its observation o 1 ,t .

Original: We rely on model free independent Reinforcement Learning to learn π α .
Revised: We rely on model-free independent Reinforcement Learning to learn π 1 .

Original: Speciﬁcally, the policy takes as function approximator a Neural Network with parameter θ α , written as
Revised: Speciﬁcally, the policy takes as function approximator a Neural Network with parameter θ 1 , written as

Original: [Equation]
Revised: [Equation]

Original: Likewise, the policy of the target can be written as
Revised: Likewise, the policy of the target can be written as

Original: [Equation]
Revised: [Equation]

Revised: Note that we further extend the policy to a tracker-aware policy as Eq.(7).

Original: The tracker intends to maximize its expected return
Revised: The tracker intends to maximize its expected return

Original: [Equation]
Revised: [Equation]

Original: by learning the parameter θ α , where T denotes the horizon length of an episode and r α,t is the immediate reward of the tracker at time step t .
Revised: by learning the parameter θ 1 , where T denotes the horizon length of an episode and r 1 ,t is the immediate reward of the tracker at time step t .

Original: In contrast, the target tries to maximize
Revised: In contrast, the target tries to maximize

Revised: E π 1 ,π(cid:34) T (cid:88) t

Original: [Equation] by learning θ β .
Revised: = r 2 ,t (cid:35) (4) by learning θ 2 .

Original: 3. R EWARD S TRUCTURE
Revised: 3. R EWARD S TRUCTURE

Original: The conventional adversarial methods (Bansal et al., 2017; Pinto et al., 2017) usually formulate the policy learning as a zero-sum game.
Revised: The conventional adversarial methods (Bansal et al., 2017; Pinto et al., 2017) usually formulate the policy learning as a zero-sum game.

Original: In the zero-sum game, the sum of the reward of each agent is always 0 , e.g. , r α,t + r β,t = 0 .
Revised: In the zero-sum game, the sum of the reward of each agent is always 0 , e.g. , r 1 ,t + r 2 ,t = 0 .

Original: However, such kind of formulation is not suitable for VAT.
Revised: However, such kind of formulation is not suitable for VAT.

Original: Considering a case that, when the two opponents are too far to observe each other, their taken actions can hardly inﬂuence the observation of their opponents directly under the partial observable game.
Revised: Considering a case that, when the two opponents are too far to observe each other, their taken actions can hardly inﬂuence the observation of their opponents directly under the partial observable game.

Original: In this case, the sampled experiences are usually meaningless and ineffective for improving the skill level of the agent.
Revised: In this case, the sampled experiences are usually meaningless and ineffective for improving the skill level of the agent.

Original: So constraining the competition in the observable range would make the learning more efﬁcient.
Revised: So constraining the competition in the observable range would make the learning more efﬁcient.

Original: Motivated by this, we shape a partial zero-sum reward structure, which utilizes the zero-sum reward only when the target is observed by the tracker, but gives penalties to each agent when they are far.
Revised: Motivated by this, we shape a partial zero-sum reward structure, which utilizes the zero-sum reward only when the target is observed by the tracker, but gives penalties to each agent when they are far.

Original: In the following, we will introduce the details of the partial zero-sum reward structure for visual active tracking.
Revised: In the following, we will introduce the details of the partial zero-sum reward structure for visual active tracking.

Original: Reward for tracker.
Revised: Reward for tracker.

Original: The reward for tracker is similar to that in (Luo et al., 2018), composing of a positive constant and an error penalty term.
Revised: The reward for tracker is similar to that in (Luo et al., 2018), composing of a positive constant and an error penalty term.

Original: However, we do not take the orientation discrepancy between the target and the tracker into consideration.
Revised: Differently, we do not take the orientation discrepancy between the target and the tracker into consideration.

Original: Moreover, we measure the relative position error based on a polar coordinate system in the 3D environment, considering the model of the camera observation.
Revised: Considering the model of the camera observation, we measure the relative position error based on a polar coordinate system, where the tracker is at the origin (0 , 0) . In this tracker-centric coordinate system, the target’s real and expected position are represented by ( ρ 2 , θ 2 ) and ( ρ ∗ 2 , θ ∗ 2 ) , respectively. Note that ρ is the distance to the tracker, θ is the relative angle to the front of the tracker.

Original: With a slight abuse of notation, we can now write the reward function as
Revised: With a slight abuse of notation, we can now write the reward function as

Original: [Equation]
Revised: [Equation]

Original: here A > 0 , ζ > 0  are tuning parameters, specially A = 1 . 0 , ζ =2 . 0 .
Revised: here A > 0 , ζ > 0 , ξ ≥ 0 are tuning parameters, ξ = 0 in the 2D environment.

Original: P β and P exp represent the target object’s position and the expected position. Note that P is the relative coordinates in the tracker-centric coordinate system. Each term of the coordinate is normalized regarding the range of the observationR . Speciﬁcally, P is plane position ( x, y ) in the 2D environment, but is ( ρ, θ ) under a polar coordinates in 3D environment.
Revised: We do not use the direction error as part of the penalty, in the reason that the observation is omnidirectional in the 2D environments.

Original: ρ  is the  distance between the tracker and target, θ  is the relative angleto the front of the camera.
Revised: ρ max is the max observable distance to the tracker. θ max is the max view angle of the camera model.

Revised: which equals to the Field of View (FoV).

Original: Besides, the reward is clipped to be in the range of [ − A, A ] to avoid over punishment when the object is far away from the expected position.
Revised: Besides, the reward is clipped to be in the range of [ − A, A ] to avoid over punishment when the object is far away from the expected position.

Original: Reward for target.
Revised: Reward for target.

Original: The reward for the target object is closely related to the reward of the tracker, written as:
Revised: The reward for the target object is closely related to the reward of the tracker, written as:

Original: [Equation]
Revised: [Equation]

Original: if tracker could observe the target

Original: [Equation]

Original: otherwise

Original: [Equation]

Original: where r α is the reward of the tracker as deﬁned in Eq.
Revised: where r 1 is the reward of the tracker as deﬁned in Eq.

Original: (5), ξ > 0 is a tuning parameter controlling the factor of the penalty term, and P far represents the farthest point to tracker within the observation.
Revised: (5), µ > 0 , ν ≥ 0 are tuning parameters controlling the factor of each penalty term. ν is 0 in the 2D environment, as the angular penalty factor ξ in Eq.

Original: The  reward  is also clipped when r t is smaller than − A .
Revised: The target is in the nearly observable range, where ρ 2 < ρ max and | θ 2 | < θ max . In the observable range, the reward function is simpliﬁed to r 2 = − r 1 , which means that the target and tracker play a zero-sum game. When the target gets out of the observable range, the penalty term will take effect on the reward.

Original: The insight behind this formulation is that, we do not encourage the target to escape from the tracker as far as possible. Alternatively, by applying this reward function, the optimal policy for the target we expect should be escaping and disappearing from the observation range of the tracker but keeping close to the tracker.
Revised: The farther the target goes out of the range, the larger the penalty it gets. By applying this reward function, the optimal policy for the target we expect should be escaping and disappearing from the observable range of the tracker but keeping close to the edge of the range. ris also clipped in the range of [ − A, A ] .

Original: In Appendix. A we visualize the r α + r β given in Eq. (5) and (6) for a better understanding.
Revised: Furthermore, we provide the details of each parameters and the visualization of the r 1 + r 2 in Appendix A .

Original: 3. T RACKER - AWARE T ARGET
Revised: 3. T RACKER - AWARE T ARGET

Original: By tracker-awareness, the target would be “stronger” than the tracker, as it knows what the tracker knows.
Revised: By tracker-awareness, the target would be “stronger” than the tracker, as it knows what the tracker knows.

Original: This idea manifests an ancient Chinese proverb, “know the enemy, know yourself, and in every battle you will be victorious”, from the masterpiece Sun Tzu on the Art of War Sun (2008).
Revised: This idea manifests an ancient Chinese proverb, “know the enemy, know yourself, and in every battle you will be victorious”, from the masterpiece Sun Tzu on the Art of War Sun (2008).

Original: The conventional adversary usually uses only its own observation Bansal et al.
Revised: The conventional adversary usually uses only its own observation Bansal et al.

Original: (2017) or shares the same observation as the protagonist Pinto et al.
Revised: (2017) or shares the same observation as the protagonist Pinto et al.

Original: Recall the target policy π β ( a β,t | o β,t ; θ β ) written as in Eq.
Revised: Recall the target policy π 2 ( a 2 ,t | o 2 ,t ; θ 2 ) written as in Eq.

Original: However, the imperfect/partial o β,t observation seriously degrades the performance of the adversary.
Revised: However, the imperfect/partial o 2 ,t observation seriously degrades the performance of the adversary.

Original: Thus, we propose a “tracker-aware” model for the target.
Revised: Thus, we propose a “tracker-aware” model for the target.

Original: Besides the target’s own observation, we additionally feed the observation and action from the tracker into the target network, in order to enrich the input information of the target.
Revised: Besides the target’s own observation, we additionally feed the observation and action from the tracker into the target network, in order to enrich the input information of the target.

Original: Moreover, we add an auxiliary task, which predicts the immediate reward of the tracker (see the TRP module in Fig. 1).
Revised: Moreover, we add an auxiliary task, which predicts the immediate reward of the tracker (see the TRP module in Fig. 1).

Original: This auxiliary task can be treated as a kind of “opponent modeling”, and alleviate the difﬁculty in its own policy learning.
Revised: This auxiliary task can be treated as a kind of “opponent modeling”, and alleviate the difﬁculty in its own policy learning.

Original: By doing so, we can write the output heads of such a “tracker-aware” policy as π β ( a β,t , ˆ r α,t | o α,t , a α,t , o β,t ; θ β ) with a slight abuse of notations, where ˆ r α,t is the predicted immediate reward for the tracker and o α,t , a α,t are respectively the observation and the action of the tracker.
Revised: By doing so, we can write the output heads of such a “tracker-aware” policy as: [Equation] where ˆ r 1 ,t is the predicted immediate reward for the tracker and o 1 ,t , a 1 ,t are respectively the observation and the action of the tracker.

Original: Empirical results show that the tracker-aware target yields more diversiﬁed escaping policy and ﬁnally helps producing a more robust tracker.
Revised: Empirical results show that the tracker-aware target yields more diversiﬁed escaping policy and ﬁnally helps producing a more robust tracker.

Original: Note that we cannot apply the trick for a tracker, as the tracker has to use its own observation during testing/deployment.
Revised: Note that we cannot apply the trick for a tracker, as the tracker has to use its own observation during testing/deployment.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The following experiments explore our approach for the VAT task from 2D to 3D environments.
Revised: The following experiments explore our approach for the VAT task from 2D to 3D environments.

Original: 4. E NVIRONMENTS
Revised: 4. E NVIRONMENTS

Original: 2D Environments.
Revised: 2D Environments.

Original: Although 2D environments exhibit unreality to some extent, they are ideal for evaluating and quantifying the effectiveness of each method, sidestepping the uncontrolled noise in observation and action.
Revised: Although 2D environments exhibit unreality to some extent, they are ideal for evaluating and quantifying the effectiveness of each method, sidestepping the uncontrolled noise in observation and action.

Original: In the 2D Environment, maps are represented by a 80 × 80 matrix, where 0 denotes free space, 1 denotes an obstacle, 2 denotes the tracker, and 4 denotes the target.
Revised: In the 2D Environment, maps are represented by a 80 × 80 matrix, where 0 denotes free space, 1 denotes an obstacle, 2 denotes the tracker, and 4 denotes the target.

Original: We randomly generate the maps of two patterns, “maze” and “block” (see examples in the top row of Fig.
Revised: We randomly generate the maps of two patterns, “maze” and “block” (see examples in the top row of Fig.

Original: We use the “block” maps for training and both kinds of maps for testing.
Revised: We use the “block” maps for training and both kinds of maps for testing.

Original: The observation of each agent is a matrix of size 13 × 13 around the agent.
Revised: The observation of each agent is a matrix of size 13 × 13 around the agent.

Original: The tracker’s goal is placing the target as close to the center of the observed matrix as possible.
Revised: The tracker’s goal is placing the target as close to the center of the observed matrix as possible.

Original: During each episode, the tracker starts from a free space in the map randomly, and the target starts around the tracker in a 3 × 3 tracker-centric matrix.
Revised: During each episode, the tracker starts from a free space in the map randomly, and the target starts around the tracker in a 3 × 3 tracker-centric matrix.

Original: At each step, the agent could take an action to move toward one of four directions.
Revised: At each step, the agent could take an action to move toward one of four directions.

Original: The experiments in the 2D environments are dedicated to evaluate and quantify the effectiveness of our approach in ideal conditions.
Revised: The experiments in the 2D environments are dedicated to evaluate and quantify the effectiveness of our approach in ideal conditions.

Original: 3D Environments.
Revised: 3D Environments.

Original: The 3D environments show high ﬁdelity, aiming to mimic the real-world active tracking scenarios.
Revised: The 3D environments show high ﬁdelity, aiming to mimic the real-world active tracking scenarios.

Original: The 3D environments are built on the Unreal Engine, which and could ﬂexibly simulate a photo-realistic world.
Revised: The 3D environments are built on the Unreal Engine, which and could ﬂexibly simulate a photo-realistic world.

Original: We employ UnrealCV
Revised: We employ UnrealCV

Original: (Qiu et al., 2017), which provides convenient
Revised: (Qiu et al., 2017), which provides convenient

Original: APIs, along with a wrapper (Zhong et al., 2017) compatible with OpenAI Gym (Brockman et al., 2016), for interactions between RL algorithms and the environment.
Revised: APIs, along with a wrapper (Zhong et al., 2017) compatible with OpenAI Gym (Brockman et al., 2016), for interactions between RL algorithms and the environment.

Original: The observation is an image of the ﬁrst-person view of the world as seen by the agent.
Revised: The observation is an image of the ﬁrst-person view of the world as seen by the agent.

Original: The actions space is discrete with seven candidate motions, move-forward, move-backward, turn-left, turn-right, turn-left-and-move-forward, turn-right-and-move-forward , and no-op .
Revised: The actions space is discrete with seven candidate motions, move-forward, move-backward, turn-left, turn-right, turn-left-and-move-forward, turn-right-and-move-forward , and no-op .

Original: For training, we build a Domain Randomized Room(DR
Revised: For training, we build a Domain Randomized Room(DR

Original: Room) with two controllable players (target and tracker).
Revised: Room) with two controllable players (target and tracker).

Original: The domain randomization techniques could help agents learn better feature representation in terms of visual observation.
Revised: The domain randomization techniques could help agents learn better feature representation in terms of visual observation.

Original: In testing, we focus on the transferring ability of the tracker to different unseen environments.
Revised: In testing, we focus on the transferring ability of the tracker to different unseen environments.

Original: We use three realistic scenarios, Urban City, Snow Village and Parking Lot , to mimic real-world scenes for evaluating.
Revised: We use three realistic scenarios, Urban City, Snow Village and Parking Lot , to mimic real-world scenes for evaluating.

Original: The bottom row in Fig.
Revised: The bottom row in Fig.

Original: 2 shows the snapshots of the four 3D environments used.
Revised: 2 shows the snapshots of the four 3D environments used.

Original: Details of the four environments are:
Revised: Details of the four environments are:

Original: • DR Room is a plain room comprised of ﬂoor and walls only, but the textures and illumination conditions are randomized.
Revised: • DR Room is a plain room comprised of ﬂoor and walls only, but the textures and illumination conditions are randomized.

Original: For the textures, we randomly choose pictures from a texture dataset (Kylberg, 2011) and place them on the surface of the walls, ﬂoor, and players.
Revised: For the textures, we randomly choose pictures from a texture dataset (Kylberg, 2011) and place them on the surface of the walls, ﬂoor, and players.

Original: For the illumination condition, we randomize the intensity and color of each light source as well as each position, orientation.
Revised: For the illumination condition, we randomize the intensity and color of each light source as well as each position, orientation.

Original: • Urban City is a high-ﬁdelity street view of an urban city, including well-modeled buildings, streets, trees and transportation facilities.
Revised: • Urban City is a high-ﬁdelity street view of an urban city, including well-modeled buildings, streets, trees and transportation facilities.

Original: Besides, there are some puddles on the road, reﬂecting the objects and buildings.
Revised: Besides, there are some puddles on the road, reﬂecting the objects and buildings.

Original: • Snow Village consists of bumpy snowﬁelds with several trees, bushes and some cabins.
Revised: • Snow Village consists of bumpy snowﬁelds with several trees, bushes and some cabins.

Original: Occasionally, the target will be occluded by trees and bushes, and the tracker will be distracted by the snowﬂake and halo.
Revised: Occasionally, the target will be occluded by trees and bushes, and the tracker will be distracted by the snowﬂake and halo.

Original: • Parking Lot is an underground parking lot with complex illumination condition.
Revised: • Parking Lot is an underground parking lot with complex illumination condition.

Original: The lack of light source makes the illumination uneven, i.e. , some places are bright but the others are dark.
Revised: The lack of light source makes the illumination uneven, i.e. , some places are bright but the others are dark.

Original: Besides, the pillars may occlude the target to track.
Revised: Besides, the pillars may occlude the target to track.

Original: [Figure]
Revised: [Figure]

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We provide two kinds of base target agent to randomly generate trajectories as baselines to compare with, Rambler (Ram) and Navigator (Nav) .
Revised: We provide two kinds of base target agent to randomly generate trajectories as baselines to compare with, Rambler (Ram) and Navigator (Nav) .

Original: Agent Ram walks randomly without any purpose like a man.
Revised: Agent Ram walks randomly without any purpose like a man.

Original: Technically, it randomly samples actions from the action space and keeps executing the action n times, where n is also a random integer in the range of (1 , 10) .
Revised: Technically, it randomly samples actions from the action space and keeps executing the action n times, where n is also a random integer in the range of (1 , 10) .

Original: Agent Nav is a navigator, which plans the shortest path to a speciﬁc goal.
Revised: Agent Nav is a navigator, which plans the shortest path to a speciﬁc goal.

Original: Thus, it could navigate to most of free space in the map.
Revised: Thus, it could navigate to most of free space in the map.

Original: To randomize the trajectories, the goal coordinate and the initial coordinate are randomly sampled.
Revised: To randomize the trajectories, the goal coordinate and the initial coordinate are randomly sampled.

Original: In most of case, Ram prefers to walk around a local area repeatedly.
Revised: In most of case, Ram prefers to walk around a local area repeatedly.

Original: In contrast, Nav would like to explore the map globally, shown as the yellow trajectories in Fig.
Revised: In contrast, Nav would like to explore the map globally, shown as the yellow trajectories in Fig.

Original: Thus we regard trajectories from the Ram as easier cases, and trajectories from Nav as more difﬁcult cases for tracker.
Revised: Thus we regard trajectories from the Ram as easier cases, and trajectories from Nav as more difﬁcult cases for tracker.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: Each agent is trained by A3C (Mnih et al., 2016), a commonly used reinforcement learning algorithm.
Revised: Each agent is trained by A3C (Mnih et al., 2016), a commonly used reinforcement learning algorithm.

Original: The code for A3C is based on a pytorch implementation (Grifﬁs).
Revised: The code for A3C is based on a pytorch implementation (Grifﬁs).

Original: Multiple workers are running in parallel when training.
Revised: Multiple workers are running in parallel when training.

Original: Speciﬁcally, 16 workers are used in the 2D experiment, and 4 workers are used in the 3D experiment.
Revised: Speciﬁcally, 16 workers are used in the 2D experiment, and 4 workers are used in the 3D experiment.

Original: Network Architecture.
Revised: Network Architecture.

Original: For the tracker, we follow the end-to-end Conv-LSTM network architecture as (Luo et al., 2018).
Revised: For the tracker, we follow the end-to-end Conv-LSTM network architecture as (Luo et al., 2018).

Original: Differently, there is no fully-connected layer between the Conv-Net and the
Revised: Differently, there is no fully-connected layer between the Conv-Net and the

Original: LSTM-Net in this paper.
Revised: LSTM-Net in this paper.

Original: The Conv-Net is a two-layer CNN for the 2D experiments and four-layer
Revised: The Conv-Net is a two-layer CNN for the 2D experiments and four-layer

Original: CNN for the 3D experiments.
Revised: CNN for the 3D experiments.

Original: In the 3D experiments, the input color images are transformed to gray image and the pixel values are scaled to [ − 1 , 1] .
Revised: In the 3D experiments, the input color images are transformed to gray image and the pixel values are scaled to [ − 1 , 1] .

Original: we also develop the same Conv-LSTM network architecture for the target, but different in the input and output, shown as Fig.
Revised: we also develop the same Conv-LSTM network architecture for the target, but different in the input and output, shown as Fig.

Original: The network parameters are updated with a shared Adam optimizer.
Revised: The network parameters are updated with a shared Adam optimizer.

Original: Hyper Parameters.
Revised: Hyper Parameters.

Original: For the tracker, the learning rates δ α and δ (cid:48) α in 2D and 3D environments are 0.001 and 0.0001, respectively.
Revised: For the tracker, the learning rates δ 1 and δ (cid:48) 1 in 2D and 3D environments are 0.001 and 0.0001, respectively.

Original: The reward discount factor γ = 0 .
Revised: The reward discount factor γ = 0 .

Original: 9 , generalized advantage estimate parameter τ = 1 . 00 , and regularizer factor for tracker λ α = 0 .
Revised: 9 , generalized advantage estimate parameter τ = 1 . 00 , and regularizer factor for tracker λ 1 = 0 .

Original: The parameter updating frequency n is 20, and the maximum global iteration for training is 150 K. Comparing to the tracker, a higher regularizer factor is used for encouraging the target to explore, λ β = 0 .
Revised: The parameter updating frequency n is 20, and the maximum global iteration for training is 150 K. Comparing to the tracker, a higher regularizer factor is used for encouraging the target to explore, λ 2 = 0 .

Original: 2 in 2D and λ (cid:48)
Revised: 2 in 2D and λ (cid:48)

Original: β = 0 .
Revised: 2 = 0 .

Original: in 3D.
Revised: in 3D.

Original: The more exploration taken by target, the more diverse the generated trajectories are.
Revised: The more exploration taken by target, the more diverse the generated trajectories are.

Original: It is useful for the learning of the tracker.
Revised: It is useful for the learning of the tracker.

Original: Validation is performed in parallel and the best validation network model is applied to report performance in testing environments.
Revised: Validation is performed in parallel and the best validation network model is applied to report performance in testing environments.

Original: Note that the validation environment is of the same settings as training, except that the target is controlled by a Nav agent.
Revised: Note that the validation environment is of the same settings as training, except that the target is controlled by a Nav agent.

Original: Compared with the
Revised: Compared with the

Original: Ram agent, the Nav agent is more challenging, thus is more suitable for validation.
Revised: Ram agent, the Nav agent is more challenging, thus is more suitable for validation.

Original: Metric.
Revised: Metric.

Original: Two metrics are employed for the experiments.
Revised: Two metrics are employed for the experiments.

Original: Speciﬁcally, Accumulated Reward (AR) and
Revised: Speciﬁcally, Accumulated Reward (AR) and

Original: Episode Length (EL) of each episode are calculated for quantitative evaluation.
Revised: Episode Length (EL) of each episode are calculated for quantitative evaluation.

Original: AR is a comprehensive metric, representing the tracker’s capability about precision and robustness.
Revised: AR is a comprehensive metric, representing the tracker’s capability about precision and robustness.

Original: It is effected by the immediate reward and the episode length.
Revised: It is effected by the immediate reward and the episode length.

Original: Immediate reward measures the goodness of tracking, and
Revised: Immediate reward measures the goodness of tracking, and

Original: EL roughly measures the duration of good tracking.
Revised: EL roughly measures the duration of good tracking.

Original: Because the episode is terminated when the tracker loses the target for continuous 10 steps or reaches the max episode length.
Revised: Because the episode is terminated when the tracker loses the target for continuous 10 steps or reaches the max episode length.

Original: [Table]
Revised: [Table]

Original: 4. R ESULTS ON THE 2D ENVIRONMENT
Revised: 4. R ESULTS ON THE 2D ENVIRONMENT

Original: We quantitatively evaluate the performance of our approach, comparing to the two baselines.
Revised: We quantitatively evaluate the performance of our approach, comparing to the two baselines.

Original: Furthermore, we conduct an ablation study to show the effectiveness of the partial zero-sum reward and tracker-aware model.
Revised: Furthermore, we conduct an ablation study to show the effectiveness of the partial zero-sum reward and tracker-aware model.

Original: Quantitative Evaluation.
Revised: Quantitative Evaluation.

Original: We test the active tracker trained with different target agents in four testing settings, showing the effectiveness of AD-VAT .
Revised: We test the active tracker trained with different target agents in four testing settings, showing the effectiveness of AD-VAT .

Original: Considering the random seed of the environments, we conduct 100 runs in each and report the mean and standard deviation of AR and EL, shown in
Revised: Considering the random seed of the environments, we conduct 100 runs in each and report the mean and standard deviation of AR and EL, shown in

Original: Table 1.
Revised: Table 1.

Original: The max episode length is 500, so the upper bound of EL is 500.
Revised: The max episode length is 500, so the upper bound of EL is 500.

Original: Thus, when EL equals to 500, we could infer that the tracker performs perfectly, without losing the target.
Revised: Thus, when EL equals to 500, we could infer that the tracker performs perfectly, without losing the target.

Original: We note that, at the beginning of the learning, the adversarial target usually walks randomly around the start point, performing similar policy as Ram .
Revised: We note that, at the beginning of the learning, the adversarial target usually walks randomly around the start point, performing similar policy as Ram .

Original: Such target is easier to be found and observed, even though the tracker is in exploration.
Revised: Such target is easier to be found and observed, even though the tracker is in exploration.

Original: Thus, the tracker could warm up faster.
Revised: Thus, the tracker could warm up faster.

Original: With the growth of the tracker, the target gradually explores other motion patterns, which could further reinforce the tracker.
Revised: With the growth of the tracker, the target gradually explores other motion patterns, which could further reinforce the tracker.

Original: Such a learning process is close to the curriculum learning, but the curriculum is automatically produced by the target via adversarial reinforcement learning.
Revised: Such a learning process is close to the curriculum learning, but the curriculum is automatically produced by the target via adversarial reinforcement learning.

Original: We also report the learning curve as the mean of cumulative rewards in the validation environment, shown as the left sub-ﬁgure in Fig.
Revised: We also report the learning curve as the mean of cumulative rewards in the validation environment, shown as the left sub-ﬁgure in Fig.

Original: It consistently shows the advantage of the proposed AD-VAT .
Revised: It consistently shows the advantage of the proposed AD-VAT .

Original: Ablation Study.
Revised: Ablation Study.

Original: In Section 3, we introduced two components to implement AD-VAT : partial zerosum reward (PZR) and tracker-aware model (TAM) for target.
Revised: In Section 3, we introduced two components to implement AD-VAT : partial zerosum reward (PZR) and tracker-aware model (TAM) for target.

Original: These two components are important as they inﬂuence the natural curriculum for the tracker.
Revised: These two components are important as they inﬂuence the natural curriculum for the tracker.

Original: Thus, we report an ablation study result to show the effectiveness of these two components, shown in Fig.
Revised: Thus, we report an ablation study result to show the effectiveness of these two components, shown in Fig.

Original: The naive method is an intuitive idea that target only uses its own observation with auxiliary task, guided by a zero-sum reward.
Revised: The naive method is an intuitive idea that target only uses its own observation with auxiliary task, guided by a zero-sum reward.

Original: As shown in the right sub-ﬁgure of Fig.
Revised: As shown in the right sub-ﬁgure of Fig.

Original: 3, using each component separately could improve the sample-efﬁciency, comparing to the naive method.
Revised: 3, using each component separately could improve the sample-efﬁciency, comparing to the naive method.

Original: Besides, PZR contributes to the improvement of the tracking performance more signiﬁcant than TAM.
Revised: Besides, PZR contributes to the improvement of the tracking performance more signiﬁcant than TAM.

Original: Moreover, when combining PZR and TAM, both sample-efﬁciency and the tracking performance are signiﬁcantly boosted, comparing to the other three settings.
Revised: Moreover, when combining PZR and TAM, both sample-efﬁciency and the tracking performance are signiﬁcantly boosted, comparing to the other three settings.

Original: 4. R ESULTS ON THE 3D ENVIRONMENT
Revised: 4. R ESULTS ON THE 3D ENVIRONMENT

Original: In the 3D experiments, we test the generalization of the model in unseen realistic scenarios, showing the transfer potential in real-world scenarios.
Revised: In the 3D experiments, we test the generalization of the model in unseen realistic scenarios, showing the transfer potential in real-world scenarios.

Original: We train three models with different target models ( Ram,
Revised: We train three models with different target models ( Ram,

Original: Nav , and AD-VAT ) in the DR Room.
Revised: Nav , and AD-VAT ) in the DR Room.

Original: And then, we directly run the three trackers in the validation and testing environments, 100 episodes for each, and report AR and EL in Fig.
Revised: And then, we directly run the three trackers in the validation and testing environments, 100 episodes for each, and report AR and EL in Fig.

Original: The result in DR Room demonstrates again that even though the target is unseen for our AD-VAT tracker, it still outperforms the others.
Revised: The result in DR Room demonstrates again that even though the target is unseen for our AD-VAT tracker, it still outperforms the others.

Original: Note that the DR Room for validation is of the same settings as the training of Nav tracker.
Revised: Note that the DR Room for validation is of the same settings as the training of Nav tracker.

Original: The results in the other environments show that the three models are able to transfer to realistic environment.
Revised: The results in the other environments show that the three models are able to transfer to realistic environment.

Original: We believe that the domain randomization method and the Conv-LSTM network endow the trackers the ability of transferring.
Revised: We believe that the domain randomization method and the Conv-LSTM network endow the trackers the ability of transferring.

Original: However, the tracker’s performance is also highly related to the behavior of the targets during training, especially in complex environments ( Snow Village and Parking Lot ).
Revised: However, the tracker’s performance is also highly related to the behavior of the targets during training, especially in complex environments ( Snow Village and Parking Lot ).

Original: Compared with the two baselines, the adversarial behavior of the target could signiﬁcantly improve the capability of the tracker in these challenging environments.
Revised: Compared with the two baselines, the adversarial behavior of the target could signiﬁcantly improve the capability of the tracker in these challenging environments.

Original: We infer that the target in AD-VAT could explore the environment more acvtively to discover more difﬁcult cases.
Revised: We infer that the target in AD-VAT could explore the environment more acvtively to discover more difﬁcult cases.

Original: For example, in DR Room, the target would prefer to move close to the wall that is similar to itself to fool the tracker (see Fig. 5).
Revised: For example, in DR Room, the target would prefer to move close to the wall that is similar to itself to fool the tracker (see Fig. 5).

Original: By competing with the target, the tracker consequently becomes stronger.
Revised: By competing with the target, the tracker consequently becomes stronger.

Original: In Appendix.
Revised: In Appendix.

Original: C, we further evaluate the capability of our tracker in the real-world video clips, qualitatively.
Revised: C, we further evaluate the capability of our tracker in the real-world video clips, qualitatively.

Original:  C ONCLUSION 
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this paper, we have proposed an asymmetric dueling mechanism for visual active tracking (ADVAT).
Revised: In this paper, we have proposed an asymmetric dueling mechanism for visual active tracking (ADVAT).

Original: Within AD-VAT, agents of tracker and target are learned in an adversarial manner.
Revised: Within AD-VAT, agents of tracker and target are learned in an adversarial manner.

Original: With the design of the partial zero-sum reward structure and tracker-aware model, the reinforced active tracker outperforms baseline methods.
Revised: With the design of the partial zero-sum reward structure and tracker-aware model, the reinforced active tracker outperforms baseline methods.

Original: Experiments including ablation study in both 2D and 3D environments verify the effectiveness of the proposed mechanism.
Revised: Experiments including ablation study in both 2D and 3D environments verify the effectiveness of the proposed mechanism.

Revised: As future work, we would like to: 1) investigate the theoretical justiﬁcation of applying modern

Revised: Multi-Agent RL methods (Lanctot et al., 2017; Srinivasan et al., 2018) to solving Partially Observable

Revised: Markov Game and ﬁnding Nash Equilibrium. 2) further develop the mechanism/model for active tracking in more complex environment ( e.g. , environments with a number of obstacles and moving distractors); 3) adapt the mechanism to other tasks ( e.g. , learning to grab a moving object).

Original:  A CKNOWLEDGMENTS
Revised:  A CKNOWLEDGMENTS

Original: The authors would like to thank Weichao Qiu for his help in extending UnrealCV.
Revised: The authors would like to thank Weichao Qiu for his help in extending UnrealCV.

Original: Fangwei Zhong,
Revised: Fangwei Zhong,

Original: Tingyun Yan and Yizhou Wang were supported in part by the following grants: NSFC-61625201,
Revised: Tingyun Yan and Yizhou Wang were supported in part by the following grants: NSFC-61625201,

Original: NSFC-61527804, Tencent AI Lab Rhino-Bird Focused Research Program
Revised: NSFC-61527804, Tencent AI Lab Rhino-Bird Focused Research Program

Original: No.JR201851,
Revised: No.JR201851,

Original: Qualcomm
Revised: Qualcomm

Original: University Collaborative Research Program.
Revised: University Collaborative Research Program.

Original: A V ISUALIZING P ARTIAL Z ERO - SUM R EWARD
Revised: A D ETAILS OF THE P ARTIAL Z ERO - SUM R EWARD

Revised: A ζ ξ µ ν ρ ∗ρ max θ ∗ θ max 2D     0 grid 6 grid \ 360 ◦ 3D     2 .

Original: {+5 m 5 . 
+}
Revised: {+5 m 5 . 
+}

Revised: m 90 ◦

Original: To help better understand the reward structure given in Eq.
Revised: To help better understand the reward structure given in Eq.

Original: (5) and (6), we visualize the sum r α + r β as heatmap in x − y plane.
Revised: (5) and (6), we visualize the sum r 1 + ras heatmap in x − y plane.

Original: See Fig. 6.
Revised: See Fig. 6.

Original: For the 2D experiment, the observations for both the tracker and target are bird-views.
Revised: For the 2D experiment, the observations for both the tracker and target are bird-views.

Original: We want to penalize that the target gets too far away from the tracker.
Revised: We want to penalize that the target gets too far away from the tracker.

Original: Therefore, the zero-sum area is a circle (Fig. 6, Left), where the tracker is in the centre.
Revised: Therefore, the zero-sum area is a circle (Fig. 6, Left), where the tracker is in the centre.

Original: With the increasing of the distance, the penalty term in r β (see Eq.
Revised: With the increasing of the distance, the penalty term in r 2 (see Eq.

Original: (6)) starts taking effect on the sum.
Revised: (6)) starts taking effect on the sum.

Original: It causes the sum to reduce gradually until the target reaches the dark area, where r β = − A .
Revised: It causes the sum to reduce gradually until the target reaches the dark area, where r 2 = − A .

Original: For the 3D experiment, the observations for both the tracker and target are front-views.
Revised: For the 3D experiment, the observations for both the tracker and target are front-views.

Original: We want to penalize that the target gets too far away from the tracker or that the target cannot be seen by the tracker.
Revised: We want to penalize that the target gets too far away from the tracker or that the target cannot be seen by the tracker.

Original: Thus, the zero-sum area is a sector area (Fig. 6, Right), which approximately ﬁts the Field of View (FoV) of the tracker’s camera.
Revised: Thus, the zero-sum area is a sector area (Fig. 6, Right), which approximately ﬁts the Field of View (FoV) of the tracker’s camera.

Original: Note that the FoV in our experiment is 90 degree.
Revised: Note that the FoV in our experiment is 90 degree.

Original: Both the relative angle θ and distance ρ contribute to the penalty term in r β .
Revised: Both the relative angle θ and distance ρ contribute to the penalty term in r 2 .

Original: Thus the sum decreases like a divergence sector.
Revised: Thus the sum decreases like a divergence sector.

Original: B V ISUALIZING THE T RAINING P ROCESS
Revised: B V ISUALIZING THE T RAINING P ROCESS

Original: For a better understanding of the training process, we record trajectories of the target and the tracker during different training stages.
Revised: For a better understanding of the training process, we record trajectories of the target and the tracker during different training stages.

Original: Speciﬁcally, we have 6 stages, ranging from early training to late training.
Revised: Speciﬁcally, we have 6 stages, ranging from early training to late training.

Original: For each stage we record 100 episodes.
Revised: For each stage we record 100 episodes.

Original: In Fig.
Revised: In Fig.

Original: 7, we plot the target position distribution, instead of the trajectory itself.
Revised: 7, we plot the target position distribution, instead of the trajectory itself.

Original: For ease of visualization, we adopt a relative coordinate system for the target position when drawing the distribution, because the start locations for both the tracker and the target are random upon each episode.
Revised: For ease of visualization, we adopt a relative coordinate system for the target position when drawing the distribution, because the start locations for both the tracker and the target are random upon each episode.

Original: In Fig.
Revised: In Fig.

Original: 7, the distributions are in a start point -centric coordinate system, while in Fig.
Revised: 7, the distributions are in a start point -centric coordinate system, while in Fig.

Original: 8 the distributions are in a tracker -centric coordinate system.
Revised: 8 the distributions are in a tracker -centric coordinate system.

Original: At early training stages (see the left of Fig. 7), AD-VAT and Ram generate similar trajectories, which are randomly walking around the start point.
Revised: At early training stages (see the left of Fig. 7), AD-VAT and Ram generate similar trajectories, which are randomly walking around the start point.

Original: In contrast, for Nav method the target usually goes along a straight line to the goal position, causing the tracker to get lost quickly at beginning.
Revised: In contrast, for Nav method the target usually goes along a straight line to the goal position, causing the tracker to get lost quickly at beginning.

Original: The random walking trajectories help the tracker observe the target appearing in various positions, and henceforth sample more diverse experiences.
Revised: The random walking trajectories help the tracker observe the target appearing in various positions, and henceforth sample more diverse experiences.

Original: As a result, a better exploration is achieved during the early training stage, which is not the case for the Nav method. 
Revised: As a result, a better exploration is achieved during the early training stage, which is not the case for the Nav method. [Figure] [Figure]

Original: With the evolution of the tracker during training, the target will gradually seek more difﬁcult cases to defeat the tracker.
Revised: With the evolution of the tracker during training, the target will gradually seek more difﬁcult cases to defeat the tracker.

Original: In this regard, the target for both Ram and Nav usually explores possible directions uniformly.
Revised: In this regard, the target for both Ram and Nav usually explores possible directions uniformly.

Original: The minor difference is that the Nav tends to explore the map globally, while the Ram is in local (see the right of Fig. 7).
Revised: The minor difference is that the Nav tends to explore the map globally, while the Ram is in local (see the right of Fig. 7).

Original: As for our AD-VAT method, however, the reinforced target could adapt to the capability of the tracker, resulting in different direction choosing patterns at different stage.
Revised: As for our AD-VAT method, however, the reinforced target could adapt to the capability of the tracker, resulting in different direction choosing patterns at different stage.

Original: For example, the target tends to move bottom-right at the third stage, but top-left at the fourth stage.
Revised: For example, the target tends to move bottom-right at the third stage, but top-left at the fourth stage.

Original: See
Revised: See

Original: [Figure]

Original: [Figure]

Original: Fig.
Revised: Fig.

Original: Besides, it seems that the reinforced target could balance the two exploration modes of Nav and Ram naturally.
Revised: Besides, it seems that the reinforced target could balance the two exploration modes of Nav and Ram naturally.

Original: Sometimes it explores the map, and sometimes it duels with the tracker locally.
Revised: Sometimes it explores the map, and sometimes it duels with the tracker locally.

Original: By the dueling mechanism, the target could ﬁnd the weakness of the tracker more often (see the right of Fig.
Revised: By the dueling mechanism, the target could ﬁnd the weakness of the tracker more often (see the right of Fig.

Original: 8), which seems to serve as a kind of importance sampling that enhances the tracker efﬁciently during training.
Revised: 8), which seems to serve as a kind of importance sampling that enhances the tracker efﬁciently during training.

Original: Such a “weakness-ﬁnding” seems absent for the Nav and Ram algorithms.
Revised: Such a “weakness-ﬁnding” seems absent for the Nav and Ram algorithms.

Original: C T ESTING T RACKER ON R EAL -W ORLD V IDEO C LIPS
Revised: C T ESTING T RACKER ON R EAL -W ORLD V IDEO C LIPS

Original: To demonstrate the capability of our tracker in real-world scenarios, we conduct a qualitative evaluation as Luo et al.
Revised: To demonstrate the capability of our tracker in real-world scenarios, we conduct a qualitative evaluation as Luo et al.

Original: In this evaluation, we feed the video clips from VOT dataset (Kristan et al., 2016) to the tracker and observe the network output actions.
Revised: In this evaluation, we feed the video clips from VOT dataset (Kristan et al., 2016) to the tracker and observe the network output actions.

Original: Note that the tracker could not really control the camera movement, hence we regard the test as “passively”.
Revised: Note that the tracker could not really control the camera movement, hence we regard the test as “passively”.

Original: However, the tracker’s output action is expected to be sensitive to the position and the scale of the target in the image.
Revised: However, the tracker’s output action is expected to be sensitive to the position and the scale of the target in the image.

Original: For example, when the target appears in the right (left) side, the tracker tends to turn right (left), trying to ﬁctitiously move the camera to “place” the target in the image center.
Revised: For example, when the target appears in the right (left) side, the tracker tends to turn right (left), trying to ﬁctitiously move the camera to “place” the target in the image center.

Original: By visualizing whether the output action is consistent with the position and scale of the target at each frame, we are able to demonstrate the potential of transferring the tracking ability to real-world.
Revised: By visualizing whether the output action is consistent with the position and scale of the target at each frame, we are able to demonstrate the potential of transferring the tracking ability to real-world.

Original: We plot three “Action” maps, shown in Fig.
Revised: We plot three “Action” maps, shown in Fig.

Original: 9, Fig. 10, and Fig.
Revised: 9, Fig. 10, and Fig.

Original: 11, respectively.
Revised: 11, respectively.

Original: Note that the meaning of each axis is the same as (Luo et al., 2018) except that we normalize the values for better understanding.
Revised: Note that the meaning of each axis is the same as (Luo et al., 2018) except that we normalize the values for better understanding.

Original: In details, the horizontal axis indicates the normalized x-axis position of the target in the image, with a positive (negative) value meaning that a target is in the right (left) side.
Revised: In details, the horizontal axis indicates the normalized x-axis position of the target in the image, with a positive (negative) value meaning that a target is in the right (left) side.

Original: The vertical axis indicates the normalized size of the target, i.e. , the area of the ground truth bounding box.
Revised: The vertical axis indicates the normalized size of the target, i.e. , the area of the ground truth bounding box.

Original: We use seven marks to represent the seven actions respectively, as shown in the legend.
Revised: We use seven marks to represent the seven actions respectively, as shown in the legend.

Original: More results on other VOT videos are available at: https://youtu.be/jv-5HVg_Sf4 .
Revised: More results on other VOT videos are available at: [Equation]