Original: Learning to Look by Self-Prediction
Revised: Learning to Look by Self-Prediction

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We present a method for learning active vision skills, for moving the camera toobserve a robot’s sensors from informative points of view, without external rewardsor labels.
Revised: We present a method for learning active vision skills, for moving the camera to observe a robot’s sensors from informative points of view, without external rewards or labels.

Original: We do this by jointly training a visual predictor network, which predictsfuture returns of the sensors using pixels and a camera control agent, which wereward using the negative error of the predictor.
Revised: We do this by jointly training a visual predictor network, which predicts future returns of the sensors using pixels, and a camera control agent, which we reward using the negative error of the predictor.

Original: The agent thus moves the camerato points of view that are most predictive for a target sensor, which we selectusing a conditioning input to the agent.
Revised: The agent thus moves the camera to points of view that are most predictive for a target sensor, which we select using a conditioning input to the agent.

Original: We show that despite this noisy learnedreward function, the learned policies are competent and precisely frame the sensorto a specific location in the view, which we call an emergent fovea.
Revised: We show that despite this noisy learned reward function, the learned policies are competent, avoid occlusions, and precisely frame the sensor to a specific location in the view, which we call an emergent fovea.

Original: We findthat replacing the conventional camera with a foveal camera further increases thepolicies’ precision.
Revised: We find that replacing the conventional camera with a foveal camera further increases the policies’ precision.

Original:  Introduction
Revised:  Introduction

Original: Human vision does not closely resemble computer vision, as commonly practiced in embodied RLand robotics.
Revised: Human vision does not closely resemble computer vision, as commonly practiced in embodied RL and robotics.

Original: Human vision has moving eyes, fovea, movements such as saccades to frame targets inview, and smooth pursuit to track them [8].
Revised: Human vision has moving eyes, fovea, movements such as saccades to frame targets in view and smooth pursuit to track them [8].

Original: By contrast, vision can be a source of training instabilityin RL [3], leading robotics researchers to avoid it altogether and use object features or untrainedvision modules instead [25], or to stabilize the camera by fixing it to the environment, limiting theagent to stay within the camera’s field of view [19].
Revised: By contrast, vision can be a source of training instability in RL [3], leading robotics researchers to avoid it altogether and use object features or untrained vision modules instead [25] or to stabilize the camera by fixing it to the environment, limiting the agent to stay within the camera’s field of view [19].

Original: An agent that has learned to visually frame objects in a consistent image location actively reducesthe image-space variance attributable to object position.
Revised: An agent that has learned to visually frame objects in a consistent image location actively reduces the image-space variance attributable to object position.

Original: Locking down the object’s position withinthe image could simplify the acquisition of visually-guided manipulation policies, as they can thenfocus on the manipulation aspect of the policy.
Revised: Locking down the object’s position within the image could simplify the acquisition of visually-guided manipulation policies, as they can then focus on the manipulation aspect of the policy.

Original: This intuition has recently seen evidence in roboticmanipulation research on hand-mounted cameras, where the object appears in a consistent position asthe hand is about to grasp it [14, 4, 11, 35, 15].
Revised: This intuition has recently seen evidence in robotic manipulation research on hand-mounted cameras, where the object appears in a consistent position as the hand is about to grasp it [14, 4, 11, 35, 15].

Original: By contrast, humans benefit from decoupling the kinematic chain of the eye from those of the limbs,allowing them to flexibly choose their visual focus in highly dynamic tasks ranging from fieldingbaseballs [21] to traversing challenging terrain [20].
Revised: By contrast, humans benefit from decoupling the kinematic chain of the eye from those of the limbs, allowing them to flexibly choose their visual focus in highly dynamic tasks ranging from fielding baseballs [21] to traversing challenging terrain [20].

Original: These visual policies all share common buildingblocks in the form of fixation and tracking (saccades and smooth pursuit).
Revised: These visual policies all share common building blocks in the form of fixation and tracking (saccades and smooth pursuit).

Original: One inspiration to ourpaper is the question: if fixation is a general visual skill that is key to acquiring more task-specificvisual policies, how can an agent learn it independently of specific tasks?
Revised: One inspiration to our paper is the question: if fixation is a general visual skill that is key to acquiring more task-specific visual policies, how can an agent learn it independently of specific tasks?

Original: This paper demonstrateshow an embodied agent may acquire visual skills in the absence of external rewards.
Revised: This paper demonstrates how an embodied agent may acquire visual skills in the absence of external rewards.

Original: Previous work has investigated related yet distinct ideas.
Revised: Previous work has investigated related yet distinct ideas.

Original: Yang et al.
Revised: Yang et al.

Original:  propose embodied perceptionwhere an agent learns to move around an object to perceive it better.
Revised: [39] propose embodied perception where an agent learns to move around an object to perceive it better.

Original: Nilsson et al.
Revised: Nilsson et al.

Original: [24] study embodiedvision in a navigating agent solving a semantic segmentation task.
Revised: [24] study embodied vision in a navigating agent solving a semantic segmentation task.

Original: Gärtner et al.
Revised: Gärtner et al.

Original: [10] use RL to selectthe best viewpoint in a panoptic camera rig for human pose estimation.
Revised: [10] use RL to select the best viewpoint in a panoptic camera rig for human pose estimation.

Original: Pirinen et al.
Revised: Pirinen et al.

Original: [28] train anagent to seek points of view for 3D reconstruction.
Revised: [28] train an agent to seek points of view for 3D reconstruction.

Original: Jayaraman & Grauman [16] use active vision to
Revised: Jayaraman & Grauman [16] use active vision to

Original: [Figure]
Revised: [Figure]

Original: forecast the effects of a motion.
Revised: forecast the effects of a motion.

Original: More generally, previous works [30, 17] also learn to look around toefficiently gather information about the agent’s surroundings.
Revised: More generally, previous works [30, 17] also learn to look around to efficiently gather information about the agent’s surroundings.

Original:  Methods
Revised:  Methods

Original: Our environment (fig.
Revised: Our environment (fig.

Original: 1a) uses the open-source (Apache 2.0) MuJoCo
Revised: 1a) uses the open-source (Apache 2.0) MuJoCo

Original: [37] physics simulator, inwhich we place a camera at the end of an invisible armature (the camera bot ), controlled by thecamera agent.
Revised: [37] physics simulator, in which we place a camera at the end of an invisible armature (the camera bot ), controlled by the camera agent.

Original: This shares the scene with a manipulator, and a block prop randomly positioned withinreach of the fingers, which gives the fingertip touch sensors something to touch.
Revised: This shares the scene with a manipulator, and a block prop randomly positioned within reach of the fingers, which gives the fingertip touch sensors something to touch.

Original: Both the camera botand manipulator are driven by velocity control, i.e. proportional-integral-derivative (PID) control inwhich the actions specify a target velocity for each actuator.
Revised: Both the camera bot and manipulator are driven by velocity control, i.e. proportional-integral-derivative (PID) control in which the actions specify a target velocity for each actuator.

Original: We treat the camera bot and manipulator as belonging to one robot that is conceptually split into twoseparate entities.
Revised: We treat the camera bot and manipulator as belonging to one robot that is conceptually split into two separate entities.

Original: The manipulator runs a fixed random behavioral policy throughout the experiment.
Revised: The manipulator runs a fixed random behavioral policy throughout the experiment.

Original: Sensors on the manipulator are available during training, to train the predictor.
Revised: Sensors on the manipulator are available during training, to train the predictor.

Original: The predictor servesas an endogenous reward function for simultaneously training the agent, which controls the camerabot.
Revised: The predictor serves as an endogenous reward function for simultaneously training the agent, which controls the camera bot.

Original: The training process only modifies the behavior of the camera bot, not the manipulator.
Revised: The training process only modifies the behavior of the camera bot, not the manipulator.

Original: The manipulator is a model of the 20-DOF tendon-driven hand by Shadow Robotics [32, 29].
Revised: The manipulator is a model of the 20-DOF tendon-driven hand by Shadow Robotics [32, 29].

Original: Werandomly drive the manipulator using Perlin noise [27].
Revised: We randomly drive the manipulator using Perlin noise [27].

Original: The camera bot has four DOFs: azimuth ,elevation , distance , and yaw .
Revised: The camera bot has four DOFs: azimuth , elevation , distance , and yaw .

Original: The first three position the camera in spherical coordinates aroundthe manipulator, and the last DOF rotates it in place.
Revised: The first three position the camera in spherical coordinates around the manipulator, and the last DOF rotates it in place.

Original: All DOFs are initialized randomly, which causesthe hand to be out of view half the time.
Revised: All DOFs are initialized randomly, which causes the hand to be out of view half the time.

Original: Cameras: The camera bot can be equipped with a conventional camera, or a foveal one [5, 12, 6, 7].
Revised: Cameras: The camera bot can be equipped with a conventional camera, or a foveal one [5, 12, 6, 7].

Original: Like Mnih et al.
Revised: Like Mnih et al.

Original: [22], we implement the foveal camera as N superimposed conventional cameraswith the same image dimensions, differing only in their field of view.
Revised: [22], we implement the foveal camera as N superimposed conventional cameras with the same image dimensions, differing only in their field of view.

Original: Fig.
Revised: Fig.

Original: 1b shows an exampleof the resulting images as seen by the agent, and fig.
Revised: 1b shows an example of the resulting images as seen by the agent, and fig.

Original: 1c visualizes their relative fields of view bystacking them on top of each other.
Revised: 1c visualizes their relative fields of view by stacking them on top of each other.

Original: In our experiments, the foveal camera has N = 5 cameras with21 × 21 pixels each.
Revised: In our experiments, the foveal camera has N = 5 cameras with 21 × 21 pixels each.

Original: Camera 1 has a FOV of 90 degrees in the vertical and horizontal directions,camera 2 has a FOV of 45 degrees, and so on, down to camera 5 with a FOV of 5 .
Revised: Camera 1 has a FOV of 90 degrees in the vertical and horizontal directions, camera 2 has a FOV of 45 degrees, and so on, down to camera 5 with a FOV of 5 .

Original: 26 degrees ( 90 / 2 4 ) .
Revised: 26 degrees ( 90 / 2 4 ) .

Original: The conventional camera is 21 × 21 pixels, with a FOV of 90 × 90 degrees.
Revised: The conventional camera is 21 × 21 pixels, with a FOV of 90 × 90 degrees.

Original: Predictor network: The predictor network shares no weights with the policy or critic networks in theagent.
Revised: Predictor network: The predictor network shares no weights with the policy or critic networks in the agent.

Original: It is a convolutional residual network (resnet) followed by a multi-layer perceptron (MLP).
Revised: It is a convolutional residual network (resnet) followed by a multi-layer perceptron (MLP).

Original: Itsarchitecture and layer sizes are taken from IMPALA’s
Revised: Its architecture and layer sizes are taken from IMPALA’s

Original: [9] convolutional residual network, replacingthe LSTM at the end with an MLP with hidden layer sizes (512 , 512 , 256) .
Revised: [9] convolutional residual network, replacing the LSTM at the end with an MLP with hidden layer sizes (512 , 512 , 256) .

Original: This predictor networktakes two consecutive images as input, and outputs predictions for all potential target sensors.
Revised: This predictor network takes two consecutive images as input, and outputs predictions for all potential target sensors.

Original: Likethe critic network, its predictions take the form of discrete distributions of estimated returns, of sensorvalues rather than rewards.
Revised: Like the critic network, its predictions take the form of discrete distributions of estimated returns, of sensor values rather than rewards.

Original: This was inspired by multi-timescale nexting [23], and the predictornetwork can be thought of as a multi-headed GVF [34] with distributional outputs.
Revised: This was inspired by multi-timescale nexting [23], and the predictor network can be thought of as a multi-headed GVF [34] with distributional outputs.

Original: The errors fromthese predictions can be thought of as a form of surprise, and driving behavior to maximize predictionerrors has been suggested as a form of curiosity [38].
Revised: The errors from these predictions can be thought of as a form of surprise, and driving behavior to maximize prediction errors has been suggested as a form of curiosity [38].

Original: In a sense, our agent does the opposite of this,as it is rewarded for minimizing error.
Revised: In a sense, our agent does the opposite of this, as it is rewarded for minimizing error.

Original: Instead of exploring by maximising short-term surprise, itexplores the state space through the diversity of  sensor targets
Revised: Instead of exploring by maximising short-term surprise, it explores the state space through the diversity of its sensor targets.

Revised: Agent networks: The agent consists of a policy network and critic network.

Revised: Like the predictor network, each of these is an MLP stacked on an IMPALA-style convolutional resnet.

Revised: The critic and policy networks share a resnet, but have different MLP heads, with hidden layer sizes (256 , 256 , 256)

Original: [Figure]
Revised: [Figure]

Original: and (512 , 512 , 256) , respectively.
Revised: and (512 , 512 , 256) , respectively.

Original: Both networks take a mix of images and other inputs.
Revised: Both networks take a mix of images and other inputs.

Original: The imagesare fed through the resnet, and the other inputs are concatenated with the result and fed to the MLP.
Revised: The images are fed through the resnet, and the other inputs are concatenated with the result and fed to the MLP.

Original: The policy network takes as input the observation (two consecutive image frames), and a one-hotvector specifying the target.
Revised: The policy network takes as input the observation (two consecutive image frames), and a one-hot vector specifying the target.

Original: It outputs bounded continuous target velocities for the camera bot’sfour joints (section 2).
Revised: It outputs bounded continuous target velocities for the camera bot’s four joints (section 2).

Original: The state-action critic additionally takes the action, and outputs a discretedistribution of the estimated return.
Revised: The state-action critic additionally takes the action, and outputs a discrete distribution of the estimated return.

Original: The agent’s objective is to minimize the predictor network’s errorfor the target sensor, by modifying the policy network to move the camera to look at the associatedbody part.
Revised: The agent’s objective is to minimize the predictor network’s error for the target sensor, by modifying the policy network to move the camera to look at the associated body part.

Original: 2.1 Training
Revised: 2.1 Training

Original: For each batch of transitions sampled from the replay buffer, we compute losses for the predictor,agent critic, and agent policy networks, then perform gradient updates on all three.
Revised: For each batch of transitions sampled from the replay buffer, we compute losses for the predictor, agent critic, and agent policy networks, then perform gradient updates on all three.

Original: To train the predictor, we compute the prediction error of sensor i using the distributional TDloss [2] from the distributed policy gradient method [1].
Revised: To train the predictor, we compute the prediction error of sensor i using the distributional TD loss [2] from the distributed policy gradient method [1].

Original: This is analogous to the standard TDerror ( δ = r t + γv t − v t − 1 ( x ) ), except that values v are represented not by scalars, but by discretedistributions over the range of possible return values.
Revised: This is analogous to the standard TD error ( δ = r t + γv t − v t − 1 ( x ) ), except that values v are represented not by scalars, but by discrete distributions over the range of possible return values.

Original: Instead of computing Q-values of reward, weestimate the expected return, v i , of sensor reading s it :
Revised: Instead of computing Q-values of reward, we estimate the expected return, v i , of sensor reading s it :

Original: [Equation]
Revised: [Equation]

Original: The predictor discount γ pred is separate from the discount γ used for training the critic.
Revised: The predictor discount γ pred is separate from the discount γ used for training the critic.

Original: Settingγ pred = 0 amounts to performing next-frame prediction, while setting it to larger values predictsits future sum over a decaying time window with half-life h =
Revised: Setting γ pred = 0 amounts to performing next-frame prediction, while setting it to larger values predicts its future sum over a decaying time window with half-life h =

Original: ∆ t ln (0 . 5) ln
Revised: ∆ t ln (0 . 5) ln

Original: ( γ pred ) .
Revised: ( γ pred ) .

Original: It is possible to predictover multiple time windows as in Horde [34], which may be useful in environments with predictabledynamics over multiple frames.
Revised: It is possible to predict over multiple time windows as in Horde [34], which may be useful in environments with predictable dynamics over multiple frames.

Original: For our environment, where the camera and manipulator have littlemomentum, we use decay γ pred chosen to have a short half-life of 0.1s.
Revised: For our environment, where the camera and manipulator have little momentum, we use decay γ pred chosen to have a short half-life of 0.1s.

Original: The predictor network’s lossis then the sum of prediction losses across all target sensors, L predt = (cid:80) i L it .
Revised: The predictor network’s loss is then the sum of prediction losses across all target sensors, L predt = (cid:80) i L it .

Original: The camera agent networks share no parameters with the predictor.
Revised: The camera agent networks share no parameters with the predictor.

Original: Each episode randomly chooses aproprioceptive sensor on the manipulator, to serve as the camera agent’s target for that episode.
Revised: Each episode randomly chooses a proprioceptive sensor on the manipulator, to serve as the camera agent’s target for that episode.

Original: Thecamera agent’s task is to position the camera in a manner that reduces the predictor’s error for thattarget sensor.
Revised: The camera agent’s task is to position the camera in a manner that reduces the predictor’s error for that target sensor.

Original: We therefore define the reward to be r t = − L targett , where L targett is the predictionerror for that episode’s target sensor, on timestep t .
Revised: We therefore define the reward to be r t = − L targett , where L targett is the prediction error for that episode’s target sensor, on timestep t .

Original: This is a dense reward, as it is available on everytimestep, but a noisy one, as it is learned.
Revised: This is a dense reward, as it is available on every timestep, but a noisy one, as it is learned.

Original: The agent critic loss is analogous to eq. 1, substituting r tfor sensor reading s it , γ for γ pred , and critic outputs q t and q t − 1 for predicted sensor returns v t andv t − 1 .
Revised: The agent critic loss is analogous to eq. 1, substituting r t for sensor reading s it , γ for γ pred , and critic outputs q t and q t − 1 for predicted sensor returns v t and v t − 1 .

Original: We train the policy network using the deterministic policy gradient loss [33].
Revised: We train the policy network using the deterministic policy gradient loss [33].

Original: Training takes 3 days to converge.
Revised: Training takes 3 days to converge.

Original: We generate data by running 512 instances of the simulator acrossas many machines, which feed training data to a single learner machine powered by a TPU.
Revised: We generate data by running 512 instances of the simulator across as many machines, which feed training data to a single learner machine powered by a TPU.

Original:  Experimental Results
Revised:  Experimental Results

Original: We present results from our trained agent, evaluated without the exploration noise used duringϵ -greedy training.
Revised: We present results from our trained agent, evaluated without the exploration noise used during ϵ -greedy training

Revised: [Table]

Original: 3.1 The trained camera policy improves target prediction accuracy
Revised: 3.1 The trained camera policy improves target prediction accuracy

Original: [Table]
Revised: [Table]

Revised: runs with different RNG seeds.

Original: Table 1 shows the target’s prediction error at the end of the episode (lower is better).
Revised: Table 1 shows the target’s prediction error at the end of the episode (lower is better).

Original: It comparestrained agents against several baselines.
Revised: It compares trained agents against the following baselines:

Revised: Blind:

Original: The Blind baseline is a predictor trained on a camera pointedaway from the hand.
Revised: The Blind baseline is a predictor trained on a camera pointed away from the hand.

Original: It can do no better than learn to output each sensor’s prior distribution, and itserves as an upper bound to the expected agent prediction error.
Revised: It can do no better than learn to output each sensor’s prior distribution, and serves as an upper bound to the expected agent prediction error.

Original:  For a lower bound from an oracle,we run a sweep over a series of hand-chosen fixed camera poses surrounding and looking at the hand,training a separate predictor for each pose.
Revised: Oracle : For a lower bound on the prediction error, we run a sweep over a series of hand-chosen fixed camera poses surrounding and looking at the hand, training a separate predictor for each pose.

Revised: The

Original: The Oracle entries show the minimum prediction errorover all viewpoints, for that target sensor.
Revised: Oracle entries show the minimum prediction error over all viewpoints for that target sensor.

Revised: The

Original: The Oracle benefits from not only specializing to a singlepoint of view, but also from not moving, which significantly reduces data variance, and improvesprediction error even in the moving-camera agent.
Revised: Oracle benefits from not only specializing to a single point of view, but also from not moving, which significantly reduces data variance and improves prediction error even in the moving-camera agent.

Original: As such, this fixed-pose Oracle is used as a lowerbound of expected agent prediction error.

Original: In practice, moving agents cannot always maintain a staticview, nor can a multi-camera agent  usually know a priori which camera will yield the most accuratepredictions.
Revised: In practice, moving agents cannot always maintain a static view, nor can an agent with multiple static cameras usually know a priori which camera will yield the most accurate predictions. Random :

Original: The Random agent is our agent with the policy and critic learning rates set to zero.
Revised: The Random agent is our agent with its policy and critic learning rates set to zero.

Original: Thecamera spawns randomly as usual, but hardly moves thereafter (brownian actions do little to move thecamera bot, which has high inertia), and the predictor must learn to predict from the resulting cameraviews.
Revised: The camera spawns randomly, as usual, but hardly moves thereafter (brownian actions do little to move the camera bot, which has high inertia). The predictor must learn to predict from the resulting random camera views.

Original: These views are mostly still, advantageously reducing input variance in a similar manner asthe still images of the Oracle.
Revised: These views are mostly static, advantageously reducing input variance in a similar manner to the still images of the Oracle.

Original: Nonetheless, our agent outperforms the Blind and Random baselinesby a statistically significant margin.
Revised: Our agent outperforms the Blind and Random baselines by a statistically significant margin.

Original: 3.2 The trained camera policy frames the subject
Revised: 3.2 The trained camera policy frames the subject

Original: A consistent outcome in our experiments is that camera agents learn to to place the target in aparticular position in the screen.
Revised: A consistent outcome in our experiments is that camera agents learn to to place the target in a particular position in the screen.

Original: This position can depend on the target, and varies from one trainingrun to another, though it is usually near the center (fig. 3a).
Revised: This position can depend on the target, and varies from one training run to another, though it is usually near the center (fig. 3a).

Original: This is not an instance of the cameraagent having memorized a particular set of preferred values for its own joint angles.
Revised: This is not an instance of the camera agent having memorized a particular set of preferred values for its own joint angles.

Original: The cameraagent is unaware of the camera bot’s joint angles, as the only inputs are pixels and the one-hotvector specifying the target sensor.
Revised: The camera agent is unaware of the camera bot’s joint angles, as the only inputs are pixels and the one-hot vector specifying the target sensor.

Original: Furthermore, it controls the camera bot joints by velocity control,whereas a position-controlled camera may learn to output a constant target camera pose withoutregard to input.
Revised: Furthermore, it controls the camera bot joints by velocity control, whereas a position-controlled camera may learn to output a constant target camera pose without regard to input.

Original: Figure 3b shows the emergence of this behavioral fovea in a conventional (non-foveal)camera.
Revised: Figure 3b shows the emergence of this behavioral fovea in a conventional (non-foveal) camera.

Original: Figure 3c show that agents with foveal cameras learn to frame the subject more precisely, inthe center.
Revised: Figure 3c show that agents with foveal cameras learn to frame the subject more precisely, in the center.

Revised: 3.3 The camera policy adopts distinct camera positions for different sensor targets

Revised: Section 3.2 showed that the policy learns to orient the camera to frame the target sensor at a specific image location.

Revised: While it is obviously important to look in the right direction, looking from the right

Original: Discussion
Revised: [Figure] [Figure] position also matters. Fig.

Revised: 4 shows the distribution of the camera position at the end of the episode for a single trained agent, showing a separate plot for each sensor target.

Revised: Figure 4a shows that the camera has learned to observe the wrist flexion in profile, i e., along the axis of rotation, from which the visual flow of flexing the wrist is most apparent.

Revised: It observes the wrist from one side or the other, hence the bimodal distribution seen in the red projection.

Revised: By contrast, observing the same wrist joint, but predicting its side-to-side deviation angle, behooves the agent to adopt a top-down view, as shown in fig.

Revised: 4b.

Revised: The differences in distribution for the camera positions when observing the finger root joints (fig. 4c, fig.

Revised: 4d are more subtle, observable in the blue projection along the floor.

Revised: Fig.

Revised: 4e shows the agent to be more position-agnostic when targeting the little finger’s touch sensor, due to its noisier predictions.

Revised: We discuss the reasons for this in section 5.

Revised: 3.4 The camera policy learns to circumnavigate occlusions

Revised: We introduced a randomized occluder to our training environment to test the generality of our selfsupervised training.

Revised: The occluder is a flat rectangle with random color, position, and dimensions.

Revised: We uniformly sampled its height and width from the range [0 . 5 , 1 . 0] , and sampled its position in spherical coordinates from azimuth ∼ [ − π, π ] , elevation ∼ [0 , π ] , distance ∼ [ . 4 , . 6] .

Revised: These spherical coordinates are roughly centered around the hand.

Revised: For reference, the size of the environment’s floor is 2 × 2 .

Revised: To maximize its effectiveness as an occluder, we orient the board to face the hand.

Revised: Fig.shows two trajectories of the camera agent, starting from the same initial conditions.

Revised: The resulting camera policy is able to sidestep the occluder when it blocks the camera’s view, thereafter exhibiting similar viewpoint preferences as in the unoccluded case.

Revised:  Discussion

Original: Many animals do not have a fovea.
Revised: Many animals do not have a fovea.

Original: Some, like rats, have a roughly even resolution across a nearlyspherical field of view.
Revised: Some, like rats, have a roughly even resolution across a nearly spherical field of view.

Original: When all directions are seen with equal acuity, one might ask if rats need tomove their eyes to look at subjects at all.
Revised: When all directions are seen with equal acuity, one might ask if rats need to [Figure] move their eyes to look at subjects at all.

Original: Yet they do, centering and fixating the subject in a specific[Figure] area of the retina [13].
Revised: Yet they do, centering and fixating the subject in a specific area of the retina [13].

Original: Like the rat, we find that our non-foveal agents exhibit framing behavior,positioning the subject in a specific area of the image, giving rise to a “behavioral fovea” even in theabsence of a physical one.
Revised: Like the rat, we find that our non-foveal agents exhibit framing behavior, positioning the subject in a specific area of the image, giving rise to a “behavioral fovea” even in the absence of a physical one.

Original: This suggests that the learning dynamics encourage a positive feedback loop between the predictorand agent: the predictor improves its expertise in a specific region of the field of vision, and the agentlearns to move the relevant subject into this visual region to receive the better prediction reward,and thus provides even more training data to further improve the predictor.
Revised: This suggests that the learning dynamics encourage a positive feedback loop between the predictor and agent: the predictor improves its expertise in a specific region of the field of vision, and the agent learns to move the relevant subject into this visual region to receive the better prediction reward, and thus provides even more training data to further improve the predictor.

Original: This behavior is incontrast to the usual emphasis placed on the data-hungry problem of learning position-invariant orposition-equivariant representations, a practice originating in computer vision research with datasetsof still images, rather than with an active camera [18].
Revised: This behavior is in contrast to the usual emphasis placed on the data-hungry problem of learning position-invariant or position-equivariant representations, a practice originating in computer vision research with datasets of still images, rather than with an active camera [18].

Original: That said, even static facial recognition hasbeen shown to benefit from normalizing the facial feature locations [36], a form of framing.
Revised: That said, even static facial recognition has been shown to benefit from normalizing the facial feature locations [36], a form of framing.

Original: Future work Giving our agent a foveal camera incentivizes it to position the subject more preciselyat the center of the image.
Revised: Future work Giving our agent a foveal camera incentivizes it to position the subject more precisely at the center of the image.

Original: One direction for future work is to investigate whether the agent canmaintain this centering as the target body part makes large movements, resulting in tracking behaviorfrom the camera as a side-effect of framing.
Revised: One direction for future work is to investigate whether the agent can maintain this centering as the target body part makes large movements, resulting in tracking behavior from the camera as a side-effect of framing.

Original: We demonstrate that a single agent can be trained tofixate on one of several targets, specified with a conditioning input.

Original: In this work, we limit the sensorsto a single hand, but a realistic body provides a rich variety of visual scales and distances, fromshoulders to toetips.
Revised: In this work, we limit the target sensors to a single hand, but a realistic body provides a rich variety of visual scales and distances, from shoulders to toetips.

Original: We posit that these present a means to learn a rich repertoire of visual fixationand tracking skills without the need to design their reward functions.
Revised: We posit that these present a means to learn a rich repertoire of visual fixation and tracking skills without the need to design their reward functions.

Original: These skills could serve as abasis for exploration while learning higher-level skills, as demonstrated with SAC-X [31], whichused manually designed reward functions.
Revised: These skills could serve as a basis for exploration while learning higher-level skills, as demonstrated with SAC-X [31], which used manually designed reward functions.

Original: Parisi et al.
Revised: Parisi et al.

Original: [26] show that visual RL agents benefit frompretraining on static image datasets.
Revised: [26] show that visual RL agents benefit from pretraining on static image datasets.

Original: Self-supervised RL presents a means of feature learning that is,in a lifelong learning sense, more natural than using human-annotated image datasets.
Revised: Self-supervised RL presents a means of feature learning that is, in a lifelong learning sense, more natural than using human-annotated image datasets.

Original: Limitations For N targets, the time spent training to target any one of them is 1/N of the total trainingtime, leading to poor scaling of the needed total training time as N increases.
Revised: Limitations For N targets, the time spent training to target any one of them is 1/N of the total training time, leading to poor scaling of the needed total training time as N increases.

Original: We plan to investigatetarget relabeling to mitigate this.
Revised: We plan to investigate target relabeling to mitigate this.

Original: The manipulator’s random policy leads to infrequent touch sensoractivations, leading to a label imbalance for the touch predictor that makes targeting touch sensorsmore difficult to learn than with other sensor types.
Revised: The manipulator’s random policy leads to infrequent touch sensor activations, leading to a label imbalance for the touch predictor that makes targeting touch sensors more difficult to learn than with other sensor types.

Original: We plan to address this label imbalance bylearning a policy for the hand that maximizes the target sensor’s entropy.
Revised: We plan to address this label imbalance by learning a policy for the hand that maximizes the target sensor’s entropy.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper we demonstrate a self-supervised means of learning to fixate a camera at areas that arepredictive to a quantity of interest, such as a sensor reading.
Revised: In this paper we demonstrate a self-supervised means of learning to fixate a camera at areas that are predictive to a quantity of interest, such as a sensor reading.

Original: We do so in the context of a simulatedrobot, learning to fixate on various parts of its own body by predicting its own sensors.
Revised: We do so in the context of a simulated robot, learning to fixate on various parts of its own body by predicting its own sensors.

Original: The learnedpolicies are performant relative to baselines,  and precise.
Revised: The learned policies are performant relative to baselines, precise, and able to circumnavigate occlusions.

Original: The precision increases when the agent isprovided with a foveal camera.Checklist
Revised: The precision increases when the agent is provided with a foveal camera.

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Section 2.2.•

Original: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] Our main claims are that the policies learned throughthe proposed self-prediction reward are performant relative to baselines, which weshow in section 3.1 and precise, which we show in section 3.2.(b)

Original: Did you describe the limitations of your work?

Original: [Yes] See section 4(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [No] The workdescribes a general method of self-supervised learning of visual framing behavior, anddoes not pose specific negative societal impacts in excess of the standard of generalrobotics research.(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)

Original: Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce

Original: the main experimental results (either in the supplemental material or as a URL)?

Original: [No] If accepted, wewill provide more detail to the specifics of the training environment, as an appendix.

Original: (b)

Original: Did you specify

Original: all

Original: the training

Original: details (e

Original: .g.

Original: data splits, hyperparameters, how they were
Revised: Acknowledgments and Disclosure of Funding

Original: chosen)?

Original: [No] If accepted, we will provide more detail to the training hyperparameters,

Original: as an appendix

Original: c)

Original: Did you report error bars (e.g

Original: with

Original: respect to the random seed after running experiments multiple times)?

Original: [Yes] Table 1 contains confidence bounds computed from 10training runs with different RNG seeds. Figure 3 shows red squares covering 2.5% to97% of the cumulative probability distribution over each axis of the histogram. d ) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Funding by DeepMind.

Original: [Yes] See the end of section 2.1.

Original: Ifaccepted, we will include more detail in an appendix.4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: Special thanks to Bobak Shahriari, Matt Hoffman, and Piotr Trochim for technical help, Adam Kosiorek, Patrick Pilarski, and Nicolas Heess for internal review, and Joel

Original: [Yes] We cite the creatorsof the MuJoCo simulator [37], and the Shadow Hand [32] (although we only use amodel of it).

Original: (b) Did you mention the license of the assets?

Original: [Yes] We mention MuJoCo’s license at the

Original: top

Original: of

Original: section

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e)

Original: Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: [N/A]5. If you used crowdsourcing or conducted research with human subjects... (a) Did you
Revised: Leibo and Kimberly Stachenfeld for neuroscience consultation.

Original: include the full text

Original: of instructions given to participants and screenshots ifapplicable

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]