Original: Aligning an optical interferometer with beam divergence control and continuous action space
Revised: Aligning an optical interferometer with beam divergence control and continuous action space

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract:
Revised: Abstract:

Original: Reinforcement learning is ﬁnding its way to real-world problem application, transferring from simulated environments to physical setups.
Revised: Reinforcement learning is ﬁnding its way to real-world problem application, transferring from simulated environments to physical setups.

Original: In this work,we implement vision-based alignment of an optical Mach-Zehnder interferometerwith a confocal telescope in one arm, which controls the diameter and divergenceof the corresponding beam.
Revised: In this work,we implement vision-based alignment of an optical Mach-Zehnder interferometerwith a confocal telescope in one arm, which controls the diameter and divergenceof the corresponding beam.

Original: We use a continuous action space; exponential scalingenables us to handle actions within a range of over two orders of magnitude.
Revised: We use a continuous action space; exponential scalingenables us to handle actions within a range of over two orders of magnitude.

Original: Ouragent trains only in a simulated environment with domain randomizations.
Revised: Ouragent trains only in a simulated environment with domain randomizations.

Original: In anexperimental evaluation, the agent signiﬁcantly outperforms an existing solutionand a human expert.
Revised: In anexperimental evaluation, the agent signiﬁcantly outperforms an existing solutionand a human expert.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) demonstrates incredible success in simulated environments, surpassinga human expert in Atari [1], chess, shogi and Go [2], as well as in more complicated games such as
Revised: Reinforcement learning (RL) demonstrates incredible success in simulated environments, surpassinga human expert in Atari [1], chess, shogi and Go [2], as well as in more complicated games such as

Original: Dota2 [3] and StarCraft [4].
Revised: Dota2 [3] and StarCraft [4].

Original: In robotics, reinforcement learning shows remarkable results in pushing[5], grasping [6] and stacking objects [7].
Revised: In robotics, reinforcement learning shows remarkable results in pushing[5], grasping [6] and stacking objects [7].

Original: RL enables robots to walk, overcoming different obstacles[8] and learn agile animal skills [9].
Revised: RL enables robots to walk, overcoming different obstacles[8] and learn agile animal skills [9].

Original: However, RL agents are not yet widely used in real roboticsand can hardly be compared to humans in a physical environment.
Revised: However, RL agents are not yet widely used in real roboticsand can hardly be compared to humans in a physical environment.

Original: The principal problems of real-world applications of RL are nonstationarity and stochasticity of thephysical environment, complexity and time intensiveness of data acquisition as well as unsafety oftraining and evaluation of policies on real robots [10].
Revised: The principal problems of real-world applications of RL are nonstationarity and stochasticity of thephysical environment, complexity and time intensiveness of data acquisition as well as unsafety oftraining and evaluation of policies on real robots [10].

Original: A common method to handle these challengesis to train an agent in a high-ﬁdelity simulation (source domain) of the real environment (targetdomain).
Revised: A common method to handle these challengesis to train an agent in a high-ﬁdelity simulation (source domain) of the real environment (targetdomain).

Original: An agent can then be transferred to a physical robot.
Revised: An agent can then be transferred to a physical robot.

Original: The main limitation of this approachis inevitable discrepancy between the source and target domains that lead to performance loss aftertransfer.
Revised: The main limitation of this approachis inevitable discrepancy between the source and target domains that lead to performance loss aftertransfer.

Original: This limitation can be mitigated with  randomization of simulator parameters [5, 9, 11].
Revised: This limitation can be mitigated with the randomization of simulator parameters [5, 9, 11].

Original: A promising application domain for smart robotics is experimental optics.
Revised: A promising application domain for smart robotics is experimental optics.

Original: Optical physicists workwith installations consisting of hundreds or thousands of elements.
Revised: Optical physicists workwith installations consisting of hundreds or thousands of elements.

Original: These setups need to be thoroughly aligned before measurements can be conducted.
Revised: These setups need to be thoroughly aligned before measurements can be conducted.

Original: Dependent on the setup size, such a setupcan require several hours of work by a group of experts on a daily basis.
Revised: Dependent on the setup size, such a setupcan require several hours of work by a group of experts on a daily basis.

Original: The automation of thisprocedure could drastically enhance the productivity of research groups.
Revised: The automation of thisprocedure could drastically enhance the productivity of research groups.

Original: Since the alignment is aniterative decision-making process with a well-deﬁned reward function, it is natural to formulate andsolve it in terms of reinforcement learning.
Revised: Since the alignment is aniterative decision-making process with a well-deﬁned reward function, it is natural to formulate andsolve it in terms of reinforcement learning.

Original: Robotic alignment ofan optical Mach-Zehnder interferometer, which is a primary component ofmany tabletop optical setups, was studied in [12]. The task of interferometer alignment consistsin matching the optical modes of the electromagnetic waves inits two arms so that they exhibita visibility of unity when interfering with each other. 
Revised: The alignment of a large optical setup can typically be separated into modular tasks which can becompleted in sequence. A common modular task is the alignment of a Mach-Zehnder interferometer(MZI), which consists in matching the optical modes of the electromagnetic waves in two paths sothat they exhibit a visibility of unity when interfering with each other. The robotization of MZI alignment was studied by Sorokin et al.

Original: The authors treated this problem  as a partially observable Markov decision process, with the observation being a sequence of interferencepatterns observed on a camera as the relative phase of the two arms is being varied.
Revised: [12], where this problem was treated as a partially observable Markov decision process, with the observation being a sequence of interference patterns observedon a camera as the relative phase of the two arms is being varied.

Original: They traineda discrete-action (VD D3QN [13]) agent, dubbed Interferobot , in a simulated environment with domain randomizations and successfully transferred the trained agent to a real setup.
Revised: They trained a discrete-action(VD D3QN [13]) agent, dubbed Interferobot , in a simulated environment with domain random- izations and successfully transferred the trained agent to a real setup.

Original: The trained agentachieved superhuman-level performance in terms of speed and quality of alignment.
Revised: The trained agent achievedsuperhuman-level performance in terms of speed and quality of alignment.

Original: However, the task of Ref.
Revised: However, the task of Ref.

Original: [12] was limited to matching the spatial positions and directions of thetwo otherwise identical beams.
Revised: [12] was limited to matching the spatial positions and directions of thetwo otherwise identical beams.

Original: It is much simpler than those alignment tasks that occur in practice,which additionally require matching the geometric sizes of the beams, their divergence, and, in thecase of pulsed lasers, arrival time.
Revised: It is much simpler than those alignment tasks that occur in practice,which additionally require matching the geometric sizes of the beams, their divergence, and, inthe case of pulsed lasers, arrival time.

Original: The original Interferobot suffers from a number of limitations thatset these tasks off limits to  this agent. First, the number of discrete actions grows exponentially withthe number of degrees of freedom [14].
Revised: These additional features of the experiment result in morecomplex interference patterns (a richer observation space in RL parlance), hence requiring a moresophisticated agent for their interpretation. The main contribution of our work is to develop a new agent for interferometer alignment, whichaddresses this challenge.

Original: In Ref. [12], this issue was circumvented by selecting andoperating only one of the controls at each moment in time; however, such an approach signiﬁcantlylengthens the alignment procedure.
Revised: Our solution is based on a different RL algorithm and features a number ofimportant innovations.

Original: Second, the actions associated with each control span multipleorders of magnitude: rough adjustments in the beginning, followed by ﬁner and ﬁner steps as thealignment progresses.
Revised: First, the action space was changed from discrete to continuous, which alsochanged the policy task from classiﬁcation to regression.

Original: Implementing this in a discrete action space complicates the heuristic handpicking of the appropriate action sizes and greatly enlarges the action space.
Revised: This enabled us to solve the issue arisingin the discrete action space, consisting in the number of actions growing exponentially with thenumber of degrees of freedom [14].

Original: Third, the interferencepatterns obtained from beams with different sizes and divergences are more difﬁcult to interpret thanthose exhibited by identical beams, so an agent would require a more complex neural network. The main contributions of our work are as follows . We add a system of two optical lensesin one of the interferometer arms, which leads to richer state and observation spaces that representreal optical experiments more closely.
Revised: However, it raises a new concern: the agent’s actions towardsthe end of the alignment process need to be much ﬁner than in the beginning; the action space spansover two orders of magnitude.

Original: We demonstrate that an optical interferometer can be successfully aligned by an agent with a continuous action space. The agent trained fully in a simulatedenvironment can easily leverage multidimensional actions on a real setup. We use exponentialaction rescaling which allows the agent to effectively explore actions of different magnitudes. 4)
Revised: We address this with our second innovation — exponential actionrescaling, which allows the agent to effectively explore actions of different magnitudes. Third, bygranting the agent a continuous action space, we run a risk to leave the safe space of experimentalparameters, which may even result in damaging the equipment.

Revised: We solve this issue by introducingpenalties that discourage the policy from approaching boundaries.

Original: We enhance the set of domain randomizations that let our agent perform better in the sim-to-realtransfer.
Revised: Fourth, we enhance the set ofdomain randomizations that help our agent perform better in the sim-to-real transfer.

Original: Our agent signiﬁcantly outperforms the original Interferobot and a human expert in termsof both time and quality of the alignment.
Revised: Our resultingagent signiﬁcantly outperforms the original Interferobot and a human expert in terms of both timeand quality of the alignment.

Original:  Related works
Revised:  Related works

Original: RL agents for robotics can be trained either on real-world data or in simulation.
Revised: RL agents for robotics can be trained either on real-world data or in simulation.

Original: The advantage of theformer is that the agent receives hands-on experience of the environment in which it will be tested.
Revised: The advantage of theformer is that the agent receives hands-on experience of the environment in which it will be tested.

Original: The shortcoming is that the acquisition of a large dataset required for training is a complicated andtime-consuming process.
Revised: The shortcoming is that the acquisition of a large dataset required for training is a complicated andtime-consuming process.

Original: This is especially the case when the observations are vision-based.
Revised: This is especially the case when the observations are vision-based.

Original: Forexample, the training of a grasping agent in Kalashnikov et al.
Revised: Forexample, the training of a grasping agent in Kalashnikov et al.

Original: [6] required several weeks, in spiteof parallelization across 7 identical robots.
Revised: [6] required several weeks, in spiteof parallelization across 7 identical robots.

Original: This complication can be addressed with the help ofbehavior cloning: Vecerik et al.
Revised: This complication can be addressed with the help ofbehavior cloning: Vecerik et al.

Original: [15] trained an agent solving the insertion task using raw visualimages using dozens of expert successful and failed trajectories.
Revised: [15] trained an agent solving the insertion task using raw visualimages using dozens of expert successful and failed trajectories.

Original: Hands-on training is simpliﬁed foragents without visual observations.
Revised: Hands-on training is simpliﬁed foragents without visual observations.

Original: For example, Haarnoja et al.
Revised: For example, Haarnoja et al.

Original: [8] developed a quadruped walkingrobot that can generalize to unseen terrains with the training requiring as little as two hours.
Revised: [8] developed a quadruped walkingrobot that can generalize to unseen terrains with the training requiring as little as two hours.

Original: Anotherapproach for training robot locomotion was presented by Yang et al.
Revised: Anotherapproach for training robot locomotion was presented by Yang et al.

Original: [16], who collected 4.5 minutesworth of data from a simple quadrupedal robot to model the robot’s dynamics and used this modelto real-time action planning.
Revised: [16], who collected 4.5 minutesworth of data from a simple quadrupedal robot to model the robot’s dynamics and used this modelto real-time action planning.

Original: An alternative approach is to train the agent in simulation and transfer it to a real-world system afterwards.
Revised: An alternative approach is to train the agent in simulation and transfer it to a real-world system afterwards.

Original: To eliminate the discrepancy between the real and simulated environments, Tobin et al.
Revised: To eliminate the discrepancy between the real and simulated environments, Tobin et al.

Original: [11]proposed a simple technique called domain randomization: they trained an object detection model onsimulated images with different textures, lights, object and camera positions, and demonstrated thatsuch a model achieves high accuracy in the real world.
Revised: [11]proposed a simple technique called domain randomization: they trained an object detection model onsimulated images with different textures, lights, object and camera positions, and demonstrated thatsuch a model achieves high accuracy in the real world.

Original: Peng et al.
Revised: Peng et al.

Original: [5] trained a robotic hand to pushan object using randomization of physical parameters such as friction, mass, damping, etc.
Revised: [5] trained a robotic hand to pushan object using randomization of physical parameters such as friction, mass, damping, etc.

Original: Anotherexample is the vision-based task of dexterous in-hand manipulation [17], which demonstrated theeffectiveness of applying randomizations to both the physical parameters and image observations.
Revised: Anotherexample is the vision-based task of dexterous in-hand manipulation [17], which demonstrated theeffectiveness of applying randomizations to both the physical parameters and image observations.

Original: The automation of optical system alignment is constantly evolving.
Revised: The automation of optical system alignment is constantly evolving.

Original: In 1987 Gabler et al.
Revised: In 1987 Gabler et al.

Original: [18]demonstrated automated alignment of an optical ﬁbre used an iterative algorithm that consequentlyﬁnds the maximum of a photo-detector signal with respect to each of the three ﬁbre movement axis.
Revised: [18]demonstrated automated alignment of an optical ﬁbre used an iterative algorithm that consequentlyﬁnds the maximum of a photo-detector signal with respect to each of the three ﬁbre movement axis.

Original: Fang and Savransky [19] considered automated alignment of a system of two lenses with 8 degreesof freedom.
Revised: Fang and Savransky [19] considered automated alignment of a system of two lenses with 8 degreesof freedom.

Original: Acquiring a focal plane image, their algorithm performs principal component analysisand Kalman ﬁltering to calculate proper control inputs.
Revised: Acquiring a focal plane image, their algorithm performs principal component analysisand Kalman ﬁltering to calculate proper control inputs.

Original: Deep machine learning, particularly RL, becomes increasingly popular in experimental optics.
Revised: Deep machine learning, particularly RL, has become increasingly popular in experimental optics.

Original: Forexample, Sun et al.
Revised: For example, Sun et al.

Original: [20] used an RL agent to stabilize a mode-locked laser by controlling waveplatesand polarizers.
Revised: [20] used an RL agent to stabilize a mode-locked laser by controlling wave- plates and polarizers.

Original: RL algorithms are also routinely used to optimize optical communications [21],e.g.
Revised: RL algorithms are also routinely used to optimize optical communications [Figure]

Original:  to route trafﬁc in optical transport networks [22, 23].
Revised: [21], e.g. to route trafﬁc in optical transport networks [22, 23].

Original: [Figure]
Revised: [Figure]

Original: Interference is a physical phenomenon that results from coherent addition of amplitudes of two ormore overlapping waves; the resulting amplitude depends on the relative phase of the componentwaves.
Revised: Interference is a physical phenomenon that results from coherent addition of amplitudes of two ormore overlapping waves; the resulting amplitude depends on the relative phase of the componentwaves.

Original: Interferometers, which are among the main instruments of experimental optics, use interference to precisely measure this phase difference.
Revised: Interferometers, which are among the main instruments of experimental optics, use interference to precisely measure this phase difference.

Original: In this paper we consider a Mach-Zehnder interferometer (MZI) displayed in Fig.
Revised: In this paper we consider an MZI displayed in Fig.

Original: A collimated(parallel) laser beam is divided by a beam splitter (BS 1).
Revised: A collimated (parallel) laser beam is dividedby a beam splitter (BS 1).

Original: The two resulting beams propagate throughdifferent paths before being recombined, with the help of steering mirrors, by another beam splitter(BS 2) and viewed by a camera.
Revised: The two resulting beams propagate through different paths before beingrecombined, with the help of steering mirrors, by another beam splitter (BS 2) and viewed by a camera.

Original: One of the mirrors is mounted on a piezoelectric transducer to varythe relative phase ϕ of the beams.
Revised: One of the mirrors is mounted on a piezoelectric transducer to vary the relative phase ϕ of thebeams.

Original: One of the beam paths (the lower one in Fig. 1) contains a systemof two lenses (telescope), which control the beam divergence and size.
Revised: One of the beam paths (the lower one in Fig. 1) contains a system of two lenses (telescope),which control the beam divergence and size.

Original: We use mororized mounts formirror 2 and BS 2; lens 2 is mounted on a motorized translation stage.
Revised: We use motorized mounts for mirror 2 and BS 2; lens 2is mounted on a motorized translation stage.

Original: Aligning the interferometerrequired precise matching of the two beams in terms of their transverse positions, directions, radiiand divergences.
Revised: Aligning the interferometer requires precise matchingof the two beams in terms of their transverse positions, directions, radii and divergences.

Original: The images from the camera serve as observations for our RL agent.
Revised: The images from the camera serve as observations for our RL agent.

Original: Examples of acquired imagesare shown in Fig.
Revised: Examples of acquired imagesare shown in Fig.

Original: The curved shape of the interference fringes is a consequence of differentdivergences of the two beams.
Revised: The curved shape of the interference fringes is a consequence of differentdivergences of the two beams.

Original: We apply an asymmetric sawtooth voltage pattern to the piezo,akin to Ref.
Revised: We apply an asymmetric sawtooth voltage pattern to the piezo,akin to Ref.

Original: [12], to view the temporal dynamics of fringes.
Revised: [12], to view the temporal dynamics of fringes.

Original: The amplitude of the piezo motioncorresponds to a phase difference of about 2 π .
Revised: The amplitude of the piezo motioncorresponds to a phase difference of about 2 π .

Original: For a misaligned interferometer, the piezo motioncauses transverse displacement of the fringes visible in Fig.
Revised: For a misaligned interferometer, the piezo motioncauses transverse displacement of the fringes visible in Fig.

Original: 2 (b-d), which permit the agent toextract the information about the sign of the difference of the beams’ angles.
Revised: 2 (b-d), which permit the agent toextract the information about the sign of the difference of the beams’ angles.

Original: When fully aligned,the interference will appear as a single blinking spot [Fig.
Revised: When fully aligned,the interference will appear as a single blinking spot [Fig.

Original: 2 (a)].
Revised: 2 (a)].

Original: The complexity of alignment arises because changing the angular orientations of the mirrors simultaneously affects the position and angle of the lower beam, while the lens movement changes thelower beam radius and divergence.
Revised: The complexity of alignment arises because changing the angular orientations of the mirrors simultaneously affects the position and angle of the lower beam, while the lens movement changes thelower beam radius and divergence.

Original: Moreover, images received from the camera can be corrupted bynoise, aberrations, ambient light and dust.
Revised: Moreover, images received from the camera can be corrupted bynoise, aberrations, ambient light and dust.

Original: The problem is additionally complicated by the inaccuracy of mounts and positioners whose relative action noise is about 4% .
Revised: The problem is additionally complicated by the inaccuracy of mounts and positioners whose relative action noise is about 4% .

Original: The position-dependent intensity of the interference pattern can be written as I ( x, y, ϕ ) =
Revised: The position-dependent intensity of the interference pattern can be written as I ( x, y, ϕ ) =

Original: 12 | E upper ( x, y ) e iϕ + E lower ( x, y )
Revised: 12 | E upper ( x, y ) e iϕ + E lower ( x, y )

Original: | 2 , where E upper / lower ( x, y ) is the amplitude of each beam.
Revised: | 2 , where E upper / lower ( x, y ) is the amplitude of each beam.

Original: Thequality metric of alignment, visibility , is deﬁned as
Revised: Thequality metric of alignment, visibility , is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where I tot ( ϕ ) =
Revised: where I tot ( ϕ ) =

Original: (cid:82)(cid:82) I
Revised: (cid:82)(cid:82) I

Original: ( x, y, ϕ ) dxdy is a total light intensity in the image plane with the phasedifference ϕ between the arms.
Revised: ( x, y, ϕ ) dxdy is a total light intensity in the image plane with the phasedifference ϕ between the arms.

Original: The intensity depends on time because of periodic piezo movementthat varies the phase difference between the two interferometer arms.
Revised: The intensity depends on time because of periodic piezo movementthat varies the phase difference between the two interferometer arms.

Original: The visibility lies in [0 , 1]range, where V = 1 corresponds to a perfectly aligned interferometer.
Revised: The visibility lies in [0 , 1]range, where V = 1 corresponds to a perfectly aligned interferometer.

Original: [Figure]
Revised: [Figure]

Original:  Background
Revised:  Background

Original: [Figure]
Revised: [Figure]

Original: We consider a standard partially observable Markov decision process (POMDP) setting where thepurpose of an agent is to maximize its cumulative reward during the policy execution.
Revised: We consider a standard partially observable Markov decision process (POMDP) setting where thepurpose of an agent is to maximize its cumulative reward during the policy execution.

Original: The observation of the environment at timestep t is deﬁned as o t ∈ O and is sampled from the distributiono t ∼ U ( o t | s t ) , where s t ∈ S is the state.
Revised: The observation of the environment at timestep t is deﬁned as o t ∈ O and is sampled from the distributiono t ∼ U ( o t | s t ) , where s t ∈ S is the state.

Original: The policy of our agent is a deterministic functiona t =
Revised: The policy of our agent is a deterministic functiona t =

Original: π ( o t ) yielding an action, which is an element of the action space A .
Revised: π ( o t ) yielding an action, which is an element of the action space A .

Original: Following the action, theagent receives a reward r t : S ×
Revised: Following the action, theagent receives a reward r t : S ×

Original: A → R and the next observation
Revised: A → R and the next observation

Original: o t +1 ∼ U ( o t +1 | s t +1 ) , wheres t +1 is produced according to a latent transition distribution s t +1 ∼ F ( s t +1 | s t , a t ) .
Revised: o t +1 ∼ U ( o t +1 | s t +1 ) , wheres t +1 is produced according to a latent transition distribution s t +1 ∼ F ( s t +1 | s t , a t ) .

Original: The discountedreward sum (return) from timestep t in each episode is R t = (cid:80)
Revised: The discountedreward sum (return) from timestep t in each episode is R t = (cid:80)

Original: Ti = t γ i − t r i , where γ ∈ [0 , 1] is thediscount factor and T is the horizon of the episode.
Revised: Ti = t γ i − t r i , where γ ∈ [0 , 1] is thediscount factor and T is the horizon of the episode.

Original: The agent’s objective is to learn the policy π ∗ that maximizes the expected return J ( π ) =
Revised: The agent’s objective is to learn the policy π ∗ that maximizes the expected return J ( π ) =

Original: E τ ∼ p ( τ | π ) [ R 0 ] where p ( τ | π ) is the distribution of trajectories τ
Revised: E τ ∼ p ( τ | π ) [ R 0 ] where p ( τ | π ) is the distribution of trajectories τ

Original: =
Revised: =

Original: ( o 0 , a 0 , o 1 , a 1 , ..., a T − 1 , o T )produced by policy π : 
Revised: ( o 0 , a 0 , o 1 , a 1 , ..., a T − 1 , o T )produced by policy π : [Equation] = [Equation]

Original: [Equation]
Revised: [Equation]

Original: Policy gradient.
Revised: Policy gradient.

Original: The policy π is deﬁned by parameters θ (i.e. π = π θ ), so the optimization (2)is performed with respect to the parameters: θ ∗ = arg max θ J ( π θ ) .
Revised: The policy π is deﬁned by parameters θ (i.e. π = π θ ), so the optimization (2)is performed with respect to the parameters: θ ∗ = arg max θ J ( π θ ) .

Original: A common approach to sucha problem is policy gradient [24], which iteratively improves the policy in terms of expected returnvia gradient ascent:
Revised: A common approach to sucha problem is policy gradient [24], which iteratively improves the policy in terms of expected returnvia gradient ascent:

Original: [Equation]
Revised: [Equation]

Original: where α is the learning rate.
Revised: where α is the learning rate.

Original: The algorithms of this family vary by the method of approximating theunknown J ( π θ ) .
Revised: The algorithms of this family vary by the method of approximating theunknown J ( π θ ) .

Original: A common approach to continuous-action Markov decision processes is TD3 [25] which is an extension of another popular algorithm DDPG [26].
Revised: A common approach to continuous-action Markov decision processes is TD3 [25] which is an extension of another popular algorithm DDPG [26].

Original: This algorithm uses three neural networks: one(actor) for a deterministic policy and two (critics) for evaluating the action-state values Q ( s t , a t ) .
Revised: This algorithm uses three neural networks: one (actor) for a deterministic policy and two (critics) for evaluating the action-state values Q ( s t , a t ) .

Original: Using two critics allows the algorithm to suffer less from overestimating the Q-values; in addition,the algorithm smoothens the Q-functions by adding Gaussian noise to the target actions when updating the parameters of the critics.
Revised: Using two critics allows the algorithm to suffer less from overestimating the Q-values; in addition,the algorithm smoothens the Q-functions by adding Gaussian noise to the target actions when updating the parameters of the critics.

Original: Gaussian noise is also added to each policy action during thetraining to encourage exploration.
Revised: Gaussian noise is also added to each policy action during thetraining to encourage exploration.

Original: Domain randomization.
Revised: Domain randomization.

Original: To lower the performance losses associated with the transferring of theagent from a simulated to real-world environment, domain randomization is used.
Revised: To lower the performance losses associated with the transferring of theagent from a simulated to real-world environment, domain randomization is used.

Original: The agent istrained for a set of tasks with different dynamics ˆ F ( s t | s t − 1 , a t − 1 , µ sim )
Revised: The agent istrained for a set of tasks with different dynamics ˆ F ( s t | s t − 1 , a t − 1 , µ sim )

Original: ≈ F ( s t | s t − 1 , a t − 1 , µ real )and sensor noise models ˆ U (
Revised: ≈ F ( s t | s t − 1 , a t − 1 , µ real )and sensor noise models ˆ U (

Original: o t | s t , µ sim )
Revised: o t | s t , µ sim )

Original: ≈ U ( o t | s t , µ real ) where µ sim / real are sets of environmentparameters.
Revised: ≈ U ( o t | s t , µ real ) where µ sim / real are sets of environmentparameters.

Original: Whereas we do not know the actual µ real of the real environment, varying µ sim in arange containing µ real helps to improve generalization and transfer quality.
Revised: Whereas we do not know the actual µ real of the real environment, varying µ sim in arange containing µ real helps to improve generalization and transfer quality.

Original:  Our method
Revised:  Our method

Original: In this work, we apply continuous control reinforcement learning methods to align the MZI.
Revised: In this work, we apply continuous control reinforcement learning methods to align the MZI.

Original: Asmentioned above, the alignment is performed step by step and the actions depend on the interferencepatterns observed by the agent, so it can be naturally viewed as a POMDP.
Revised: Asmentioned above, the alignment is performed step by step and the actions depend on the interferencepatterns observed by the agent, so it can be naturally viewed as a POMDP.

Revised: The agent is trained in simulation and evaluated on a real interferometer.

Revised: The training algorithm islisted in Appendix A and the MZI simulator is described in Appendix B.

Original: State, observation and action.
Revised: State, observation and action.

Original: The state of the environment is a vector consisting of the transverseposition ( x, y ) , direction angle ( α x , α y ) , radius r lower and divergence curvature ρ of the lower beamin the plane of the camera.
Revised: The state of the environment is a vector consisting of the transverseposition ( x, y ) , direction angle ( α x , α y ) , radius r lower and divergence curvature ρ of the lower beamin the plane of the camera.

Original: The position and transverse direction of the upper beam are assumedto be zero.
Revised: The position and transverse direction of the upper beam are assumedto be zero.

Original: This state is fully determined by the angles of the mirror and the beam splitter and theposition of the lens.
Revised: This state is fully determined by the angles of the mirror and the beam splitter and theposition of the lens.

Original: The range of allowed states, listed in Table 1, is restricted by the requirementthat the beam remains visible on the camera and the position of the lens stays within the travel rangeof the translation stage.
Revised: The range of allowed states, listed in Table 1, is restricted by the requirementthat the beam remains visible on the camera and the position of the lens stays within the travel rangeof the translation stage.

Original: The observation is a set of 16 consecutive 64 × 64 images acquired by the camera during oneperiod of the piezo mirror.
Revised: The observation is a set of 16 consecutive 64 × 64 images acquired by the camera during oneperiod of the piezo mirror.

Original: The actions are ﬁve-dimensional vectors specifying the relative angulardeﬂections of both mirrors along the x and y axes and the linear displacement of the lens with respectto their current positions.
Revised: The actions are ﬁve-dimensional vectors specifying the relative angulardeﬂections of both mirrors along the x and y axes and the linear displacement of the lens with respectto their current positions.

Original: Each action vector component lies in the [ − 1 , 1] range, where the valuesof ± 1 correspond to the maximum and minimum values as listed in Table 1.
Revised: Each action vector component lies in the [ − 1 , 1] range, where the valuesof ± 1 correspond to the maximum and minimum values as listed in Table 1.

Original: The absolute valuesof each action are restricted to the interval [2 . 5 · 10 − 3 , 1] , because smaller actions do not produceobservable changes in the interference pattern and fall within the uncertainty range of motorizedmounts.
Revised: The absolute valuesof each action are restricted to the interval [2 . 5 · 10 − 3 , 1] , because smaller actions do not produceobservable changes in the interference pattern and fall within the uncertainty range of motorizedmounts.

Original: Episode and reset.
Revised: Episode and reset.

Original: The agent is trained in simulation and evaluated on a real interferometer.

Original: Tocompare the performance of our agent with that of Sorokin et al.
Revised: To compare the performance of our agent with that of Sorokin et al.

Original: [12], we keep the episode lengthequal to 100 actions.
Revised: [12], we keep the episode lengthequal to 100 actions.

Original: At the beginning of each episode, the beam in the lower path is misaligned bysetting the mirror angles and lens position to random values within the allowed range.
Revised: At the beginning of each episode, the beam in the lower path is misaligned bysetting the mirror angles and lens position to random values within the allowed range.

Original: Reward.
Revised: Reward.

Original: Under the normal alignment procedure, the reward at each step is positive and consists oftwo terms:
Revised: Under the normal alignment procedure, the reward at each step is positive and consists oftwo terms:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is the visibility, which, as discussed above, is the primary metric for the interferometeralignment quality.
Revised: The ﬁrst term is the visibility, which, as discussed above, is the primary metric for the interferometeralignment quality.

Original: The second term rewards high quality of the ﬁnal alignment, which is critical foroptical experiments; it tends to inﬁnity for V → 1 [12].
Revised: The second term rewards high quality of the ﬁnal alignment, which is critical foroptical experiments; it tends to inﬁnity for V → 1 [12].

Revised: However, if the agent proposes an action that takes one of the controls out of the boundaries deﬁnedin Table 1, the episode is terminated to avoid damage to the equipment.

Revised: Additionally, the agent

Original: Element angle / position
Revised: Element angle / position

Original: [Equation]
Revised: [Equation]

Original: Amplitude of max deﬂection
Revised: Amplitude of max deﬂection

Original: [Equation]
Revised: [Equation]

Original:         )UDPH       0 HDQ  L Q W HQ V L W \ 6LPXODWLRQZLWKRXWSKDVHQRLVH         )UDPH 6LPXODWLRQZLWKSKDVHQRLVH         )UDPH ([SHULPHQW 
Revised:         )UDPH       0 HDQ  L Q W HQ V L W \ 6LPXODWLRQZLWKRXWSKDVHQRLVH         )UDPH 6LPXODWLRQZLWKSKDVHQRLVH         )UDPH ([SHULPHQW is penalized with the reward of P = − 0 .

Original: However, if the agent proposes an action that takes one of the controls out of the boundaries deﬁnedin Table 1, the episode is terminated to avoid damage to the equipment.

Original: Additionally, the agentis penalized with the reward of P = − 0 .

Original: This penalty is important during early stages of thetraining, when the reward (4) is normally close to zero.
Revised: This penalty is important during early stages of thetraining, when the reward (4) is normally close to zero.

Original: A small negative reward for unsafe actionswill train the agent to be aware of the bounds, but will not discourage exploration.
Revised: A small negative reward for unsafe actionswill train the agent to be aware of the bounds, but will not discourage exploration.

Original: The speciﬁcvalue of P was handpicked via experimentation.
Revised: The speciﬁcvalue of P was handpicked via experimentation.

Original: On the other hand, when the agent is well-trained,it normally receives a signiﬁcant positive reward for each step.
Revised: On the other hand, when the agent is well-trained,it normally receives a signiﬁcant positive reward for each step.

Original: At this stage, terminating an episodehas a major negative effect on the return, thereby strongly discouraging unsafe actions.
Revised: At this stage, terminating an episodehas a major negative effect on the return, thereby strongly discouraging unsafe actions.

Original: Domain randomization.
Revised: Domain randomization.

Original: To learn a more generalized policy and facilitate sim-to-real transfer, weintroduce several domain randomizations, which are used during the training.
Revised: To learn a more generalized policy and facilitate sim-to-real transfer, weintroduce several domain randomizations, which are used during the training.

Original: The only randomization that changes the dynamic of the environment F ( s t
Revised: The only randomization that changes the dynamic of the environment F ( s t

Original: +

Revised: +

Original: | s t , a t ) is the beam radius randomizationwithin ± 20% of the measured radius r = 0 .
Revised: | s t , a t ) is the beam radius randomizationwithin ± 20% of the measured radius r = 0 .

Original: 71 mm.
Revised: 71 mm.

Original: The randomization is applied at the beginningof each episode.
Revised: The randomization is applied at the beginningof each episode.

Original: Additionally, we introduce the following randomizations of the observation U ( o t | s t ) .
Revised: Additionally, we introduce the following randomizations of the observation U ( o t | s t ) .

Original: First, to address the noise of the camera detector, scattering, air ﬂuctuations and dust effects, we add Gaussiannoise to each pixel acquired by the camera.
Revised: First, to address the noise of the camera detector, scattering, air ﬂuctuations and dust effects, we add Gaussiannoise to each pixel acquired by the camera.

Original: The standard deviation of the noise is 20% with respect to the simulated intensity value.
Revised: The standard deviation of the noise is 20% with respect to the simulated intensity value.

Original: Second, we rearrange cyclically the images within the videosequence to account for the randomness of the camera trigger.
Revised: Second, we rearrange cyclically the images within the videosequence to account for the randomness of the camera trigger.

Original: In addition, we randomize the dutycycle (fraction of time spent on the forward and backward passes) of the piezo.
Revised: In addition, we randomize the dutycycle (fraction of time spent on the forward and backward passes) of the piezo.

Original: Third, we vary thebeams’ brightness by ± 30% to model the variance of the camera exposure.
Revised: Third, we vary thebeams’ brightness by ± 30% to model the variance of the camera exposure.

Original: All these randomizationsare applied to each time step (i.e. they are constant for the 16 frames acquired in each step).
Revised: All these randomizationsare applied to each time step (i.e. they are constant for the 16 frames acquired in each step).

Original: The randomizations described above have been used in Sorokin et al.
Revised: The randomizations described above have been used in Sorokin et al.

Original: [12] and have proven theireffectiveness.
Revised: [12] and have proven theireffectiveness.

Original: An additional randomization we introduce in this work addresses irregularities in themotion of the piezo mirror and the frame rate, as well as ﬂuctuations in the air density in differentarms of the interferometer, all of which lead to random variation of the optical path length differencebetween the two MZI arms.
Revised: An additional randomization we introduce in this work addresses irregularities in themotion of the piezo mirror and the frame rate, as well as ﬂuctuations in the air density in differentarms of the interferometer, all of which lead to random variation of the optical path length differencebetween the two MZI arms.

Original: To simulate these effects, we add Gaussian noise with the standarddeviation of 0 .
Revised: To simulate these effects, we add Gaussian noise with the standarddeviation of 0 .

Original: 5 rad to this phase difference.
Revised: 5 rad to this phase difference.

Original: Action rescaling
Revised: Action rescaling

Original: As the alignment progresses, the agent’s actions become smaller and moreprecise.
Revised: As the alignment progresses, the agent’s actions become smaller and moreprecise.

Original: The typical action magnitude decreases during an episode by about two orders (as illustratedin the experimental Section 6 below).
Revised: The typical action magnitude decreases during an episode by about two orders (as illustratedin the experimental Section 6 below).

Original: It is therefore desirable that the exploration noise also decreasewith the action magnitude.
Revised: It is therefore desirable that the exploration noise also decreasewith the action magnitude.

Original: We satisfy this by setting the agent’s neural network to output a “rawaction” value a 0 ∈ [ − 1 , 1] , from which the actual action is calculated according to
Revised: We satisfy this by setting the agent’s neural network to output a “rawaction” value a 0 ∈ [ − 1 , 1] , from which the actual action is calculated according to

Original: [Equation]
Revised: [Equation]

Original: This transformation produces rescaled actions with absolute values | a | ∈ { 0 } ∪ [2 . 5 · 10 − 3 , 1] .
Revised: This transformation produces rescaled actions with absolute values | a | ∈ { 0 } ∪ [2 . 5 · 10 − 3 , 1] .

Original: Algorithm and network architecture.
Revised: Algorithm and network architecture.

Original: We use the standard TD3 algorithm with handpickedhyperparameters to produce a deterministic policy, which yields the raw action a 0 as describedabove.
Revised: We use the standard TD3 algorithm with handpickedhyperparameters to produce a deterministic policy, which yields the raw action a 0 as describedabove.

Original: For both critics and the actor, we use the VGG-16 [27] architecture, modiﬁed as follows.
Revised: For both critics and the actor, we use the VGG-16 [27] architecture, modiﬁed as follows.

Original: The number of convolutional layers in the encoder is set to 8, followed by three MLP layers, withno dropout.
Revised: The number of convolutional layers in the encoder is set to 8, followed by three MLP layers, withno dropout.

Original: We have chosen this architecture because max pooling operations help reducing the [Table] overﬁtting and sensitivity to individual pixel noise.
Revised: We have chosen this architecture because max pooling operations help reducing theoverﬁtting and sensitivity to individual pixel noise.

Original: We use orthogonal initialization in all models aswe found it to ensure faster convergence. 
Revised: We use orthogonal initialization in all models aswe found it to ensure faster convergence. [Table]

Original: We train the agentin a simulated environment with a discount factor γ = 0 . 8 ; this relatively shortreward sight inspires our agent to reach high visibility faster.
Revised: We train the agent with a discount factor γ = 0 . 8 ; this relatively short reward sight inspires ouragent to reach high visibility faster.

Original: A Gaussian exploration noise is addedto the raw action a 0 with the standard deviation decreasing exponentially from 0.5 to 0.02 during thetraining.
Revised: A Gaussian exploration noise is added to the raw action a 0with the standard deviation decreasing exponentially from 0.5 to 0.02 during the training.

Revised: Althoughthe variance of this noise is independent of the magnitude of a 0 , its effect on the actual action a isproportional to its magnitude due to the exponential dependence (5).

Original: The total number of steps is 10 6 and the replay buffer size is 10 5 .
Revised: The total number of steps is10 6 and the replay buffer size is 10 5 .

Original: Updates are performedevery ten steps.
Revised: Updates are performed every ten steps.

Original: The whole training on an NVidia RTX 2080 GPU takes about 26 hours.
Revised: The whole training onan NVidia RTX 2080 GPU takes about 26 hours.

Original:  Experimental evaluation
Revised:  Experimental evaluation

Revised: [Figure]

Original:    
Revised:     

Original: [Figure]
Revised: [Figure]

Original:       9 L V L E LOL W \
Revised: L V L [Figure] LOL W \

Original: [Figure]
Revised: [Figure]

Original:      
Revised:        

Original:     
Revised:  

Original:  
Revised:  

Original: [Figure]
Revised: [Figure]

Original: \
Revised: \

Original: [Figure]
Revised: [Figure]

Original:  indicating higher stability of the former.
Revised: , indicating higher stability of the former.

Original: All results are averaged over 50 episodes.
Revised: All results are averaged over 50 episodes.

Original: For our experiments, we build an MZI according to the scheme shown in Fig.
Revised: For our experiments, we build an MZI according to the scheme shown in Fig.

Original: We use a continuous
Revised: We use a continuous

Original: HeNe laser with wavelength λ
Revised: HeNe laser with wavelength λ

Original: = 632 nm, Newport Picomotor mirror mounts, a Standa 8MT16725LS linear translation stage, a CMOS camera with 16 fps acquisition rate, and a high-bandwidthphotodetector to precisely measure the visibility.
Revised: = 632 nm, Newport Picomotor mirror mounts, a Standa 8MT16725LS linear translation stage, a CMOS camera with 16 fps acquisition rate, and a high-bandwidthphotodetector to precisely measure the visibility.

Original: In addition to our agent, we also train and evaluate the original Interferobot [12] with the action space extended to include the lens movement.
Revised: In addition to our agent, we also train and evaluate the original Interferobot [12] with the action space extended to include the lens movement.

Original: Foreach agent, we run 50 episodes of evaluation with 100 timesteps.
Revised: Foreach agent, we run 50 episodes of evaluation with 100 timesteps.

Original: Figure 4 shows the evaluation results.
Revised: Figure 4 shows the evaluation results.

Original: Our agent signiﬁcantly outperforms Interferobot in terms of the alignment time, ﬁnal visibility(averaged over the last 40 steps of each episode) and its variance.
Revised: Our agent signiﬁcantly outperforms Interferobot in terms of the alignment time, ﬁnal visibility(averaged over the last 40 steps of each episode) and its variance.

Original: Additionally, our agent has been compared with a human expert, who aligned the interferometer 10times by manually turning the controls on the physical setup.
Revised: Additionally, our agent has been compared with a human expert, who aligned the interferometer 10times by manually turning the controls on the physical setup.

Original: As evidenced by Table 2, our agentsurpasses the human both in the speed and quality of the alignment.
Revised: As evidenced by Table 2, our agentsurpasses the human both in the speed and quality of the alignment.

Original:       6WHS       
Revised: A common trick used by human experts in aligning optics is to turn two mirrors steering the samebeam by equal and opposite angles.

Original: [Figure]

Original: 

Original:   

Original: [Figure]

Original:     [Figure] A common trick used by human experts in aligning optics is to turn two mirrors steering the samebeam by equal and opposite angles. This action moves the position of the beam without changingits direction.
Revised: This action moves the position of the beam without changingits direction.

Original: Fig.
Revised: Fig.

Original: 5a shows that our agent learns this method and uses it extensively: by the endof the procedure, about 70% of the actions contain such parallel movements.
Revised: 5a shows that our agent learns this method and uses it extensively: by the endof the procedure, about 70% of the actions contain such parallel movements.

Original: Fig.
Revised: Fig.

Original: 5b shows theaction norm as a function of the step number.
Revised: 5b shows theaction norm as a function of the step number.

Original: The descending trend means that at the beginningour agent uses large actions to obtain rough alignment and decreases the action size to ﬁne-tune theinterferometer.
Revised: The descending trend means that at the beginning

Original: Table 3 presents the results of an ablation study.
Revised:       6WHS        [Figure]

Revised: 

Revised:    

Revised: [Figure]

Revised:     

Revised: [Figure]

Revised: F W

Revised: [Figure]

Revised: RQ

Revised: [Figure]

Revised: U P

Revised: [Figure]

Revised: our agent uses large actions to obtain rough alignment and decreases the action size to ﬁne-tune theinterferometer

Revised: Table 3 presents the results of an ablation study.

Original: A standard TD3 agent does not outperform theoriginal Interferobot and achieves a mean visibility of V = 0 .
Revised: A standard TD3 agent does not outperform theoriginal Interferobot and achieves a mean visibility of V = 0 .

Original: Action rescaling and phase noiserandomizations signiﬁcantly boost the performance of our agent, leading to a visibility of V =0 .
Revised: Action rescaling and phase noiserandomizations signiﬁcantly boost the performance of our agent, leading to a visibility of V =0 .

Original: It is notable that the phase noise randomization slightly decreases the mean visibility of the
Revised: It is notable that the phase noise randomization slightly decreases the mean visibility of the

Original: Interferobot but improves the standard deviation.
Revised: Interferobot but improves the standard deviation.

Original: This bias-variance trade-off can be due to therelative simplicity of the Interferobot model which cannot capture richer observations produced bythis randomization.
Revised: This bias-variance trade-off can be due to therelative simplicity of the Interferobot model which cannot capture richer observations produced bythis randomization.

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: We demonstrated a novel RL algorithm for the automatic alignment of an optical interferometer,which contains a system of lenses in one of its optical paths.
Revised: We demonstrated a novel RL algorithm for the automatic alignment of an optical interferometer,which contains a system of lenses in one of its optical paths.

Original: Such a setup results in richer observation and action spaces and constiututes a better approximation of a general problem of achievingmode matching between two arbitrary Gaussian beams.
Revised: Such a setup results in richer observation and action spaces and constiututes a better approximation of a general problem of achievingmode matching between two arbitrary Gaussian beams.

Original: Our reinforcement learning agent with acontinuous action space solves the alignment problem successfully, surpassing both a previous solution and a human expert.
Revised: Our reinforcement learning agent with acontinuous action space solves the alignment problem successfully, surpassing both a previous solution and a human expert.

Original: Important innovations include exponential scaling of the action space,an additional domain randomization that helps the agent generalize to a real setup after training insimulation as well as a discrete reward structure that trains the agent to avoid unsafe actions.
Revised: Important innovations include exponential scaling of the action space,an additional domain randomization that helps the agent generalize to a real setup after training insimulation as well as a discrete reward structure that trains the agent to avoid unsafe actions.