Original: Learning Interpretable BEV Based VIO without Deep Neural Networks
Revised: Learning Interpretable BEV Based VIO without Deep Neural Networks

Original: Anonymous Author(s)

Original: Abstract: Monocular visual-inertial odometry (VIO) is a critical problem inrobotics and autonomous driving.
Revised: Abstract: Monocular visual-inertial odometry (VIO) is a critical problem in robotics and autonomous driving.

Original: Traditional methods solve this problem basedon filtering or optimization.
Revised: Traditional methods solve this problem based on filtering or optimization.

Original: While being fully interpretable, they rely on manualinterference and empirical parameter tuning.
Revised: While being fully interpretable, they rely on manual interference and empirical parameter tuning.

Original: On the other hand, learning-basedapproaches allow for end-to-end training but require a large number of trainingdata to learn millions of parameters.
Revised: On the other hand, learning-based approaches allow for end-to-end training but require a large number of training data to learn millions of parameters.

Original: However, the non-interpretable and heavymodels hinder the generalization ability.
Revised: However, the non-interpretable and heavy models hinder the generalization ability.

Original: In this paper, we propose a fully differentiable, and interpretable, monocular VIO model  that can be trained without deepneural networks.
Revised: In this paper, we propose a fully differentiable, and interpretable, bird-eye-view (BEV) based VIO model for robots with local planar motion that can be trained without deep neural networks.

Original: Specifically, we first adopt Unscented Kalman Filter as a differentiable layer to predict the pitch and roll, where the covariance matrices of noiseare learned to filter out the noise of the IMU raw data.
Revised: Specifically, we first adopt Unscented Kalman Filter as a differentiable layer to predict the pitch and roll, where the covariance matrices of noise are learned to filter out the noise of the IMU raw data.

Original: Second, the refined pitchand roll are adopted to retrieve a gravity-aligned BEV image of each frame usingdifferentiable camera projection.
Revised: Second, the refined pitch and roll are adopted to retrieve a gravity-aligned BEV image of each frame using differentiable camera projection.

Original: Finally, a differentiable pose estimator is utilized to estimate the remaining 3 DoF poses between the BEV frames, leading toa 5 DoF pose estimation.
Revised: Finally, a differentiable pose estimator is utilized to estimate the remaining 3 DoF poses between the BEV frames: leading to a 5 DoF pose estimation.

Original: Our method allows for learning the covariance matricesend-to-end supervised by the pose estimation loss, demonstrating superior performance to empirical baselines.
Revised: Our method allows for learning the covariance matrices end-to-end supervised by the pose estimation loss, demonstrating superior performance to empirical baselines.

Original: Experimental results on synthetic and real-worlddatasets demonstrate that our simple approach is competitive with state-of-the-artmethods and generalizes well on unseen scenes.
Revised: Experimental results on synthetic and real-world datasets demonstrate that our simple approach is competitive with state-of-the-art methods and generalizes well on unseen scenes.

Original:  Introduction
Revised:  Introduction

Original: Visual-inertial odometry (VIO) is a fundamental component of the visual-inertial simultaneous localization and mapping system with many applications in robotics and autonomous driving.
Revised: Visual-inertial odometry (VIO) is a fundamental component of the visual-inertial simultaneous localization and mapping system with many applications in robotics and autonomous driving.

Original: Asa cheap and efficient solution, monocular VIO has attracted growing interest recently.
Revised: As a cheap and efficient solution, monocular VIO has attracted growing interest recently.

Original: However,monocular VIO is a challenging task considering the limited sensory information.
Revised: However, monocular VIO is a challenging task considering the limited sensory information.

Original: Traditional monocular VIO approaches have demonstrated promising results based on classical feature detection and feature matching [1][2][3][4], see Fig.
Revised: Traditional monocular VIO approaches have demonstrated promising results based on classical feature detection and feature matching [1][2][3][4], see Fig.

Original: 1 (top).
Revised: 1 (top).

Original: These methods are interpretableand generalize well, but require manual interference and empirical parameter tuning.
Revised: These methods are interpretable and generalize well, but require manual interference and empirical parameter tuning.

Original: Moreover,these methods only leverage sparse features, resulting in the degraded performance in texturelessregions [5].
Revised: Moreover, these methods only leverage sparse features, resulting in the degraded performance in textureless regions [5].

Original: More recently, learning-based approaches address these limitations by learning the ego-motion inan end-to-end fashion from the raw, dense monocular images, as illustrated in Fig.
Revised: More recently, learning-based approaches address these limitations by learning the ego-motion in an end-to-end fashion from the raw, dense monocular images, as illustrated in Fig.

Original: 1 (middle).
Revised: 1 (middle).

Original: However, they lack interpretability and thus pose a new challenge to generalization.
Revised: However, they lack interpretability and thus pose a new challenge to generalization.

Original: While manyworks improve the interpretability by predicting an intermediate optical flow [6][7][8][9] or depthmap [10][11][12][13], the following pose regression network remains uninterpretable.
Revised: While many works improve the interpretability by predicting an intermediate optical flow [6][7][8][9] or depth map [10][11][12][13], the following pose regression network remains uninterpretable.

Original: Furthermore,existing learning-based approaches rely on a large number of training images to learn millions ofparameters while the ego-motion has essentially 6 DoF only.
Revised: Furthermore, existing learning-based approaches rely on a large number of training images to learn millions of parameters while the ego-motion has essentially 6 DoF only.

Original: We thus ask the following question: Is it possible to obtain a fully interpretable model with a minimal set of parameters being learnedend-to-end?
Revised: We thus ask the following question: Is it possible to obtain a fully interpretable model with a minimal set of parameters being learned endto-end?

Original: In this paper, we focus on the visual inertial odometry (VIO) forvehicles with local planar motionwhich consists of 5 DoF  withthe elevation fixed. This is not trivial, as  driving  in urban and highwayscan usually be treated as planar motion locally with a fixed elevation.
Revised: In this paper, we focus on the BEV based VIO for 5 DoF local planar motion with elevation fixed. This is not trivial as such motion pattern is applicable for unmanned ground vehicles and autonomous driving vehicles in urban, underground parking lots, tunnels and highways [14][15].

Original: We combine the advantages of the traditional methods with learning-based approaches.
Revised: We combine the advantages of the traditional methods with learning-based approaches.

Revised: Several former works [16][17] explore this possibility using traditional filtering for interpretable fusion, but leaving the sensor model a plain regression network.

Original: As illustrated in Fig. 1(bottom), our key idea is to design a fully differentiable, interpretable model that allows for replacingempirical parameter tuning with learning from the data.
Revised: Our key idea is to design a fully differentiable, interpretable model that eliminates all black-box neural networks, yet allowing for replacing empirical parameter tuning with learning from the data, see Fig. 1 (bottom).

Original: Additionally, our method exploits denseappearance information instead of sparse features.
Revised: Additionally, our method exploits dense appearance information instead of sparse features.

Original: As it is difficult to differentiate the6 DoF poseestimation that involves robust and non-linear optimization, we focus on the 3 DoF image basedpose estimation conditioned on the estimated pitch and roll angles.
Revised: As it is difficult to differentiate the DoF pose estimation that involves robust and non-linear optimization, we focus on the 3 DoF image based pose estimation conditioned on the estimated pitch and roll angles.

Original: More formally, we consider the  monocular VIO as a function that outputs a relative pose takingas input i) two consecutive frames, and ii) IMU data.
Revised: More formally, we consider the BEV based monocular VIO as a function that outputs a relative pose taking as input i) two consecutive frames, and ii) IMU data.

Original: First, we utilize a Differentiable Unscented Kalman Filter (DUKF) to filter out noise from the IMU raw measurements and retrieve pitch androll.
Revised: First, we utilize a Differentiable Unscented Kalman Filter (DUKF) to filter out noise from the IMU raw measurements and retrieve pitch and roll.

Original: As the covariance matrices of noise in both the motion model and the measurement model arenot known precisely in practice, classical UKF relies on empirical parameter tuning given a specificscene.
Revised: As the covariance matrices of noise in both the motion model and the measurement model are not known precisely in practice, classical UKF relies on empirical parameter tuning given a specific scene.

Original: In contrast, we exploit the fact that UKF is differentiable and optimize the covariance matricesin an end-to-end manner leveraging the training data.
Revised: In contrast, we exploit the fact that UKF is differentiable and optimize the covariance matrices in an end-to-end manner leveraging the training data.

Original: This requires the estimator of the remaining3 DoF pose to be differentiable as well.
Revised: This requires the estimator of the remaining 3 DoF pose to be differentiable as well.

Original: Therefore, in the second step, we estimate the remaining3 DoF poses by converting monocular images to bird-eye view conditioned on the denoised pitchand roll [14].
Revised: Therefore, in the second step, we estimate the remaining 3 DoF poses by converting monocular images to bird-eye view conditioned on the denoised pitch and roll [14].

Original: The bird-eye view representation allows us to obtain a global optimal solution using Differentiable Phase Correlation (DPC) [16] in a single forward pass.
Revised: The bird-eye view representation allows us to obtain a global optimal solution using Differentiable Phase Correlation (DPC) [18] in a single forward pass.

Original: Note that DPC leveragesdense appearance information and does not introduce any trainable parameters.
Revised: Note that DPC leverages dense appearance information and does not introduce any trainable parameters.

Original: More importantly,it allows for back-propagating the gradient to the DUKF module to update the covariance matrices.
Revised: More importantly, it allows for back-propagating the gradient to the DUKF module to update the covariance matrices.

Original: The combination of DUKF and DPC provides us a fully differentiable, and interpretable BEV basedmodel for VIO, named BEVO, with the parameters of the covariance matrices as the only trainableparameters and no deep neural networks involved.
Revised: The combination of DUKF and DPC provides us a fully differentiable, and interpretable BEV based model for VIO, named BEVO, with the parameters of the covariance matrices as the only trainable parameters and no deep neural networks involved.

Revised: We emphasize that by carefully combining the designed mechanistic model with learning approaches, the model is inherently interpretable.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • We present a fully differentiable and interpretable monocular VIO framework  that combines theadvantages of both traditional and learning-based approaches.• We propose to use the Unscented Kalman Filter as a differentiable layer and leverage the Differentiable Phase Correlation on BEV images for pose estimation.
Revised: • We present a fully differentiable and interpretable monocular VIO framework for planar moving robots, e.g., autonomous driving or unmanned ground vehicles, that combines the advantages of both traditional and learning-based approaches. • We propose to use the Unscented Kalman Filter as a differentiable layer and leverage the Differentiable Phase Correlation on BEV images for pose estimation.

Original: This novel combination allowsfor learning the covariance matrices from data in an end-to-end manner.
Revised: This novel combination allows for learning the covariance matrices from data in an end-to-end manner.

Original: • Extensive experiments on KITTI, CARLA and AeroGround demonstrate that, surprisingly, oursimple approach shows competitive results compared to the state-of-the-art.
Revised: • Extensive experiments on KITTI, CARLA, and AeroGround demonstrate that, surprisingly, our simple approach shows competitive results compared to the state-of-the-art.

Original:  Fully Differentiable and Interpretable Model for VIO
Revised:  Fully Differentiable and Interpretable Model for VIO

Original: We aim to estimate the odometry with the input of one monocular camera and IMU data by adifferentiable and interpretable model.
Revised: We aim to estimate the odometry with the input of one monocular camera and IMU data by a differentiable and interpretable model.

Original: Following the pipeline in Fig.
Revised: Following the pipeline in Fig.

Original: 2, we employ a trainable
Revised: 2, we employ a trainable

Original: UKF to integrate IMU data for pitch and roll estimation.
Revised: UKF to integrate IMU data for pitch and roll estimation.

Original: Then the denoised pitch and roll, togetherwith the corresponding front image are fed into the second differentiable module, for differentiablepose estimation of the remaining 3 DoF pose.
Revised: Then the denoised pitch and roll, together with the corresponding front image are fed into the second differentiable module, for differentiable pose estimation of the remaining 3 DoF pose.

Original: To this goal, we transform the image to the BEVimages conditioned on the denoised pitch and roll.
Revised: To this goal, we transform the image to the BEV images conditioned on the denoised pitch and roll.

Original: Following the BEV projection, the SIM (2)pose of two consecutive BEV images is estimated by the DPC.
Revised: Following the BEV projection, the SIM (2) pose of two consecutive BEV images is estimated by the DPC.

Original: Note that the UKF is a continuousforward function and is differentiable in essence.
Revised: Note that the UKF is a continuous forward function and is differentiable in essence.

Original: In this work, we emphasize its backward path byreferring to it as DUKF, to distinguish it from the traditional UKF utilization which focuses onlyon the forward path.
Revised: In this work, we emphasize its backward path by referring to it as DUKF, to distinguish it from the traditional UKF utilization which focuses only on the forward path.

Original: As a result, the only trainable parameters are the 4 numbers that make up thecovariance matrices of motion and measurement in the DUKF and are trained end-to-end with thepose loss altogether.[Figure]
Revised: As a result, the only trainable parameters are the 4 numbers that make up the covariance matrices of motion and measurement in the DUKF and are trained end-to-end with the pose loss altogether.

Original: 2.1 Differentiable Unscented Kalman Filter
Revised: 2.1 Differentiable Unscented Kalman Filter

Original: [Figure]

Original: For the DUKF, there are two basic models in the pipeline, namely, the motion model and the measurement model.
Revised: For the DUKF, there are two basic models in the pipeline, namely, the motion model and the measurement model.

Original: We follow the right-hand system through the paper.
Revised: We follow the right-hand system through the paper.

Original: Motion Model:
Revised: Motion Model:

Original: Given the filtered pitch and roll in time t − 1 , we have the state in time t − 1 as:
Revised: Given the filtered pitch and roll in time t − 1 , we have the state in time t − 1 as:

Original: [Equation]
Revised: [Equation]

Original:  where α , and β are the pitch and roll angle respectively.
Revised: 1) where α , and β are the pitch and roll angle respectively.

Original: Following the sampling theory of UKF[17], the sampled state in t − 1 becomes X t − 1 =
Revised: Following the sampling theory of UKF [20], the sampled state in t − 1 becomes X t − 1 =

Original: [Equation]

Original: ( x t − 1 ) , where Γ is the sampling rule.
Revised: Γ( x t − 1 ) , where Γ is the sampling rule.

Original: Given themotion model G , we have the initial guess of pitch and roll in time t as:
Revised: Given the motion model G , we have the initial guess of pitch and roll in time t as:

Original: [Equation]
Revised: [Equation]

Original: where ω α , and ω β are the pitch and roll rate from the gyroscope, and ¯ µ t and ¯ σ t are the mean andvariance of the initial guess of ¯ X ∗ t , respectively.
Revised: where ω α , and ω β are the pitch and roll rate from the gyroscope, and ¯ µ t and ¯ σ t are the mean and variance of the initial guess of ¯ X ∗ t , respectively.

Original: ∆ t is the interval of the IMU data.
Revised: ∆ t is the interval of the IMU data.

Original: Note that w mand w c are the weights for each sampled state in ¯ X ∗ t based on the theory in [17], and O t is thecovariance matrix for the noise of the motion model to be trained.
Revised: Note that w m and w c are the weights for each sampled state in ¯ X ∗ t based on the theory in [20], and O t is the covariance matrix for the noise of the motion model to be trained.

Original: Measurement Model: After updating the covariance matrix O t which is the motion prediction, weretrieve a new sampled state ¯ X t in time t :[Equation]
Revised: Measurement Model: After updating the covariance matrix O t which is the motion prediction, we retrieve a new sampled state ¯ X t in time t :

Original: [Equation]
Revised: [Equation]

Original: where ¯ x t satisfies the normal distribution.
Revised: where ¯ x t satisfies the normal distribution.

Original: Given the measurement model H , the initial guess ofmeasurement is
Revised: Given the measurement model H , the initial guess of measurement is

Original: [Equation]
Revised: [Equation]

Original: where ¯ M t and ¯Σ t are the mean and variance of ¯ X t respectively, and Q t is the covariance matrix forthe noise of the measurement model to be trained.
Revised: where ¯ M t and ¯Σ t are the mean and variance of ¯ X t respectively, and Q t is the covariance matrix for the noise of the measurement model to be trained.

Original: Now, considering a real-world measurement is obtained:
Revised: Now, considering a real-world measurement is obtained:

Original: [Equation]
Revised: [Equation]

Original: where ˆ α t and ˆ β t are the measurement of pitch and roll in time t with respect to the raw accelerationdata in each axes acc xt , acc yt , and acc zt .
Revised: where ˆ α t and ˆ β t are the measurement of pitch and roll in time t with respect to the raw acceleration data in each axes acc xt , acc yt , and acc zt .

Original: Altogether, the real-world measurement state is formed as
Revised: Altogether, the real-world measurement state is formed as

Original: Z t .
Revised: Z t .

Original: By now, we come to the last stage of the DUKF, update the final state x t as follows:
Revised: By now, we come to the last stage of the DUKF, update the final state x t as follows:

Original: [Equation]
Revised: [Equation]

Original: where ¯Σ X,Zt , K t are the cross-covariance and Kalman Gain respectively, µ t and σ t are the mean andvariance of x t which will also guide the sampling in the following time t + 1 .
Revised: where ¯Σ X,Zt , K t are the cross-covariance and Kalman Gain respectively, µ t and σ t are the mean and variance of x t which will also guide the sampling in the following time t + 1 .

Original: 2.2 Differentiable Bird-eye View Projection
Revised: 2.2 Differentiable Bird -eye View Projection

Original: To achieve the projection from the front view to the birds-eye view with DBEV, we first define thetransformation matrix from the IMU to the ground beneath the vehicle considering its pitch, roll,and elevation:
Revised: To achieve the projection from the front view to the birds-eye view with DBEV, we first define the transformation matrix from the IMU to the ground beneath the vehicle considering its pitch, roll, and elevation:

Original: [Equation]
Revised: [Equation]

Original: the rotation matrix R IMUt of the IMU at time t changes constantly with α t and β t which are estimated by the DUKF.
Revised: the rotation matrix R IMUt of the IMU at time t changes constantly with α t and β t which are estimated by the DUKF.

Original: The translation vector T IMUt of the IMU is known and fixed since it is theurban road and highway that we are considering in which the height of the IMU to the groundconst z stays unchanged through the journey [14].
Revised: The translation vector T IMUt of the IMU is known and fixed since it is the urban road and highway that we are considering in which the height of the IMU to the ground const z stays unchanged through the journey [14].

Original: Transformation matrices from camera to theworld [ R CAMt , T CAMt ] are obtained with the calibration from IMU to camera and [ R IMUt , T IMUt ] .
Revised: Transformation matrices from camera to the world [ R CAMt , T CAMt ] are obtained with the calibration from IMU to camera and [ R IMUt , T IMUt ] .

Original: Besides the unchanged height, one more assumption can be made for a safe trip: there should besufficient distance ahead, forming that the center bottom part of the front image should represent aplane ground, shown as the yellow dashed box in Fig. 
Revised: Besides the unchanged height, one more assumption can be made for a safe trip: there should be sufficient distance ahead which should be about 10 m at 34 km/h according to the mounting height and angle (similar to [14] [15]), forming that the center bottom part of the front image should represent a plane ground, shown as the yellow dashed box in Fig. 2.

Original: With the assumption above, we consider pixels within the red dashed box as a set of points in the3D space, and map them from the camera plane to the ground plane.
Revised: With the assumption above, we consider pixels within the yellow dashed box as a set of points in the 3D space, and map them from the camera plane to the ground plane.

Original: To begin with, we generate a set of even points P g on the ground plane that stands for the locationthat the points { u, v } in the BEV should be projected to:
Revised: To begin with, we generate a set of even points P g on the ground plane that stands for the location that the points { u, v } in the BEV should be projected to:

Original: [Equation]
Revised: [Equation]

Original: (15) where X ′ , Y ′ , and Z ′ are the coordinates of points.
Revised: (15) where X ′ , Y ′ , and Z ′ are the coordinates of points.

Original: P g is then transformed to the camera coordinatewith:
Revised: P g is then transformed to the camera coordinate with:

Original: [Equation]
Revised: [Equation]

Original: and P tc is projected to the camera plane with the focal lengths f x , f y and principal point ( c x , c y ) :
Revised: and P tc is projected to the camera plane with the focal lengths f x , f y and principal point ( c x , c y ) :

Original: [Equation]
Revised: [Equation]

Original: c

Original: [Equation]
Revised: [Equation]

Original: f
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: c
Revised: Z

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: ′′ t Y ′′ t Z ′′ t   , (17) where ( u ′ t , v ′ t ) are the indexes on the front camera image of each corresponding point in P g in timet .
Revised: (17) where ( u ′ t , v ′ t ) are the indexes on the front camera image of each corresponding point in P g in time t .

Original: Therefore, by remapping the original pixels ( u, v ) to ( u ′ t , v ′ t ) , we retrieve the BEV projection I bevtout of the yellow dashed box I ft in Fig. 2:
Revised: Therefore, by remapping the original pixels ( u, v ) to ( u ′ t , v ′ t ) , we retrieve the BEV projection I bevt out of the yellow dashed box I ft in Fig. 2:

Original: [Equation]
Revised: [Equation]

Original: 18) where { u ′ t , v ′ t } =
Revised: 18) where { u ′ t , v ′ t } =

Original: F ( { u t , v t } ) is the remapping from BEV to front.
Revised: F ( { u t , v t } ) is the remapping from BEV to front.

Original: Since DBEV is parameterizedby the pitch and roll, which is related to the variance of IMU data, with the help of DUKF, thegradient can be back-propagated to the variance.
Revised: Since DBEV is parameterized by the pitch and roll, which is related to the variance of IMU data, with the help of DUKF, the gradient can be back-propagated to the variance.

Original: 2.3 Differentiable Phase Correlation
Revised: 2.3 Differentiable Phase Correlation

Original: [Figure] Given two images I bevt − 1 and I bevt with pose transformations, a variation of the differentiable phase correlation [16] is utilized to estimate the the overallrelative pose ξ i between I bevt − 1 and I bevt .
Revised: Given two images I bevt − 1 and I bevt with pose transformations, a variation of the differentiable phase correlation [18] is utilized to estimate the the overall relative pose ξ i between I bevt − 1 and I bevt .

Original: The rotationand scale are calculated by:
Revised: The rotation and scale are calculated by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where ξ θ,st is the rotation and scale part of ξ i between I bevt − 1 and I bevt , F is the discrete Fourier
Revised: (20) where ξ θ,st is the rotation and scale part of ξ i between I bevt − 1 and I bevt , F is the discrete Fourier

Original: Transform, L is the log-polar transform Log-Polar Transform), and C is the phase correlation solver.
Revised: Transform, L is the log-polar transform Log-Polar Transform), and C is the phase correlation solver.

Original: Fourier Transformation F transforms images into the Fourier frequency domain of which the magnitude has the property of translational insensitivity, therefore the rotation and scale are decoupled withdisplacements and are represented in the magnitude (Fig.
Revised: Fourier Transformation F transforms images into the Fourier frequency domain of which the magnitude has the property of translational insensitivity, therefore the rotation and scale are decoupled with displacements and are represented in the magnitude (Fig.

Original: 3 Magnitude of FFT).
Revised: 3 Magnitude of FFT).

Original: Log-polar transformation L transforms Cartesian coordinates into log-polar coordinates so that such rotation and scalein the magnitude of Fourier domain are remapped into displacement in the new coordinates, makingit solvable with the afterward phase correlation solver (Fig. 3.
Revised: Log-polar transformation L transforms Cartesian coordinates into log-polar coordinates so that such rotation and scale in the magnitude of Fourier domain are remapped into displacement in the new coordinates, making it solvable with the afterward phase correlation solver (Fig. 3.

Original: Phase correlation solver C outputs aheatmap indicating the displacements of the two log-polar images, which eventually stands for therotation θ t and scale s t of the two input images I bevt − 1 and I bevt (Fig.
Revised: Phase correlation solver C outputs a heatmap indicating the displacements of the two log-polar images, which eventually stands for the rotation θ t and scale s t of the two input images I bevt − 1 and I bevt (Fig.

Original: 3 Phase Correlation of Rotation).
Revised: 3 Phase Correlation of Rotation).

Original: Note that in our case, the scale s t is fixed as 1.
Revised: Since we assume that the height of the robot does not change drastically along the trajectory, the scale s t is fixed as 1.

Original: To make the solver differentiable, we use expectation E ( · ) as the estimation of · .
Revised: To make the solver differentiable, we use expectation E ( · ) as the estimation of · .

Original: Then I bevt is rotated and scaled referring to ξ θ,s t with the result of ¯ I bevt .
Revised: Then I bev t is rotated and scaled referring to ξ θ,s t with the result of ¯ I bev t .

Original: In the same manner, translations ξ t t between ¯ I bevt and I bevt − 1 is calculated with p ( ξ t t )
Revised: In the same manner, translations ξ t t between ¯ I bev t and I bev t − 1 is calculated with p ( ξ t t )

Original: =
Revised: =

Original: C (¯ I bevt , I bevt − 1 ) and t t = E ( p ( ξ t t )) .
Revised: C (¯ I bevt , I bevt − 1 ) and t t = E ( p ( ξ t t )) .

Original: Withthe relative rotation θ t , scale s t , and translation t t between I bevt − 1 and I bevt , and by consulting withthe fixed Distance per Pixel (DPP), we finally arrive at the SIM (2) pose between frame t − 1 andt of vehicle:
Revised: With the relative rotation θ t , scale s t , and translation t t between I bevt − 1 and I bevt , and by consulting with the fixed Distance per Pixel (DPP), we finally arrive at the SIM (2) pose between frame t − 1 and t of vehicle:

Original: [Equation]
Revised: [Equation]

Original: Note that the gradient in the part can be back-propagated all the way back to the variance of the IMUdata since the phase correlation is differentiable.
Revised: Note that the gradient in the part can be back-propagated all the way back to the variance of the IMU data since the phase correlation is differentiable.

Original: 2.4 Loss Design, Back-Propagation, and Training
Revised: 2.4 Loss Design, Back-Propagation, and Training

Original: With DUKF, DBEV and DPC, we define a one-step VIO as a function: 
Revised: With DUKF, DBEV and DPC, we define a one-step VIO as a function: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where VIO O , Q is fully differentiable and interpretable with 4 trainable parameters in O t and Q t .
Revised: where VIO O , Q is fully differentiable and interpretable with 4 trainable parameters in O t and Q t .

Original: I , ω , and a are the front images, gyroscope data, and accelerometer data.
Revised: I , ω , and a are the front images, gyroscope data, and accelerometer data.

Original: Since the BEVO is fully differentiable, it is trained end to end and supervised only with the ground truth of the odometrybetween frames.
Revised: Since the BEVO is fully differentiable, it is trained end to end and supervised only with the ground truth of the odometry between frames.

Original: Such supervision is applied to optimize the covariance matrices of noise in boththe motion model and the measurement model of DUKF.
Revised: Such supervision is applied to optimize the covariance matrices of noise in both the motion model and the measurement model of DUKF.

Original: Elaborately, we wish to supervise the distribution of the poses ξ θ,st and ξ t t and therefore, we calculatethe Kullback–Leibler Divergence loss KLD between { p ( ξ θ,st ) , p ( ξ t t ) } and the ground truth:
Revised: Elaborately, we wish to supervise the distribution of the poses ξ θ,st and ξ t t and therefore, we calculate the Kullback–Leibler Divergence loss KLD between { p ( ξ θ,st ) , p ( ξ t t ) } and the ground truth:

Original: [Equation]
Revised: [Equation]

Original: 23) where 1 θ ∗ and 1 t ∗ are the Gaussian Blurred one-peak distributions centered at the ground truth θ ∗and t ∗ respectively.
Revised: 23) where 1 θ ∗ and 1 t ∗ are the Gaussian Blurred one-peak distributions centered at the ground truth θ ∗ and t ∗ respectively.

Original: Training Details: all trainings are conducted on a GPU server withtwo NVIDIA RTX 3090 as
Revised: Training Details: all trainings are conducted on a GPU server with NVIDIA RTX 3090.

Original: GPUs and Intel i9 9900k as the CPU.

Original: For the initialization, we randomize all initial values of O and Q within the range of [0 , 1] uniformly.
Revised: For the initialization, we randomize all initial values of O and Q within the range of [0 , 1] uniformly.

Original: Since O and Q are 2 × 2 diagonal matrices, we considerthe learning process of the spherical covariances of them following [1][18] so that the number oftrainable parameters will be reduced to 4.
Revised: Since O and Q are 2 × 2 diagonal matrices, we consider the learning process of the spherical covariances of them following [1][21] so that the number of trainable parameters will be reduced to 4.

Original: We supervise the model only with 3 DoF ground truth(GT) data of the same frequency as the images, i.e. the high frequency GT of the IMU is not needed.
Revised: We supervise the model only with 3 DoF ground truth (GT) data of the same frequency as the images, i.e. the high frequency GT of the IMU is not needed.

Revised: We set the BEV size to 100 × 100 as a trade-off between speed and accuracy.

Revised: The inference time is 21 ms and takes up about 500 MB of the resources.

Original: The pseudocode for training BEVO will be given in Appendix I.
Revised: The pseudocode for training BEVO will be given in Appendix I.

Original: Plug into Map-Based Localization System: in addition to the odometry, we also explore the possibility of integrating BEVO with fully differentiable localization to achieve a drift-free system.
Revised: Plug into Map-Based Localization System: in addition to the odometry, we also explore the possibility of integrating BEVO with fully differentiable localization to achieve a drift-free system.

Original: Weshow results as demonstration of the application along with the pseudocode shown in Appendix II.
Revised: We show results as demonstration of the application along with the pseudocode shown in Appendix II.

Original:  Experiment
Revised:  Experiment

Original: We conduct two main experiments: i) odometry estimation on the KITTI [19] odometry dataset onwhich the experiments on ablation study, testing, generalization are conducted, and ii) heterogeneouslocalization on CARLA [20] and the AeroGround Dataset [21].
Revised: We conduct two main experiments: i) odometry estimation on the KITTI [22] odometry dataset on which the experiments on ablation study, testing, generalization are conducted, and ii) heterogeneous localization on CARLA [23] and the AeroGround Dataset [24].

Original: Dataset and Corresponding Experiments:
Revised: Dataset and Corresponding Experiments:

Original: KITTI: We evaluate BEVO on the KITTI odometry dataset using sequences 00-08 for training and09-10 for the test following the same train/validate/test split as [22][23].
Revised: KITTI & CUD: We evaluate BEVO on the KITTI odometry dataset using sequences 00-08 for training and 09-10 for the test following the same train/validate/test split as [25][26].

Original: In addition to the widelyadopted train/validate split where 20% samples are randomly sampled from the training set forvalidation, we also train our model in the first three quarters of each sequence of 00-08, and evaluatethe performance in the last quarter.
Revised: In addition to the widely adopted train/validate split where 20% samples are randomly sampled from the training set for validation, we also train our model in the first three quarters of each sequence of 00-08, and evaluate the performance in the last quarter.

Original: This train/validate split is more realistic even though it mightpresent a worse result.
Revised: This train/validate split is more realistic even though it might present a worse result.

Original: Moreover, we further investigate the generalization ability by training BEVOwith 20% data of each sequence.
Revised: Moreover, we further investigate the generalization ability by training BEVO with 20% data of each sequence.

Original: Note that sequence 03 cannot be used for training or evaluation inthis study since the raw data where we grab the IMU data is missing.
Revised: Note that sequence 03 cannot be used for training or evaluation in this study since the raw data where we grab the IMU data is missing.

Original: CARLA & AG: We further verify the plausibility of applying the BEVO as a front-end plugin for endto-end heterogeneous localization following the routine in [24].
Revised: CARLA & AG: We further verify the plausibility of applying the BEVO as a front-end plugin for endto-end heterogeneous localization following the routine in [27].

Original: Such experiments are conducted in
Revised: Such experiments are conducted in

Original: CARLA and on the real-world AeroGround Dataset.
Revised: CARLA and on the real-world AeroGround Dataset.

Original: The setups are elaborated in Appendix III.
Revised: The setups are elaborated in Appendix III.

Original: Evaluation Matrices: We evaluate our trajectories using t rel and r rel from the standard KITTItoolkit as in [23].
Revised: Evaluation Matrices: We evaluate our trajectories using t rel and r rel from the standard KITTI toolkit as in [26].

Original: In our case, we do not estimate the elevation due to the assumption that thevehicle will not leave the ground.
Revised: In our case, we do not estimate the elevation due to the assumption that the vehicle will not leave the ground.

Original: However, we still record the initial elevation and keep it constantthrough the journey for comparison, which might degrade our performance in t rel to some extent.
Revised: However, we still record the initial elevation and keep it constant through the journey for comparison, which might degrade our performance in t rel to some extent.

Original: Baselines: For the odometry experiment, we compare to both traditional method VINS [1] and
Revised: Baselines: For the odometry experiment, we compare to both traditional method VINS [1] and

Original: SOTA depth dependent learning-based methods D EEP VIO [5], S ELF VIO [25], L I ET AL .
Revised: SOTA depth dependent learning-based methods D EEP VIO [5], S ELF VIO [28], L I ET AL .

Original: G EO N ET [26], and depth independent learning-based method T ARTAN VO [27].
Revised: G EO N ET [29], and depth independent learning-based method T ARTAN VO [30].

Original: For the localizationexperiments, we compare the performance with VINS-M ONO [1].
Revised: For the localization experiments, we compare the performance with VINS-M ONO [1] and ORB-M [21].

Original: 3.1 Odometry Estimation
Revised: 3.1 Odometry Estimation

Original: We now compare the odometry estimation performance of the proposed BEVO to the state-of-theart methods in two training settings as introduced in Dataset : i) following the train/validate splits[Table] in [22][23] ( Ours 1 ), and ii) split each sequence by quarters, take the first three quarters of eachsequence to train and the last quarter to validate ( Ours 2 ).
Revised: We now compare the odometry estimation performance of the proposed BEVO to the state-of-theart methods in two training settings as introduced in Dataset : i) following the train/validate splits in [25][26] ( Ours 1 ), and ii) split each sequence by quarters, take the first three quarters of each sequence to train and the last quarter to validate ( Ours 2 ).

Original: In addition to these two settings, wefurther train BEVO using the first 20% of the training data of each sequence in Ours 2 to evaluate itsperformance on insufficient data ( Ours 3 ). 
Revised: In addition to these two settings, we further train BEVO using the first 20% of the training data of each sequence in Ours 2 to evaluate its performance on insufficient data ( Ours 3 ). [Table]

Original:  Fig.4 and Table 1 show the qualitative andquantitative comparisons, respectively.
Revised: [Figure] Fig.4 and Table 1 show the qualitative and quantitative comparisons, respectively.

Original: Table 1shows that when trained in the same settings asother baselines, Ours 2 outperforms most of thebaselines in validation results in sequence 00 to08.
Revised: Tableshows that when trained in the same settings as other baselines, Ours 2 outperforms most of the baselines in validation results in sequence 00 to 08.

Original: However, regarding the unseen testing sequences, our Ours 1 beats all other baseline inboth t rel and r rel except for the t rel of D EEPVIO in sequence 10.
Revised: However, regarding the unseen testing sequences, our Ours 1 beats all other baseline in both t rel and r rel except for the t rel of D EEPVIO in sequence 10.

Original: This may be explainedby the fact that D EEP VIO utilizes stereo cameras instead of monocular camera considered inour method.
Revised: This may be explained by the fact that D EEP VIO utilizes stereo cameras instead of monocular camera considered in our method.

Original: The result of the testing sequenceshows that our method is better at generalization compared to other learning-based methods.
Revised: The result of the testing sequence shows that our method is better at generalization compared to other learning-based methods.

Original: Fig.
Revised: Fig.

Original: 4 also shows that, even though pitchand roll are not directly supervised, but withthe low-frequency image-level supervision in[-
-]
Revised: 4 also shows that, even though pitch and roll are not directly supervised, but with the low-frequency image-level supervision in[-
-]

Original: DoF only , they are still accurate.
Revised: DoF only , they are still accurate.

Original: This provesthe effectiveness of the end to end training andthat the interpretability provides inductive bias, which accounts for a good generalization.
Revised: This proves the effectiveness of the end to end training and that the interpretability provides inductive bias, which accounts for a good generalization.

Original: It is worth addressing that without employing any nonlinear optimization such as sliding window,our method is surprisingly comparable with others on sequences 00-08, and is even slightly betterin testing on sequences 09 and 10.
Revised: It is worth addressing that without employing any nonlinear optimization such as sliding window, our method is surprisingly comparable with others on sequences 00-08, and is even slightly better in testing on sequences 09 and 10.

Original: The result of Ours 3 suggests that, in contrast to learning-basedapproaches reliant on a large number of training images, our method is more data-efficient thanks tothe fully interpretable model.
Revised: The result of Ours 3 suggests that, in contrast to learning-based approaches reliant on a large number of training images, our method is more data-efficient thanks to the fully interpretable model.

Original: 3.2 Plugin for Heterogeneous Localization
Revised: 3.2 Plugin for Heterogeneous Localization

Original: In this part, we show the possibility of enabling an end-to-end localization with BEVO as the odometry.
Revised: In this part, we show the possibility of enabling an end-to-end localization with BEVO as the odometry.

Original: Elaborations on the implementation details of the fully differentiable drift-free localization canbe found in the supplementary material.
Revised: Elaborations on the implementation details of the fully differentiable drift-free localization can be found in the supplementary material.

Original: By combining BEVO with the heterogeneous localization(denoted as BEVO+), the whole pipeline is fully differentiable, thus allowing for more accurate poseestimation.
Revised: By combining BEVO with the heterogeneous localization (denoted as BEVO+), the whole pipeline is fully differentiable, thus allowing for more accurate pose estimation.

Original: We compare the localization BEVO+ with BEVO itself as well as the VINS-M ONO (denoted as VINS in the table).
Revised: We compare the localization BEVO+ with BEVO itself as well as the VINS-M ONO (denoted as VINS in the table).

Original: Results in Fig.
Revised: Results in Fig.

Original: 5 and Table 2 prove that the BEVO is able to achievegood performance even when it acts as a plugin in an end-to-end trainable localization framework.
Revised: 5 and Table 2 prove that the BEVO is able to achieve good performance even when it acts as a plugin in an end-to-end trainable localization framework.

Original: It indicates a promising future for upcoming end-to-end autonomous driving, e.g., auto-wheelchair.
Revised: It indicates a promising future for upcoming end-to-end autonomous driving, e.g., auto-wheelchair.

Original: [Table]

Original: 3.3 Ablation Study
Revised: 3.3 Ablation Study

Original: [Table] Several experiments conducted on KITTI are designed for the ablation study to reassure the important role a DUKF plays in the BEVO as well as therole trainable covariance matrices play in DUKF.
Revised: Several experiments conducted on KITTI are designed for the ablation study to reassure the important role a DUKF plays in the BEVO as well as the role trainable covariance matrices play in DUKF.

Original: First, we replace the learned covariance matrix O tby one that is empirically set in [17].
Revised: First, we replace the learned covariance matrix O t by one that is empirically set in [20].

Original: The quantitative result in Table.
Revised: The quantitative result in Table.

Original: 3 shows that the accuracydrops drastically when using a manually tuned matrix, suggesting that the empirical parameter tuning leads to inaccurate pitch and roll estimation, andsubsequently deteriorates the BEV generation and odometry estimation.
Revised: 3 shows that the accuracy drops drastically when us- [Table] ing a manually tuned matrix, suggesting that the empirical parameter tuning leads to inaccurate pitch and roll estimation, and subsequently deteriorates the BEV generation and odometry estimation.

Original: Next, we restore O t withthe learned matrix and replace Q t with an empirical covariance matrix as [17], and note it as “w/o Q t ”.
Revised: Next, we restore O t with the learned matrix and replace Q t with an empirical covariance matrix as [20], and note it as “w/o Q t ”.

Original: The quantitative result also verifies that trainable covariance matrices for noise help to improvethe performance of UKF.
Revised: The quantitative result also verifies that trainable covariance matrices for noise help to improve the performance of UKF.

Original: However, even though the performances are degraded with empirical parameters, they are still competitive due to the dense matching of DPC who seeks global optima.
Revised: However, even though the performances are degraded with empirical parameters, they are still competitive due to the dense matching of DPC who seeks global optima.

Original: Wefurther study the importance of DUKF on the BEVO by eliminating the whole DUKF and feedingthe raw pitch and roll (directly calculated from the accelerometer) to the BEV projection.
Revised: We further study the importance of DUKF on the BEVO by eliminating the whole DUKF and feeding the raw pitch and roll (directly calculated from the accelerometer) to the BEV projection.

Original: We noteit as “w/o DUKF”.
Revised: We note it as “w/o DUKF”.

Revised: [Table]

Original: The quantitative result shown in Table.
Revised: The quantitative result shown in Table.

Original: 3 proves that when the filter is gone, the BEVO will encounter serious tracking failure in bumpy roads.
Revised: 3 proves that when the filter is gone, the BEVO will encounter serious tracking failure in bumpy roads.

Original: One more ablation study is conductedby replacing the output pitch and roll of DUKF with the ground truth, denoted as “w/ GT”.
Revised: One more ablation study is conducted by replacing the output pitch and roll of DUKF with the ground truth, denoted as “w/ GT”.

Original: The result that BEVO is comparable with “w/ GT” indicates that by learning the covariance matrices, theerror brought by DUKF is too small to influence the odometry than the error by DPC, showing theeffectiveness of the training.
Revised: The result that BEVO is comparable with “w/ GT” indicates that by learning the covariance matrices, the error brought by DUKF is too small to influence the odometry than the error by DPC, showing the effectiveness of the training.

Original: 3.4 Limitation
Revised: 3.4 Limitation

Original: Though being competitive in odometry estimation for most of the ground mobile robots, BEVO,limited by the assumption of constant elevation, is yet hardly applicable to robots that move withlarge elevation changes, e.g. drones.
Revised: Though being competitive in odometry estimation for most of the ground mobile robots, BEVO, limited by the assumption of constant elevation, is yet hardly applicable to robots that move with large elevation changes, e.g. drones.

Original: However, since the phase correlation in Section 2.3 is ableto estimate the scale changes along with the rotation (yaw) between images, it should potentiallyserve as a way to calculate the relative elevation changes.
Revised: However, since the phase correlation in Section 2.3 is able to estimate the scale changes along with the rotation (yaw) between images, it should potentially serve as a way to calculate the relative elevation changes.

Original: We will consider such scale for elevationestimation in future works and hopefully make BEVO compatible in all scenarios.
Revised: We will address the above limitations in future works and hopefully make BEVO compatible in all scenarios.

Original:  Conclusion
Revised:  Conclusion

Original: We present a fully differentiable, interpretable and lightweight model for monocular VIO, namely,
Revised: We present a fully differentiable, interpretable and lightweight model for monocular VIO, namely,

Original: BEVO .
Revised: BEVO .

Original: It comprises a UKF to denoise pitch and roll via trainable covariance matrices, a BEV projection, and phase correlation to estimate the SIM (2) pose on the BEV image.
Revised: It comprises a UKF to denoise pitch and roll via trainable covariance matrices, a BEV projection, and phase correlation to estimate the SIM (2) pose on the BEV image.

Original: The differentiabilityallows us to train the covariance matrices end-to-end, thus avoiding empirical parameter tuning.
Revised: The differentiability allows us to train the covariance matrices end-to-end, thus avoiding empirical parameter tuning.

Original: Invarious experiments, BEVO presents a competitive performance in accuracy and generalization.
Revised: In various experiments, BEVO presents a competitive performance in accuracy and generalization.

Original: Webelieve our interpretable, simple approach provides an alternative perspective of traditional methodsand should be considered as a baseline for future works on learning-based monocular VIO. 
Revised: We believe our interpretable, simple approach provides an alternative perspective of traditional methods and should be considered as a baseline for future works on learning-based monocular VIO. Acknowledgments

Revised: This work was supported in part by the National Key R&D Program of China (Grant

Revised: No.2020YFB1313300), Zhejiang Provincial Natural Science Foundation of China (LD22E050007), and Science and Technology on Space Intelligent Control Laboratory (2021-JCJQ-LB-010-13).

Revised: We also thank reviewers who gave useful comments.