IMU-Assisted Learning of Single-View Rolling Shutter Correction
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Rolling shutter distortion is highly undesirable for photography andcomputer vision algorithms ( e.g. , visual SLAM) because pixels can be potentiallycaptured at different times and poses.
In this paper, we propose a deep neural network to predict depth and row-wise pose from a single image for rolling shuttercorrection.
Our contribution in this work is to incorporate inertial measurementunit (IMU) data into the pose reﬁnement process, which, compared to the stateof-the-art, greatly enhances the pose prediction.
The improved accuracy and robustness make it possible for numerous vision algorithms to use imagery capturedby rolling shutter cameras and produce highly accurate results.
We also extend adataset to have real rolling shutter images, IMU data, depth maps, camera poses,and corresponding global shutter images for rolling shutter correction training.
Wedemonstrate the efﬁcacy of the proposed method by evaluating the performanceof Direct Sparse Odometry (DSO) algorithm on rolling shutter imagery correctedusing the proposed approach.
Results show marked improvements of the DSOalgorithm over using uncorrected imagery, validating the proposed approach.
Introduction
Computer vision is one of the most commonly-used sensing modalities in robotics, empowered bythe low cost, low power consumption of cameras, and the rich information they provide.
For the underlying image readout mechanism, cameras either use a global shutter ( GS ) or rolling shutter ( RS )system.
The GS cameras capture the entire image at once, while the RS cameras capture the imagerow by row.
As the camera can move arbitrarily when capturing the image for RS cameras, the pixelsin different rows can be recorded at different camera poses, which leads to the RS distortions.
Mostvision algorithms are incapable of accounting for RS distortions and consequently perform poorlyin their presence.
However, most consumer-grade cameras ( e.g. , smartphone cameras) are RS systems, which creates a challenge in making vision-based computing more ubiquitous.
Even though some vision algorithms can be tuned for RS cameras, the adaptation process is non-trivial.
For visual SLAM systems ( e.g. , [1]), intermediate poses are usually interpolated for features in differentimage rows.
RS distortions can thus be signiﬁcantly detrimental to visual SLAM performance.
Alternatively, the RS images can be rectiﬁed before routing into the vision algorithms.
Such RScorrection algorithms can be categorized into multi-view methods [2, 3] and single-view methods[4, 5].
Traditionally, multi-view methods explore the relative geometry between images for RScorrection; while the single-view methods rely on special features ( e.g. , straight lines in [5]) dueto the lack of geometric information.
With the emergence of deep learning, several research workshave addressed the problem of single-view automatic RS correction and have reported inspiringresults [6, 7].
However, the accuracy and robustness of the existing methods for single-view RScorrection are limited by the ill-posed nature of the problem.
Besides, these approaches also adoptsimpliﬁed formulations by making various assumptions on the camera motion such as restrictingit in two-dimensions in [6] and assuming a constant velocity in [7].
Nevertheless, single-view RScorrection is more appealing in real-world applications for its input simplicity.
Moreover, the RStwo-view geometry degenerates when the camera motion is pure translational [7], which is notuncommon in many robotic applications.
In this paper, we propose a novel deep neural network for correcting RS distortions; an overview ofthe system is illustrated in Fig.
In the proposed network, we estimate the depth of each pixel inan RS image and also recover its row-wise camera poses, which we subsequently use to reconstructthe corresponding GS image.
This two-step process is analogous to the one by Zhuang et al.
However, [7] incorporates a constant velocity assumption on camera motion which is not alwaysvalid in natural RS images (see Sec. 4.1); we propose to predict a camera pose for each image row toeliminate such assumption.
To improve the accuracy and robustness of single-view RS correction,we propose to integrate the inertial measurement unit (IMU) data for pose prediction.
An IMUestimates angular velocity and linear accelerations and is widely used in SLAM domain ( e.g. , [8, 1])as a complementary sensor to cameras.
IMUs provide high-frequency motion cues that can help RScorrection, and consequently, offers signiﬁcantly more learning capacity for deep RS correction inmore realistic scenarios.
To train and test the proposed neural network, we generate a dataset fromthe TUM rolling shutter dataset [8], including RS images, IMU data, depth maps, row-wise cameraposes, and corresponding GS images.
Training the proposed neural network using this dataset,we show that RS distortions are correctly removed and the vision algorithm ( i.e. , DSO [9]) worksaccurately on the resulting images.
In summary, the contributions of this work are the following: ( i ) a deep neural network that learnsfrom single-view images and IMU data for accurate and robust RS correction; ( ii ) a novel datasetwith real images and data for RS correction training; and ( iii ) extensive experimental validations ofthe proposed dataset and deep neural network for RS correction.
Our implementation and datasetwill be released after the double-blind review.
Related Work
Removing RS distortions from images has several beneﬁts as discussed in the previous section, andit is useful in computer vision, robotics, consumer mobile photography, etc.
Several neural networks have been proposed recently addressing the problem of single view RScorrection.
Rengarajan et al.
[6] proposed a convolutional neural network architecture to predictthe camera motion for RS correction.
The predicted motions are up-sampled using a polynomialfunction to get row-wise components of camera motion, which is subsequently used to correct the
RS distortion.
This approach is feasible because the camera motion is limited to two dimensions( i.e. , x -axis translation and z -axis rotation).
The mapping function relating the coordinates of the
RS pixel to its undistorted correspondence is fully determined by this 2D motion.
The work by
Zhuang et al.
[7] eliminates the 2D motion assumption above and enables full (6D) camera motion.
They proposed the VelocityNet to predict the camera velocity, which is used to calculate the camerapose for each image row with the constant velocity assumption.
Meanwhile, they adapt DispNet [10](referred to as DepthNet) to predict pixel-wise depth in the RS image.
With the camera pose and thedepth, the undistorted GS image is generated by projection.
Other than single-view approaches, Liu et al.
[3] proposed a network to remove RS distortion usingtwo consecutive frames.
They propose a motion estimation network to compute the cost volumes between two consecutive frames, which are used to predict the pixel displacement between RSimage and GS image.
Karpenko et al.
[11] proposed to use gyroscopes for video stabilization and
RS correction based on classic geometry-based computer vision.
It is challenging to get a large set of GS/RS image pairs for RS correction training.
Not only the
GS/RS cameras need to have identical conﬁgures ( e.g. , intrinsic parameters), but also they have tobe co-located, which is mostly impossible in real world.
Existing works use synthesized imagesinstead.
Rengarajan et al.
[6] generate RS images from GS image datasets.
For each GS image, theygenerate a random 2D camera motion and construct a mapping function to get the corresponding RSimage.
Subsequently, they train their neural network to perform the inverse action.
Zhuang et al.
[7],in the same vein, use a stereo GS image dataset to generate RS images.
For each GS image pair,they run stereo matching to get a depth map; with a randomly-generated 6D camera velocity, theysynthesize an RS image under the constant camera velocity assumption.
Liu et al.
[3] proposed twodatasets for RS correction.
One is from simulation; in the other dataset, the RS image is synthesizedby sequentially copying a row of pixels from consecutive GS images captured by a high-speed (
FPS) camera, which is super expensive and its recording time is short.
The TUM rolling shutter dataset [8] has been proposed to benchmark visual-inertial SLAM with RScameras, captured by the device shown in Fig.
The device is equipped with a pair of synchronized
RS/GS cameras, an IMU, and a motion caption system recording ground-truth poses.
However, theconstant displacement between the RS camera and the GS camera prevents us from directly usingthe dataset for RS correction training, which we are going to solve in the following section.
[Figure]
Methodology
[Figure]
3.1 Dataset Generation
[Figure]
Currently, the TUM rolling shutter dataset [8] is the only publicly available dataset with synchronized RS/GS image pairs, camera poses, and IMU data.
The RS/GS images are captured at 20 Hzwith 1280 × 1024 resolution; as shown in Fig.
2, we mark the GS camera as cam 0 and the RScamera as cam 1 .
The time difference between two consecutive rows in an RS image is reported as29 .
4737 µs .
The ground truth poses are recorded by a motion capture system running at 120 Hz.
The
IMU runs at 200 Hz.
Please refer to [8] for more details about this dataset.
The RS images captured by cam 1 and IMU data are used as inputs to our neural network.
However, we cannot directly use the GS images from cam 0 as ground truth because there is a constantdisplacement between the cameras.
Hence, we recover the GS images at the location of cam 1 .
Forclarity, we mark the GS images captured from cam 0 as GS 0 images and the recovered GS images as
GS 1 .
As seen in Fig.
2, the idea is to recover a depth map from the stereo conﬁguration and projectpoints to the pose of the ﬁrst image row of cam 1 to get the GS 1 image.
Recovering depth from a GS 0 image and corresponding RS image is more challenging than froma pair of GS images.
The rolling shutter distortion breaks the stereo epipolar constraint; even afterstereo rectiﬁcation, the pixel correspondences are not necessarily located in the same row, whichmakes conventional stereo matching algorithms ( e.g. , [12]) inapplicable.
Hence, we abandon the stereo constraint and adopt PWC-Net [13], the state-of-the-art optical ﬂow algorithm, to ﬁnd theoptical ﬂow between the GS 0 /RS image pairs.
For robustness, we ﬁlter the correspondences bystandard bi-directional matching.
In addition to the pixel correspondence, the relative camera poses are also required for depth recovery.
This is because the rolling shutter mechanism makes the underlying camera poses nolonger consistent for all pixels in the RS image.
Unfortunately, the motion capture system is notefﬁcient enough to capture a pose for every image row, even after we downsize the image resolution to 320 × 256 in this work.
Since the motion capture system runs at 120 Hz and it takes1024 × 29 .
µs = 30181 .
1 µs to read out the entire RS image, we have about 3 .
6 poses forall 256 rows in the RS image, which is too sparse.
To solve this problem, we adopt cubic splineinterpolation [14] on the sparse poses to get a smooth pose for each row in the RS image.
Since lensdistortion is corrected during image preprocessing, we maintain a distortion lookup table to ﬁnd theoriginal scan-line when querying the pose for each pixel; that is, each transformation matrix ( T ) inthe following equations already considers the lens distortion lookup table implicitly.
With the pixel correspondence ( u GS 0 , u RS ) from PWC-Net and the row-wise camera pose T GS
u RSfrom cubic spline interpolation, we get the depth d RS and corresponding 3D point X RS by solvingthe following triangulation problem:
[Equation]
Here, K RS and K GS 0 are known camera intrinsic parameters.
We also considered the depth ﬁlter [15] for further reﬁnements; however, the improvement margins were insigniﬁcant, and hence itis omitted in our ﬁnal implementation.
Finally, we recover the GS 1 image ( u GS 1 ) by projecting the3D points X RS to the pose of the ﬁrst row in RS frame ( i.e. , T GS 1 u RS =
T RS row 0 u RS ) asd GS(cid:20) u GS (cid:21) =
K RS T GS 1 u RS (cid:20) X RS (cid:21) .
3.2 Network Architecture
[Figure]
As mentioned earlier, our training pipeline for learning rolling shutter correction is inspired by Zhuang et al.
We outline the end-to-end pipeline in Fig.
Here, we adapt the
DispNet [10] as our RsDepthNet ( i.e. , DepthNet in [7]) for depth estimation.
To predict row-wiseposes ( i.e. , the T GS 1 u RS in Eq. 1), we propose a novel deep visual model named RsPoseNet ; its detailed architecture is illustrated in Fig.
First, we to high-
PoseConv layers to
high-level features and learn row-wise poses
PoseConv convolution layer
Following that, we use a convolution layer and a tanh activation layer to ﬁnalize the poses.
We usetanh as the ﬁnal
We will show empirically in Sec. 4.3 that predicting row-wise poses purely from images is not veryaccurate; hence, we propose to reﬁne the poses by using IMU data.
For uniﬁed input, we rotate the
IMU data to the RS camera frame; we also interpolate the IMU data for each image row since IMUdata is sparse running at a different frequency.
The processed IMU data is then concatenated withthe predicted poses from the image and fed into the 2-layer LSTM [19] for the ﬁnal row-wise poses.
3.3 Training Details
Since our goal is to enable vision algorithms ( e.g. , visual SLAM) on RS cameras, we adopt the endpoint error (EPE) in [7], a standard geometric loss, for training and validation.
The EPE measuresthe average Euclidean distance between the distorted and undistorted pixels.
It is deﬁned as
[Equation]
Here, for each pixel u in the input RS image I , we project it to the corresponding GS frame in Eq.
3,which is analogous to Eq.
but using predicted pose T (cid:48) .
We compare the projection with groundtruth u GS 1 ( u ) to get the EPE.
Note that the d here is the ground-truth depth, which decouples thetraining of RsDepthNet and RsPoseNet.
We initialize the ResNet-34 in RsPoseNet with the pre-trained weights from ImageNet classiﬁcation [20].
We empirically ﬁnd that training directly using EPE does not yield optimal results, hencewe initialize RsPoseNet using pose loss (mean squared error of the row-wise poses) for 50 epochswith a learning rate of 1 e − 3 ; then we reﬁne RsPoseNet using the EPE for another 50 epochs with
e − 4 learning rate.
Pose loss is straightforward for training, but a minimized pose loss does notguarantee a minimized EPE because rotation and translation play different roles in projection.
Reﬁning RsPoseNet with EPE optimizes the weights in each dimension of poses to minimize the EPE,which is meaningful and necessary.
Experimental Evaluation
By applying the data generation process described in Sec.
3.1 on the TUM rolling shutter dataset [8],we get 10 sequences of RS images with corresponding IMU data, depth maps, row-wise cameraposes, and recovered GS 1 images.
There are 9495 frames in total, which is slightly less than the totalnumber of frames ( 9523 ) in the TUM rolling shutter dataset, because some frames at the beginning orend of a few sequences were not recorded by the motion capture system, hence they were discarded.
4.1 Data Veriﬁcation
In the evaluation, we quantify RS distortion by the EPE formulations deﬁned in Eq.
2 and Eq.
Since EPE is evaluated with respect to the GS 1 images generated by the proposed method, we needto verify the validity of GS 1 images.
To do so, we run DSO [9] on a continuous sequence of images;being a direct method for visual odometry, DSO is very sensitive to RS distortion.
Fig.
4 showsthe qualitative results for a particular sequence (#02) in the TUM dataset; as the top-left block (RS)illustrates, DSO completely fails on the RS images with an EPE of 5 .
738 pixels.
The scene in thebottom-left block (GS 1 gt) is reconstructed by DSO on our recovered GS 1 images, which clearlyshows the accurate room layout; EPE for this case is zero by deﬁnition.
Moreover, we run DSO 10times and calculate the average absolute trajectory error [21] (ATE) with Sim (3) alignment.
The
ATE on the RS images is 0 .
269 meters, while it reduces to 0 .
047 meters on the GS 1 images.
Thesequalitative and quantitative results validate our data generation process and justify the use of EPE asan evaluation metric.
We also generate data with the constant velocity assumption as in [7]; here weverify that its EPE is 0 .
228 pixels.
The reconstructed scene in the top-right (GS 1 gt cv) of Fig.
4is not as good as the one without such assumption, which is further validated by the fact that
its
ATE= 0 .
157 meters.
This shows that a constant velocity assumption is sub-optimal in real-worldsettings.
Table 1 quantitatively summarizes these experimental results.
[Table]
4.2 Network Evaluation
We split the dataset as follows for training and testing: we use sequence 02 and sequence 07 fortesting; the remaining 8 sequences are used for training and validation, among them, we reservethe middle 10% data for validation and the rest for training.
Consequently, we have 7362 framesfor training, 815 frames for validation, and 534 + 784 = 1318 frames for testing.
Our networkimplementation is based on Keras [22] and trained on a Nvidia™ GTX Titan V GPU with 12 GBmemory.
We mainly focus on [7] (denoted as SMARSC) for comparison with the goal of showingthe contribution of IMU data for RS correction.
The SMARSC is also trained with the EPE.
Wealso ﬁne-tune the state-of-the-art two-view learning-based RS correction approach [3] (denoted as
TwoView) on our dataset for comparison.
However, this approach outputs the corrected GS imagedirectly so that we cannot accurately calculate EPE; thus we include it for qualitative comparison.
[Table]
EPE
[Table] [Table]
Ratio -
[Table]
EPE
[Table] [Table]
Table 2a reports the performance comparisons on our test data.
The term Ratio represents the percentage of testing images whose RS distortion (measured by EPE) gets reduced.
For SMARSC [7],the EPE of sequence 02 is reduced by half and it predicts RS distortion in the correct direction formore than 85% of the tests.
It shows that learning RS correction from single-view images is feasible.
As for the proposed method ( i.e. , ‘Ours’), it predicts RS distortion in the correct direction for more than 95% of the tests with sub-pixel EPE, which is a signiﬁcant improvement over SMARSC [7].
As we will discuss in Sec. 4.3, the performance improvement mainly comes from the IMU data.
Wepresent some qualitative results of RS correction by the proposed system in Fig.
5a.
The predicted
GS images look very similar to the ground truth images, as most of the visible RS distortions areremoved.
For TwoView [3], even though it predicts the correct direction of RS distortions, the corrected results introduce new undesired image distortions.
Fig.
5b conﬁrms that the new distortioncomes from the TwoView [3] itself instead of the process of ﬁne-tuning on our dataset.
[Figure]
Application Study We evaluate the performance of DSO on GS images recovered by our method.
The reconstructed scene is shown in the bottom-right of Fig.
4; it is as good as the one on groundtruth GS 1 images and signiﬁcantly better than the one on the original RS images.
We get the average
ATE of 0 .
064 meters, which is much smaller than the ATE on RS images ( 0 . 269 meters) and veryclose to the ATE on ground-truth GS 1 images ( 0 . 047 meters).
From the quantitative and qualitativeresults, we show that the proposed RS correction network enables downstream vision algorithms(DSO in this case) to run on RS cameras without any modiﬁcation.
We also run DSO on the resultsby SMARSC [7] and by TwoView [3] but DSO fails consistently.
For computational efﬁciency,the network inference is able to run in real-time ( ∼ 60 FPS on our Titan V GPU), including theprojection of pixel from the input RS image to the rectiﬁed GS image ( i.e. , Eq. 3).
However, sincethe projection is not surjective [23] (not all GS pixel is guaranteed to be ﬁlled), the output GS image is incomplete as shown in Fig.
5c.
We apply interpolation to complete the output image, with Fig.
5ashowing the results after interpolation.
As interpolation is computationally expensive ( ∼ 20 FPS on
CPU), we will explore alternatives ( e.g. , median ﬁlter) for higher computational efﬁciency in futurework.
4.3 Ablation Study on IMU Data
To analyze the contribution of IMU data for RS correction, we disable the gyroscope, or the accelerator, or both when feeding the IMU data into the network; we re-train the network and report theresults in Table 2b.
Using gyroscope alone ( i.e. , ‘Gyr.’) already yields good results, while integratingthe accelerator data ( i.e. , ‘Full’) further reduces the EPE slightly.
However, the accuracy of usingaccelerator data alone ( i.e. , ‘Acc.’) is low.
Gyroscope measures the camera motion more directlythan accelerator does (angular velocity versus linear acceleration), thus gyroscope being more important for RS correction is expected.
Nevertheless, even when the IMU data is completely disabled( i.e. , ‘None’), it still slightly outperforms SMARSC [7].
The potential reason is that the row-wiseposes are highly correlated with the constant velocity assumption so that the velocity error affectsevery pixel, whereas row-wise poses are predicted individually in our method.
4.4 Generalization Performance
We also test our network on the WHU RS Visual-Inertial Dataset [24], which contains RS/GS imagepairs, IMU data, and ground-truth poses.
We use the training results on our proposed dataset withoutﬁne-tuning on the WHU dataset to test the generalization performance of our RS correction network.
Fig.
6 gives some qualitative results.
One important observation is that when the camera rotates, ournetwork works most of the time, with the predicted GS images well-aligned to the ground-truth GSimages as illustrated in the left part of the ﬁgure; however, for pure translational motion, our methodis more likely to fail ( e.g. , right part of Fig. 6).
The potential reason is that pure translational motionis very rare in our training dataset from the TUM RS dataset.
For future work, we plan to increasegeneralization capability by augmenting the training set with more diverse data.
Conclusions
We propose a novel neural network for single-view rolling shutter correction, where we use IMUdata to signiﬁcantly improve rolling shutter pose prediction.
Rather than synthesizing rolling shutter images, we generate real images to train the proposed network.
Our experiments validate theimproved accuracy and robustness in rolling shutter correction and downstream vision algorithmsby using the proposed approach.
For future work, we will extend the proposed method for diverseenvironments by designing our own data acquisition device.
We also intend to deploy the proposedapproach for vision-based robot localization and odometry purposes while using inexpensive andubiquitous rolling shutter cameras.