Learning Eye-in-Hand Camera Calibration from a Single Image
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Eye-in-hand camera calibration is a fundamental and long-studiedproblem in robotics.
We present a study on using learning-based methods forsolving the problem online from a single RGB image, whilst training our modelswith entirely synthetic data.
We study three main approaches, one using a directregression model predicting the extrinsic matrix directly from images, one sparsecorrespondence model regressing 2D keypoints and using PnP, and one dense correspondence model that uses regressed depth and segmentation maps to enable ICP pose estimation.
In our experiments, we benchmark these methods againsteach other and well established classical methods to ﬁnd the surprising resultthat direct regression outperforms other approaches, and perform noise-sensitivityanalysis to gain further insights into those results.
Introduction
[Figure]
Eye-in-hand setups, where images from a camera rigidlymounted to the end-effector are used to control a robot,are a popular choice for precise robot manipulation.
Compared to control using a ﬁxed external camera, theeye-in-hand setup enables precise control because imagescan be captured very close to an object, as well as generalisation across the workspace when actions are deﬁnedrelative to the camera or end-effector [1, 2, 3].
To apply to precise manipulation tasks, these methods requireaccurate eye-in-hand extrinsic calibration, to ensure thatactions or poses inferred in the camera’s frame can betransformed to the robot’s frame.
Whilst calibration-freeapproaches exist, such as end-to-end control [4, 5], these are only suitable for model-free policylearning, and cannot be used for methods that involve explicit planning or reasoning about 3D space.
Classically, solving the eye-in-hand calibration problem is performed off-line [6, 7, 8, 9] and involves the use of a speciﬁc external calibration object, such as a checkerboard or an AprilTag.
Posesof the calibration object can be estimated by the camera, and by recording them from several robotconﬁgurations, the rigid-body transformation between the camera and end-effector can be estimatedanalytically.
Although with perfect information these methods can yield exact calibrations, theyare in practice limited by how precisely the pose of the calibration object can be estimated, andby the accuracy of the recorded end-effector poses.
Furthermore, these methods are unsuited foruse by non-experts since they require dedicated external calibration objects and a new calibrationdataset to be collected for each re-calibration. Finally, they are impractical as re-calibration needsto be performed every time the camera position changes due to (1) re-mounting, (2) wear and tear,and (3) collisions between the camera and the environment.
Deep learning methods can alleviatethese drawbacks. Given the initial overhead required to generate training data and train a model, adeep learning-based calibration method can be readily used to re-calibrate a camera when necessary. Using parts of the robot as the calibration object, they have the potential to perform calibration on- line from a single image without the use of an external calibration objects. This potential has beendemonstrated for eye-to-hand camera calibration, where an external camera observes the robot froma distance [10, 11].
To the best of our knowledge, there are currently no studies on using deep learning methods foreye-in-hand calibration, which is the focus of this paper.
We propose that this calibration can beestimated directly from the camera’s image itself, without requiring any external apparatus, as longas the image can observe part of the end-effector, such as the gripper’s ﬁngers (see Fig. 1).
Based onthis, we identify and benchmark three natural approaches for leveraging the power of deep learningin this way: two based on classical, analytical/geometric calibration where deep learning is used inplace of components which are typically manually engineered, and one which uses deep learning todirectly regress the camera calibration matrix.
We show in our real-world experiments that the twoapproaches that rely on geometric analysis perform poorly, and we analyse empirically the structurallimitations and challenges that cause this.
We also show that, perhaps surprisingly, direct regressionoutperforms all alternatives, including even classical methods based on ﬁducial markers that collectdata automatically.
Finally, given that direct regression can estimate the calibration matrix from asingle image online, we propose and evaluate a method for fusing multiple estimates of the camerapose, to increase the accuracy of the aggregated estimate.
As such, our work has three key contributions.
(1) We investigate the suitability of three naturalalternatives of using deep-learning for enabling eye-in-hand calibration from a single image, (2) We show the potential of end-to-end deep learning for online eye-in-hand camera calibration ineveryday environments, outperforming typical classical approaches that collect data automatically,and (3) We provide analysis and insights for the two learning-based methods that rely on geometricanalysis, through which we discover why, although useful for calibration with an external camera,these methods are not suitable for eye-in-hand calibration.
A video and supplementary material forour paper can be found on our website: https://sites.google.com/view/eye-in-hand .
Related Work
Eye-in-hand calibration is a long-studied problem, with signiﬁcant advances introduced in the 1990’swith what are now well-established solutions.
The problem consists of inferring the camera to endeffector pose for a camera mounted on a wrist of a robotic manipulator.
To do so, several end-effectorto robot base poses are recorded, along with corresponding estimates of calibration object to cameraposes.
The problem is then reduced to solving the AX =
XB equation through a formalismintroduced by Shiu and Ahmad in [12].
Several works were then followed that mainly differ intheir strategy for solving the AX =
XB equation.
For instance, Tsai and Lenz [7] improve on theefﬁciency of Shiu and Ahmad’s solution by proposing a closed-form solution.
Park and Martin [8]and Dornaika and Harod [9] also consider such a closed-form solution while relying on Lie theoryand unit quaternions respectively.
While the above approaches solve for rotation and translation separately, Dornaika and Harod [9]also propose a non-linear technique for solving for both simultaneously.
Daniilidis proposes a solution to the problem using dual quaternions [6].
Finally, more recent works study various extensionsfor particular settings such as using structure for motion for calibration [13], using model-basedpose estimation and tracking for online calibration [14], simultaneously considering the data timesynchronisation problem [15], calibrating depth sensors with non-overlapping views [16], automatically doing both calibration pattern localisation and eye-in-hand calibration [17], and optimising therobot kinematics parameters and the camera to end-effector pose simultaneously [18].
Recently, deep learning methods have had great successes on closely related tasks such as pose estimation [19, 20] and end-to-end robotics manipulation from wrist-mounted cameras [4, 3].
Moreover,sim-to-real transfer has shown promise in alleviating the large data requirements that hinder the scalability of such methods [4, 21, 22, 23].
In this work, we investigate the naturally emerging questionof how one might use such methods for eye-in-hand calibration, which is fundamental to a largenumber of robotics pipelines.
Closest to our work, Lee et al.
[11] and Labbe et al.
[10] investigatetwo different deep learning approaches for eye-to-hand calibration of a camera looking straight at arobot.
[11] is based on keypoint regression followed by Perspective-n-Point (PnP) [24] while [10]adopts a render and compare approach.
In our work, we consider the eye-in-hand setup that is morecommon when precise manipulation needs to be achieved by looking closely at the end-effector Dense Correspondence Model ImageEncoder x z y aa 6 ... [Figure]
ICP initialisation estimate
ICP Pose estimation Intrinsics
Camera to End effector transformation ImageEncoder ImageDecoder SpatialSoftArgmax 2D Keypoint Locations 3D Keypoint Locations 2D - 3D Correspondances
PnP & Ransac
Camera to End effector transformation
Intrinsics
U n e t s k
i p c onn ec ti on s Fully ConnectedLayers
End Effector Model
Segmented Depth Image
ImageEncoder
ImageDecoder
U n e t s k
i p c onn ec ti on s
ImageEncoder
ImageDecoder
U n e t s k
i p c onn ec ti on s Direct
[Figure]
of the robot [4, 25].
This problem also differs in its properties from the eye-to-hand setup, sincein a typical image there is now a much more constrained view that only allows for the tip of theend-effector to be visible and to be used as an anchor for pose estimation.
Methods
In this section, we present three methods which are conceptually promising for eye-in-hand calibration from a single RGB image, and which also raised interesting and novel scientiﬁc questionsthat we answer in sections 4 and 5.
The ﬁrst method is an end-to-end direct regression method thatestimates the pose of the camera from an RGB image, while the other two methods use deep learning to provide key missing components for well established geometric pose estimation approaches.
The latter has been shown to work well in other problem settings [11, 26, 27, 28], but eye-in-handcalibration has the particularity that not only the camera is too close to the end-effector for typicalvision sensors to capture a depth image of it, but also the visible geometry varies widely withinour inference space.
To test whether it is still feasible to use such approaches, our second methoduses deep learning to regress 2D locations of predeﬁned keypoints on the end-effector, that are thenpassed into the PnP [24] algorithm to solve for the pose of the camera.
The main challenge withthis method is that the large variations in the visible geometry lead to keypoint occlusions and somekeypoints being often outside of the image frame.
This is illustrated in Fig.
Our ﬁnal method testswhether we can alleviate this pitfall by using deep learning to regress a segmented depth image ofthe end-effector and an initial guess of the camera pose, and then reﬁne this initial estimate with the Iterative Closest Point (ICP) algorithm [29, 30, 31].
All three methods are illustrated in Fig.
3.1 Problem Setting
Our problem setting consists of the typical eye-in-hand calibration problem, where the aim is to estimate the camera’s extrinsic matrix, which is the camera to end-effector pose, T EC =
[ R EC | t TC ] ∈
SE (3) , where R EC ∈ SO (3) and t EC ∈ R 3 is the orientation and position of the camera in theend-effector frame, respectively.
We also deﬁne the end-effector to robot base pose as T BE , thecalibration object to camera pose as T CO , and an image captured by the wrist-mounted camera as I .
Throughout this paper, we use ˜ · to denote an estimated quantity.
As opposed to classical approaches, we constrain ourselves to eye-in-hand calibration from a single
RGB image without any external apparatus, enabling our methods to be readily deployed in the wildfor online eye-in-hand calibration.
We further constrain ourselves to using only synthetic trainingdata.
This has the beneﬁt of alleviating the costly data requirements of deep learning approacheswhile also providing us with ground truth labels that enable our proposed methods to learn effec- [Figure]
tively. Finally, we assume that we have an accurate estimate of the camera intrinsic matrix, which istypically provided by the manufacturers, and access to the CAD model of our robot’s end-effector.
3.2 Direct Regression Model
Direct regression (DR) is the simplest of the methods we consider.
As illustrated in the right diagramof Fig. 2, a neural network (NN) is tasked with regressing the camera pose parameters from a single
RGB image and is trained in simulation using ground truth labels in a supervised setting.
The onlyconstraint we introduce into the problem comes from the parameterisation of the orientation, wherewe use the 6D rotation encoding introduced in [32].
Overall, our network outputs are 9 dimensional, with 3 dimensions representing position and 6 theorientation.
It consists of a convolutional encoder followed by linear layers that regress the camerapose.
We trained the network on the ground truth labels using the Mean Squared Error (MSE) loss,and we fully breakdown the architecture and training details in our supplementary material.
3.3 Sparse Correspondence Model
Our sparse correspondence model (SC) is illustrated in the middle of Fig.
Starting from an RGBimage, it uses a U-net [33] type architecture to regress a set of 2D keypoints in image space.
The
Unet architecture downsamples the input image into some abstract feature space and then upsamplesit back to the original resolution, while providing skip connections through concatenation at eachresolution level [33].
On the output layer we apply a spatial-soft-argmax [34] activation in orderto regress 2D image keypoints.
We use the sum of an L1 and MSE losses to train our network onthe ground truth labels, where each dimension of the output predicts the 2D location of a predeﬁned3D keypoint. A detailed breakdown our architecture and training parameters can be found in oursupplementary material.
At test time, we use the network outputs, the corresponding 3D keypoint locations deﬁned inthe object’s model, and the camera’s intrinsic matrix to estimate the camera pose with PnP and
RANSAC [24], which is a popular outlier rejection algorithm.
Finally, in order to alleviate the issue of some keypoints being out of frame depending on the viewpoint, we (1) consider 38 keypoints that are present in at least 70% of the training images, (2) forany keypoint that remains out of frame, we train the model to predict its projection in the image.
This way, at test time, we can infer which keypoints are out of frame by looking at the predictionsat the rim, and ignore these predictions for the PnP pose estimation.
3.4 Dense Correspondence Model
Our Dense Correspondence Model (DC) is illustrated on the right of Fig.
Starting from an RGB
Image, it uses three NNs that are trained independently to estimate the pose of the camera in twostages.
In the ﬁrst stage, the three NNs are used to predict (1) a depth map, (2) a segmentationmask of the gripper, and (3) an ICP initialisation.
In the second stage, (1) the segmentation maskis used to segment the depth map to only return depth values on gripper pixels, (2) the cameraintrinsic matrix is used to project the segmented depth map to a point cloud, and (3) ICP is usedto approximate dense correspondences between the estimated point cloud and the model of theend-effector in order to reﬁne the camera pose estimate from the ﬁrst stage.
The network used for initialisation is the same as the one used for direct regression, and the same UNet architecture asfor the sparse correspondence model is used to predict the depth and segmentation maps.
See thesupplementary material for further architecture design choices and training details.
3.5 Fusing Multiple Estimates
Compared to classical eye-in-hand calibration methods, which require calibration to be done ofﬂineand with special apparatus, the proposed deep learning alternatives can estimate the pose of thecamera online in unstructured environments, such as in the middle of a manipulation task.
Thiscreates the possibility of fusing multiple estimates to increase the overall calibration accuracy.
Inorder to explore this, we implemented a simple aggregation algorithm that rejects 20% of the leastlikely samples from a set of candidate estimates, under a Gaussian data model.
It then averages theremaining samples together to yield the ﬁnal estimate.
Pseudocode for this procedure is available inour supplementary material.
3.6 Dataset Generation and Sim-to-Real Transfer
We train our networks entirely with simulated data using the Coppelia [35] simulator.
In orderto generate the dataset, we randomise the pose of the camera relative to the end-effector at eachsimulated timestep and record (1) the current RGB, depth and end-effector segmentation images,(2) the ground truth extrinsic matrix and (3) the 2D keypoint locations in image space.
In order to overcome the ”reality gap” we apply visual Domain Randomisation [21, 23, 36].
Specifically, we (1) randomise the colours/textures of the simulated gripper and the light sources on thesimulator, (2) replace the background of our images with random images of textures and indoorscenes [37, 38], and (3) apply a post-processing colour jitter operation [39], further randomly perturbing the brightness, contrast, saturation and hue of the whole image.
In total, we generated 10000labelled images in approximately 30 minutes, examples of which can be seen in Fig.
Experiments
In our experiments, we evaluated how the different proposed deep learning approaches compare toeach other and to various standard off the shelf calibration methods both in simulation and in thereal world.
From them emerges the surprising result that simple direct regression using end-to-enddeep learning outperforms both all classical approaches tested and the correspondence based deeplearning methods.
As such, we performed a series of analysis experiments that we describe in sec. 5that are aimed speciﬁcally at gaining insights onto this surprising observation.
In our comparisons we include all our deep learning based methods and the following establishedmethods readily available on OpenvCV [40]: Dainiilidis et al.
[6], Tsai and Lenz [7], Doraika and
Harod [9], and Park and Martin [8].
For all experiments, we use the Realsense D435 [41] (or itssimulated counterpart) at a 480 × 848 resolution, and for the deep learning methods we downsamplethese images to a resolution of 144 × 256 .
Comparing classical methods to our deep learning ones fairly can be somewhat challenging sinceclassical methods require a training set of several end-effector to robot and calibration object tocamera poses to perform a single calibration, while our methods only require a single image.
Insections 4.1 and 4.2 we describe our procedures and what we did to mediate this issue, and in oursupplementary material we provide pseudocode for each of the procedures.
4.1 Simulation Experiment
Our simulated environment consists of a wrist-mounted camera attached to a Sawyer robot’s gripperthat is free to move around an AprilTag (see Fig. 3).
To benchmark all methods, we (1) Collect adataset { I i , T iBE , ˜ T iCO } 15 i =1 of images and corresponding AprilTag and end-effector poses, with thetag poses ˜ T CO estimated using the AprilTags3 library [42, 43, 44], (2) Use all of the 15 datapointsin this dataset to estimate the extrinsic matrix using classical methods, and evaluate this estimate,(3)
Use each of the 15 datapoints in this dataset independently to estimate the extrinsic matrix usingthe proposed learned methods, and evaluate each of the single image estimates, and (4) For each [Table] of the learned methods, we fuse together all 15 independent predictions using our fusion proceduredescribed in 3.5 and evaluate the fused estimate.
We repeat this procedure for 100 different ground truth extrinsic matrices, average the results anddisplay them in table 1.
We evaluate translation and rotation independently:
Given an estimate of theextrinsic ˜ T EC =
[ ˜ R EC | ˜ t EC ] , and the ground truth T ∗ EC =
[ R ∗ EC | t ∗ EC ] , we deﬁne the position erroras e t
= || ˜ t EC − t ∗ EC || 2 , where || · || 2 is the L2 norm, and the rotational error as e R =
θ , the anglefrom the axis-angle representation of the rotation matrix R ∆ =
( w, θ ) that satisﬁes the relationship R ∗ EC = R ∆ ˜ R EC , where w is the axis of rotation, and θ is the angle of rotation.
4.2 Real World Experiment
The real world evaluation environment is analogous to the simulated environment and consists of a
Sawyer robot with a wrist-mounted camera moving around an AprilTag (see Fig. 3).
In order to evaluate our methods in the real world we (1) Collect a training data bank D train
= { I i , T iBE , ˜ T iCO }
i
=
1and an evaluation dataset D eval =
{ T iBE , ˜ T iCO } 60 i =1 , where AprilTag poses are estimated using anoff-the-shelf library [43, 44], (2) We sample 40 training datasets of 15 datapoints each from ourtraining data bank, (3) Use all 15 datapoints in each training set to get an estimate of the cameraextrinsic matrix using each classical method, and evaluate that estimate, (4) We use each one of the15 datapoints in each dataset to get an estimate of the extrinsic matrix using our learned methods,and evaluate each single estimate, and (5) For each of the learned methods, we fuse together all 15independent predictions using the procedure described in 3.5 and evaluate the fused estimate.
We repeat this procedure for two different ground truth extrinsic, average the results and displaythem in table 1.
Since ground truth extrinsic values are not available in the real world, we takeinspiration from [6, 7] and use an indirect error metric:
For each corresponding end-effector to robotpose and calibration object to camera pose from the evaluation dataset, { T iBE , ˜ T iCO } ∈ D eval , weestimate the pose of the calibration object in the robot frame, ˜ T iBO = T iBE ˜ T EC ˜ T iCO =
[ ˜ R iWO | ˜ T iBO ] ,where ˜ T EC is an estimate of the camera to end-effector pose that we are evaluating.
We then deﬁnethe error metric (cid:15) std = (cid:16) 1 / 60 (cid:80) 60 i =1 (cid:12)(cid:12) | ˜ t iBO − µ (cid:12)(cid:12) | 22 (cid:17) 1 / 2 as the standard deviation of the estimatedcalibration object position, where µ = 1 / 60 (cid:80)
i =1 ˜ t iBO is the estimated mean object position.
We emphasise that although the AprilTag is visible in all images in both the simulated and real environments, it is never used to help the calibration of our deep learning methods.
We use these imagessimply to ensure that we have the exact same training and evaluation datasets for both classical andour learned methods.
4.3 Results
All benchmarking results are shown in table 1.
Our main observation is that the direct regressionmethod outperforms all others, which is a striking and surprising result.
First, it was not expected tooutperform the classical methods since these use analytical solutions to estimate the extrinsic, which in absence of noise in the system should give perfect calibration.
This raises the question about thesensitivity of these methods to said noise, which we investigate in section 5.
Second, it was also not expected to outperform the correspondence based learned methods, sincethose introduce strong geometric constraints that intuitively should help guide the model to goodsolutions.
We thoroughly investigate the reasons behind this observation in sections 5.2 and 5.3.
Analysis
In this section, we aim to build a deeper understanding of the surprising results that we observedin our experiments, which showed that a direct regression of calibration parameters outperformsmethods which incorporate well-understood, geometric modelling.
We split this section into threeparts, each analysing the performance of one particular method.
5.1 Classical Methods
Sparse Correspondence Model
[Figure]
In order to better understand the results from our experiments in section 4, we ﬁrst aim to assess how sensitiveclassical methods are to sources of noise in the system.
Assuming the main source of error stems from the calibration object pose detection, we ﬁx every other sourceof noise in simulation to the ground truth value.
We thenvary the noise injected to the calibration object’s pose bycontrolled amounts, and observe how this affects the quality of the calibration result from classical methods.
Precisely, the noise scale added is varied from 0 mm to10 mm and 0 ◦ to 10 ◦ , in increments of 0 .
5 mm and 0 . 5 ◦ ,and for each noise tier calibration is performed with eachof the classical methods considered.
We then comparethe result to the ground truth extrinsic and obtain positionerrors.
This procedure is repeated for 100 camera extrinsic, with the average errors obtained and their standarddeviations illustrated in the top graph of Fig.
We can see that as expected with perfect information theclassical methods return an exact solution.
However, wealso see that the quality of the calibration rapidly deteriorates with increased noise in the calibration object’sposition.
This illustrates our motivation for using a deeplearning based approach, since our methods are by construction independent from any external pose estimationalgorithm’s quality.
5.2 Sparse Correspondences
In order to better understand why our sparse correspondence model did not perform strongly we setup another controlled experiment.
Starting from ground truth 2D-3D correspondences, we addedﬁxed amounts of noise to the 2D keypoints and observed the effect on the quality of the PnP +
RANSAC pose estimation.
Speciﬁcally, we iterated through 200 simulated datapoints, and for each{ 2D keypoints, 3D keypoints, extrinsic } tuple, we perform PnP with the noise-injected 2D keypointsand compared the result to the ground truth.
We repeated this for different magnitudes of noise,ranging from 0 to 9 pixels, in increments of 1 pixel, and averaged over all the datapoints considered.
We plotted the resulting errors in the ”70% in frame” curve in the bottom graph of Fig. 4.
During our investigation we also considered a smaller amount of keypoints, but ones that alwaysremain in frame.
With those we observed that our network training performance increased, but ourﬁnal extrinsic estimation did not improve.
In order to understand this behaviour, we repeated thecontrolled noise experiment but only considering the keypoints that always appearing in frame inthe images, which resulted in 12 remaining keypoints.
This is plotted in the ”100% in frame” curvein the bottom graph of Fig.
P r e d i c t i on s
G r ound
T r u t h
[Figure]
From Fig. 4, we clearly see that there is a very high sensitivity of sparse correspondence poseestimation to errors in 2D keypoint pixel locations. For the keypoints in our training set, in orderto get an extrinsic calibration position error of less than 1cm, the average keypoint prediction errorfrom our networks would need to be less than 4 to 5 pixels.
This is with the additional challengeof the network having to keep consistent predictions even though sometimes the keypoints appearout of frame.
On the other hand, if we consider the easier to learn problem of only predictingkeypoints that are always appear in-frame, then the sensitivity skyrockets after single pixel averageerror in keypoint 2D locations.
Overall, we believe that this sensitivity to noise in the keypoints isthe primary reason why the sparse correspondence model did not give a strong performance.
5.3 Dense Correspondences
Although conceptually using a dense geometric correspondence approach to the pose estimationshould alleviate some of the drawbacks of using sparse correspondences, we found that in practiceour model that uses ICP to reﬁne estimates made by the DC method decreases their accuracy.
The reason for this becomes apparent when considering the illustrations shown in Fig.
5: Althoughthe depth values obtained by the neural networks look appealing when projected into the imageplane, they fall short when considering the full 3D structure.
We conjecture that the main reason forthis is that in order to recover a correct 3D structure from such close distances, depth values need tohave strong discontinuous jumps that typical neural networks have a lot of trouble modelling.
Thisis supported by the observations in Fig.
5: We can see that the overall depth increase seems to matchthe ground truth values, but there is an averaging effect that tends to make the depth values increasesmoothly, which doesn’t allows us to recover the correct shape.
As such, we can conclude that usinga simple depth regression technique in this setting is inappropriate for this task’s requirements, withfurther research needed to overcome the discontinuity problem.
Conclusion
In this work, we presented and evaluated three deep learning based methods for online eye-in-handcalibration from a single image.
We trained all our models entirely on synthetic images, and evaluated them both against each other and against long established calibration methods in simulationand the real world.
Surprisingly, our experiments indicated that a direct regression method from images to camera extrinsics outperformed other alternatives.
In order to better understand this result,we then conducted a series of introspection experiments, which indicated that a strong shortcoming from geometry-based methods seem to be their sensitivity to noise in their input data.
Finally,while depth map regression for dense correspondences conceptually could have provided the answer to such shortcomings, our experiments indicated that due to pronounced discontinuities in thedepth proﬁles from such close-up viewpoints, further research is needed into learning-based depthprediction for eye-in-hand robot manipulation.