Learning Eye-in-Hand Camera Calibration from a Single Image
Abstract: Eye-in-hand camera calibration is a fundamental and long-studied problem in robotics.
We present a study on using learning-based methods for solving the problem online from a single RGB image, whilst training our models with entirely synthetic data.
We study three main approaches, one using a direct regression model predicting the extrinsic matrix directly from images, one sparse correspondence model regressing 2D keypoints and using PnP, and one dense correspondence model that uses regressed depth and segmentation maps to enable ICP pose estimation.
In our experiments, we benchmark these methods against each other and against well established classical methods to ﬁnd the surprising result that direct regression outperforms other approaches, and perform noise-sensitivity analysis to gain further insights into those results.
Introduction
[Figure]
Eye-in-hand setups, where images from a camera rigidly mounted to the end-effector are used to control a robot, are a popular choice for precise robot manipulation.
Compared to control using a ﬁxed external camera, the eye-in-hand setup enables precise control because images can be captured very close to an object, as well as generalisation across the workspace when actions are deﬁned relative to the camera or end-effector [1, 2, 3].
To apply to precise manipulation tasks, these methods require accurate eye-in-hand extrinsic calibration, to ensure that actions or poses inferred in the camera’s frame can be transformed to the robot’s frame.
Whilst calibration-free approaches exist, such as end-to-end control [4, 5], these are only suitable for model-free policy learning, and cannot be used for methods that involve explicit planning or reasoning about 3D space.
Classically, solving the eye-in-hand calibration problem is performed off-line [6, 7, 8, 9] and involves the use of a speciﬁc external calibration object, such as a checkerboard or an AprilTag.
Poses of the calibration object can be estimated by the camera, and by recording them from several robot conﬁgurations, the rigid-body transformation between the camera and end-effector can be estimated analytically.
Although with perfect information these methods can yield exact calibrations, they are in practice limited by the quality of the data they use, including how precisely the pose of the calibration object can be estimated or the accuracy of the recorded end-effector poses.
Furthermore, these methods are unsuited for use by non-experts since they require dedicated external calibration objects and a new calibration dataset to be carefully collected for each re-calibration.
Finally, they are impractical as re-calibration needs to be performed every time the camera position changes due to (1) re-mounting, (2) wear and tear, and (3) collisions between the camera and the environment.
Deep learning methods can alleviate these drawbacks.
Given the initial overhead required to generate training data and train a model, a deep learning-based calibration method can be readily used to re-calibrate a camera when necessary.
Using parts of the robot as the calibration object, they have the potential to perform calibration online from a single image without the use of an external calibration object.
This potential has been demonstrated for eye-to-hand camera calibration, where an external camera observes the robot from a distance [10, 11].
To the best of our knowledge, there are currently no studies on using deep learning methods for eye-in-hand calibration, which is the focus of this paper.
We propose that this calibration can be estimated directly from the camera’s image itself, without requiring any external apparatus, as long as the image can observe part of the end-effector, such as the gripper’s ﬁngers (see Fig. 1).
Based on this, we identify and benchmark three natural approaches for leveraging the power of deep learning in this way: two based on classical, analytical/geometric calibration where deep learning is used in place of components which are typically manually engineered, and one which uses deep learning to directly regress the camera calibration matrix.
We show in our real-world experiments that the two approaches that rely on geometric analysis perform poorly, and we analyse empirically the structural limitations and challenges that cause this.
We also show that, perhaps surprisingly, direct regression outperforms all alternatives, including even classical methods based on ﬁducial markers with automatic data collection.
Finally, given that direct regression can estimate the calibration matrix from a single image online, we propose and evaluate a method for fusing multiple estimates of the camera pose, to increase the accuracy of the aggregated estimate.
As such, our work has three key contributions.
(1) We investigate the suitability of three natural alternatives of using deep-learning for enabling eye-in-hand calibration from a single image, (2)
We show the potential of end-to-end deep learning for online eye-in-hand camera calibration in everyday environments, and (3) We provide an analysis for the two learning-based methods that rely on geometric approaches, through which we discover why, although useful for calibration with an external camera, these methods may not be suitable for eye-in-hand calibration.
Supplementary material and a video can be found at https://sites.google.com/view/eye in hand .
Related Work
Eye-in-hand calibration is a long-studied problem, with signiﬁcant advances introduced in the 1990’s with what are now well-established solutions.
The problem consists of inferring the camera to end-effector pose for a camera mounted on the wrist of a robotic manipulator.
To do so, several end-effector to robot base poses are recorded, along with corresponding estimates of calibration object to camera poses.
The problem is then reduced to solving the AX =
XB equation through a formalism introduced by Shiu and Ahmad in [12].
Several works then followed that mainly differ in their strategy for solving the AX =
XB equation.
For instance, Tsai and Lenz [7] improve on the efﬁciency of Shiu and Ahmad’s solution by proposing a closed-form solution.
Park and Martin [8] and Dornaika and Harod [9] also consider such a closed-form solution while relying on Lie theory and unit quaternions respectively.
While the above approaches solve for rotation and translation separately, Dornaika and Harod [9] also propose a non-linear technique for solving for both simultaneously.
Daniilidis proposes a solution to the problem using dual quaternions [6].
Finally, more recent works study various extensions for particular settings such as using structure for motion for calibration [13], using model-based pose estimation and tracking for online calibration [14], simultaneously considering the data time synchronisation problem [15], calibrating depth sensors with non-overlapping views [16], automatically doing both calibration pattern localisation and eye-in-hand calibration [17], and optimising the robot kinematics parameters and the camera to end-effector pose simultaneously [18].
Recently, deep learning methods have had great successes on closely related tasks such as pose estimation [19, 20] and end-to-end robotics manipulation from wrist-mounted cameras [4, 3].
Moreover, sim-to-real transfer has shown promise in alleviating the large data requirements that hinder the scalability of such methods [4, 21, 22, 23].
In this work, we investigate the naturally emerging question of how one might use such methods for eye-in-hand calibration, which is fundamental to a large number of robotics pipelines.
Closest to our work, Lee et al.
[11] and Labbe et al.
[10] investigate two different deep learning approaches for eye-to-hand calibration of a camera looking straight at a robot.
[11] is based on keypoint regression followed by Perspective-n-Point (PnP) [24] while [10] adopts a render and compare approach.
In our work, we consider the eye-in-hand setup that is more common when precise manipulation needs to be achieved by looking closely at the end-effector
Dense Correspondence
[Figure] x z y aa 6 ...
[Figure]
ICP initialisation estimate
ICP Pose estimation Intrinsics
Camera to End effector transformation
ImageEncoder
ImageDecoder
SpatialSoftArgmax 2D Keypoint Locations 3D Keypoint Locations 2D - 3D Correspondances
PnP & Ransac
Camera to End effector transformation
Intrinsics
U n e t s k
i p c onn ec ti on s Fully ConnectedLayers
End Effector Model
Segmented Depth Image
ImageEncoder
ImageDecoder
U n e t s k
i p c onn ec ti on s
ImageEncoder
ImageDecoder
U n e t s k
i p c onn ec ti on s
[Figure]
of the robot [4, 25].
This problem also differs in its properties from the eye-to-hand setup, since in a typical image there is now a much more constrained view that only allows for the tip of the end-effector to be visible and to be used as an anchor for pose estimation.
Methods
In this section, we present three methods which are conceptually promising for eye-in-hand calibration from a single RGB image, and which also raised interesting and novel scientiﬁc questions that we answer in sections 4 and 5.
The ﬁrst method is an end-to-end direct regression method that estimates the pose of the camera from an RGB image, while the other two methods use deep learning to provide key missing components for well established geometric pose estimation approaches.
The latter has been shown to work well in other problem settings [11, 26, 27, 28], but eye-in-hand calibration has the particularity that not only the camera is too close to the end-effector for typical vision sensors to capture a depth image of it, but also the visible geometry varies widely within our inference space.
To test whether it is still feasible to use such approaches, our second method uses deep learning to regress 2D locations of predeﬁned keypoints on the end-effector, that are then passed into the PnP [24] algorithm to solve for the pose of the camera.
The main challenge with this method is that the large variations in the visible geometry leads to keypoint occlusions and to some keypoints being often outside of the image frame.
This is illustrated in Fig.
Our ﬁnal method tests whether we can alleviate this pitfall by using deep learning to regress a segmented depth image of the end-effector and an initial guess of the camera pose, and then reﬁne this initial estimate with the
Iterative Closest Point (ICP) algorithm [29, 30, 31].
All three methods are illustrated in Fig.
3.1 Problem Setting
Our problem setting consists of the typical eye-in-hand calibration problem, where the aim is to estimate the camera’s extrinsic matrix, which is the camera to end-effector pose, T EC =
[ R EC | t TC ] ∈
SE (3) , where R EC ∈ SO (3) and t EC ∈ R 3 is the orientation and position of the camera in the end-effector frame respectively.
We also deﬁne the end-effector to robot base pose as T BE , the calibration object to camera pose as T CO , and an image captured by the wrist-mounted camera as I .
Throughout this paper, we use ˜ · to denote an estimated quantity.
As opposed to classical approaches, we constrain ourselves to eye-in-hand calibration from a single
RGB image without any external apparatus, enabling our methods to be readily deployed in the wild for online eye-in-hand calibration.
We further constrain ourselves to using only synthetic training data.
This has the beneﬁt of alleviating the costly data requirements of deep learning approaches while also providing us with ground truth labels that enable our proposed methods to learn effec-
[Figure]
tively.
Finally, we assume that we have an accurate estimate of the camera intrinsic matrix, which is typically provided by the manufacturer, and access to the CAD model of our robot’s end-effector.
3.2 Direct Regression Model
Direct regression (DR) is the end-to-end deep learning method we consider.
As illustrated in the right diagram of Fig. 2, a neural network (NN) is tasked with regressing the camera pose parameters from a single RGB image and is trained in simulation using ground truth labels in a supervised setting.
The only architectural constraint we introduce comes from the parameterisation of the orientation, where we use the 6D rotation encoding introduced in [32].
Overall, our network outputs are 9 dimensional, with 3 dimensions representing position and 6 the orientation.
It consists of a convolutional encoder followed by linear layers that regress the camera pose.
We trained the network on the ground truth labels using the Mean Squared Error (MSE) loss, and we fully breakdown the architecture and training details in our supplementary material.
3.3 Sparse Correspondence Model
Our sparse correspondence model (SC) is illustrated in the middle of Fig.
Starting from an RGB image, it uses a U-Net [33] type architecture to regress a set of 2D keypoints in image space.
The UNet architecture downsamples the input image into some abstract feature space and then upsamples it back to the original resolution, while providing skip connections through concatenation at each resolution level [33].
On the output layer we apply a spatial-soft-argmax [34] activation in order to regress 2D image keypoints.
We use the sum of an L1 and MSE losses to train our network on the ground truth labels, where each dimension of the output predicts the 2D location of a predeﬁned 3D keypoint.
A detailed breakdown our architecture and training parameters can be found in our supplementary material.
At test time, we use the network outputs, the corresponding 3D keypoint locations deﬁned in the object’s model, and the camera’s intrinsic matrix to estimate the camera pose with PnP and
RANSAC [24], which is a popular outlier rejection algorithm.
Finally, in order to alleviate the issue of some keypoints being out of frame depending on the viewpoint, we (1) consider 38 keypoints that are present in at least 70% of the training images, (2) for any keypoint that remains out of frame, we train the model to predict its projection in the image.
This way, at test time, we can infer which keypoints are out of frame by looking at the predictions at the rim, and ignore these predictions for PnP pose estimation.
3.4 Dense Correspondence Model
Our Dense Correspondence Model (DC) is illustrated on the right of Fig.
Starting from an RGB
Image, it uses three NNs that are trained independently to estimate the pose of the camera in two stages.
In the ﬁrst stage, the three NNs are used to predict (1) a depth map, (2) a segmentation mask of the gripper, and (3) an ICP initialisation.
In the second stage, (1) the segmentation mask is used to segment the depth map to only return depth values on gripper pixels, (2) the camera intrinsic matrix is used to project the segmented depth map to a point cloud, and (3) ICP is used to approximate dense correspondences between the estimated point cloud and the model of the end-effector in order to reﬁne the camera pose estimate from the ﬁrst stage.
The network used for initialisation is the same as the one used for direct regression, and the same U-Net architecture as for the sparse correspondence model is used to predict the depth and segmentation maps.
See our supplementary material for further architecture design choices and training details.
3.5 Fusing Multiple Estimates
Compared to classical eye-in-hand calibration methods, which require calibration to be done ofﬂine and with special apparatus, the proposed deep learning alternatives can estimate the pose of the camera online in unstructured environments.
This creates the possibility of fusing multiple estimates to increase the overall calibration accuracy.
In order to explore this, we implemented a simple aggregation algorithm that rejects 20% of the least likely samples from a set of candidate estimates, under a Gaussian data model.
It then averages the remaining samples together to yield the ﬁnal estimate.
Pseudocode for this procedure is available in our supplementary material.
3.6 Dataset Generation and Sim-to-Real Transfer
We train our networks entirely with simulated data using the Coppelia [35] simulator.
In order to generate the dataset, we randomise the pose of the camera relative to the end-effector at each simulated timestep and record (1) the current RGB, depth and end-effector segmentation images, (2) the ground truth extrinsic matrix and (3) the 2D keypoint locations in image space.
In order to overcome the ”reality gap” we apply visual Domain Randomisation [21, 22, 23].
Specifically, we (1) randomise the colours/textures of the simulated gripper and the light sources on the simulator, (2) replace the background of our images with random images of textures and indoor scenes [36, 37], and (3) apply a post-processing colour jitter operation [38], further randomly perturbing the brightness, contrast, saturation and hue of the whole image.
In total, we generatedlabelled images in approximately 30 minutes, examples of which can be seen in Fig.
Experiments
In our experiments, we evaluated how the different proposed deep learning approaches compare to each other and to various standard off-the-shelf calibration methods both in simulation and in the real world.
From them emerges the surprising result that simple direct regression using end-to-end deep learning outperforms both the classical approaches tested and the correspondence based deep learning methods.
As such, we performed a series of analysis experiments that we describe in sec.that are aimed at gaining insights into this surprising observation.
In our comparisons we include all our deep learning based methods and the following established methods readily available on OpenvCV [39]: Dainiilidis et al.
[6], Tsai and Lenz [7], Doraika and
Harod [9], and Park and Martin [8].For all experiments, we use the Realsense D435 [40] (or its simulated counterpart) at a 480 × 848 resolution, and for the deep learning methods we downsample these images to a resolution of 144 × 256 .
Comparing classical methods to our deep learning ones fairly can be somewhat challenging since classical methods require a training set of several end-effector to robot and calibration object to camera poses to perform a single calibration, while our methods only require a single image.
In sections 4.1 and 4.2 we describe our procedures and what we did to mediate this issue, and in our supplementary material we provide pseudocode for each of the procedures.
4.1 Simulation Experiment
Our simulated environment consists of a wrist-mounted camera attached to a Sawyer robot’s gripper that is free to move around an AprilTag (see Fig. 3).
To benchmark all methods, we (1) Collect a dataset { I i , T iBE , ˜ T iCO } 15 i =1 of images and corresponding AprilTag and end-effector poses, with the tag poses ˜ T CO estimated using the AprilTags3 library [41, 42, 43], (2) Use all of the 15 datapoints in this dataset to estimate the extrinsic matrix using each classical method, and evaluate these estimates, (3) Use each of the 15 datapoints in this dataset independently to estimate the extrinsic matrix using each of the proposed learned methods, and evaluate each of the single image estimates, and (4)
For each of the learned methods, we fuse together all 15 independent predictions using our fusion procedure described in 3.5 and evaluate the fused estimate.
[Table]
We repeat this procedure for 100 different ground truth extrinsic matrices, average the results and display them in table 1.
We evaluate translation and rotation independently:
Given an estimate of the extrinsic ˜ T EC =
[ ˜ R EC | ˜ t EC ] [Table] and the ground truth T ∗ EC =
[ R ∗ EC | t ∗ EC ] , we deﬁne the position error as e t = ||
˜ t EC − t ∗ EC || 2 , where || · || 2 is the L2 norm, and the rotational error as e R =
θ , the angle from the axis-angle representation of the rotation matrix R ∆ =
( w, θ ) that satisﬁes the relationship
R ∗ EC = R ∆ ˜ R EC , where w is the axis of rotation, and θ is the angle of rotation.
4.2 Real World Experiment
The real world evaluation environment is analogous to the simulated environment and consists of a
Sawyer robot with a wrist-mounted camera moving around an AprilTag (see Fig. 3).
In order to evaluate our methods in the real world we (1) Collect a training data bank D train
= { I i , T iBE , ˜ T iCO }
i = and an evaluation dataset D eval =
{ T iBE , ˜ T iCO } 60 i =1 automatically by scripting a trajectory around the AprilTag, where AprilTag poses are estimated using an off-the-shelf library [42, 43], (2) We sample 40 training datasets of 15 datapoints each from our training data bank, (3) We use alldatapoints in each training set to get an estimate of the camera extrinsic matrix using each classical method, and evaluate that estimate, (4) We use each one of the 15 datapoints in each dataset to get an estimate of the extrinsic matrix using our learned methods, and evaluate each single estimate, and (5)
For each of the learned methods, we fuse together all 15 independent predictions using the procedure described in 3.5 and evaluate the fused estimate.
We repeat this procedure for two different ground truth extrinsic, average the results and display them in table 1.
Since ground truth extrinsic values are not available in the real world, we take inspiration from [6, 7] and use an indirect error metric:
For each corresponding end-effector to robot pose and calibration object to camera pose from the evaluation dataset, { T iBE , ˜ T iCO } ∈ D eval , we estimate the pose of the calibration object in the robot frame, ˜ T i BO = T i BE ˜ T EC ˜ T i CO =
[ ˜ R i WO | ˜ T i BO ] , where ˜ T EC is the estimate of the camera to end-effector pose that we are evaluating.
We then deﬁne the error metric (cid:15) std = (cid:16) 1 / 60 (cid:80) 60 i =1 (cid:12)(cid:12) | ˜ t iBO − µ (cid:12)(cid:12) | 22 (cid:17) 1 / 2 as the standard deviation of the estimated calibration object position, where µ = 1 / 60 (cid:80)
i =1 ˜ t iBO is the estimated mean object position.
We emphasise that although the AprilTag is visible in all images in both the simulated and real environments, it is never used to help the calibration of our deep learning methods.
We use these images simply to ensure that we have the exact same calibration and evaluation datasets for both classical and our learned methods.
4.3 Results
All benchmarking results are shown in table 1.
Our main observation is that the direct regression method outperforms all others, which is a surprising result.
First, it was not expected to outperform the classical methods since they use analytical solutions to estimate the extrinsic, which in the absence of noise in the system should give a perfect calibration.
We believe this stems from using automatic data gathering for the classical methods, which does not allow for carefully curating the calibration dataset and the distribution of poses within it, a manual trial-and-error process that is generally required in order to bolster calibration accuracy.
This also raises the question about the sensitivity of these methods to said noise, which we investigate in section 5.
Second, it was also not expected to outperform the correspondence based learned methods, since those introduce strong geometric constraints that intuitively should help guide the model to good solutions.
We thoroughly investigate the reasons behind this observation in sections 5.2 and 5.3.
Analysis
In this section, we aim to build a deeper understanding of the surprising results that we observed in our experiments, which showed that a direct regression of calibration parameters outperforms methods which incorporate well-understood, geometric modelling.
We split this section into three parts, each analysing the performance of one particular method.
5.1 Classical Methods
[Figure]
In order to better understand the results from our experiments in section 4, we ﬁrst aim to assess how sensitive classical methods are to sources of noise in the system.
Assuming the main source of error stems from the calibration object pose estimation, we ﬁx every other source of noise in simulation to their ground truth values.
We then vary the noise injected to the calibration object’s poses by controlled amounts, and observe how this affects the quality of the calibration result from classical methods.
Precisely, the noise scale added is varied from 0 mm to 10 mm and 0 ◦ to 10 ◦ , in increments of 0 .
5 mm and 0 .
5 ◦ , and for each noise tier calibration is performed with each of the classical methods considered.
We then compare the result to the ground truth extrinsic and obtain position errors.
This procedure is repeated for 100 camera extrinsics, with the average errors obtained and their standard deviations illustrated in the top graph of Fig.
We can see that as expected with perfect information the classical methods return an exact solution.
However, we also see that the quality of the calibration rapidly deteriorates with increased noise in the calibration object’s poses.
This illustrates our motivation for using a deep learning based approach, since our methods are independent of any test-time data gathering or calibration object pose estimation quality.
5.2 Sparse Correspondences
In order to better understand why our sparse correspondence model did not perform strongly we set up another controlled experiment.
Starting from ground truth 2D-3D correspondences, we added ﬁxed amounts of noise to the 2D keypoints locations and observed the effect of this on the quality of
PnP + RANSAC pose estimation.
Speciﬁcally, we iterated through 200 random extrinsic values, and for each { 2D keypoints, 3D keypoints, extrinsic } tuple, we performed PnP with the noise-injected 2D keypoints and compared the result to the ground truth.
We repeated this for different magnitudes of noise, from 0 to 9 pixels in increments of 1 pixel, and averaged over all the datapoints considered.
We plotted the resulting errors in the ”70% in frame” curve in the bottom graph of Fig. 4.
During our investigation we also considered a smaller amount of keypoints, but ones that always remain in frame.
With those we observed that even though this is an easier task for the network prediction, our ﬁnal extrinsic estimation did not improve.
In order to understand this behaviour, we repeated the controlled noise experiment but only considering the keypoints that always appearing
P r e d i c t i on s
G r ound
T r u t h
[Figure]
in frame in the images, which resulted in 12 remaining keypoints.
This is plotted in the ”100% in frame” curve in the bottom graph of Fig.
From Fig.
4, we clearly see that there is a very high sensitivity of sparse correspondence pose estimation to errors in 2D keypoint pixel locations.
For the keypoints in our training set, in order to get an extrinsic calibration position error of less than 1cm, the average keypoint prediction error from our networks would need to be less than 4 to 5 pixels.
This is with the additional challenge of the network having to keep consistent predictions even though sometimes the keypoints appear out of frame.
On the other hand, if we consider the easier to learn problem of only predicting keypoints that always appear in-frame, then the sensitivity skyrockets after single pixel average error in keypoint 2D locations.
Overall, we believe that this sensitivity to noise in the keypoints is the primary reason for why the sparse correspondence model did not give a strong performance.
5.3 Dense Correspondences
Although conceptually using a dense geometric correspondence approach to pose estimation should alleviate some of the drawbacks of using sparse correspondences, we found that in practice our model that uses ICP to reﬁne estimates made by the DC method decreases their accuracy.
The reason for this becomes apparent when considering the illustrations shown in Fig.
Although the depth values obtained by the neural networks look appealing when projected into the image plane, they fall short when considering the full 3D structure.
We conjecture that the main reason for this is that in order to recover a correct 3D structure from such close distances, depth values need to have strong discontinuous jumps that typical neural networks have difﬁculty modelling.
This is supported by the observations in Fig.
5: We can see that the overall depth increase seems to match the ground truth values, but there is an averaging effect that tends to make the depth values increase smoothly, which does not allow us to recover the correct shape.
As such, we can conclude that using a simple depth regression technique in this setting is inappropriate for this task’s requirements, with further research needed to overcome the discontinuity problem.
Conclusion
In this work, we presented and evaluated three deep learning based methods for online eye-in-hand calibration from a single image.
We trained all our models entirely on synthetic images, and evaluated them both against each other and against long established calibration methods in simulation and the real world.
Surprisingly, our experiments indicated that a direct regression method from images to camera extrinsics outperformed other alternatives.
In order to better understand this result, we then conducted a series of introspection experiments, which indicated that a strong shortcoming from geometry-based methods seem to be their sensitivity to noise in their input data.
Finally, while depth map regression for dense correspondences conceptually could have provided the answer to such shortcomings, our experiments indicated that due to pronounced discontinuities in the depth proﬁles from such close-up viewpoints, further research is needed, speciﬁcally in learning depth prediction for eye-in-hand robot manipulation.