D EEP O RIENTATION U NCERTAINTY L EARNING BASED ON A B INGHAM L OSS
Anonymous authors
Paper under double-blind review
A BSTRACT
Reasoning about uncertain orientations is one of the core problems in many perception tasks such as object pose estimation or motion estimation.
In these scenarios, poor illumination conditions, sensor limitations, or appearance invariance may result in highly uncertain estimates.
In this work, we propose a novel learning based representation for orientation uncertainty.
Characterizing uncertainty over unit quaternions with the Bingham distribution allows us to formulate a loss that naturally captures the antipodal symmetry of the representation.
We discuss the interpretability of the learned distribution parameters and demonstrate the feasibility of our approach on several challenging real-world pose estimation tasks involving uncertain orientations.
I NTRODUCTION
Reasoning about uncertain poses and orientations, speciﬁcally 3-dimensional (3d) positions and 3-axes orientations, is one of the main inference tasks in computer vision (Sattler et al., 2019), robotics (Glover et al., 2011), aerospace (Crassidis & Markley, 2003), and other ﬁelds.
Proper representation and estimation of uncertainty is important, e.g. when dealing with structural ambiguities in object pose estimation or coping with sensor corruption while estimating a camera’s ego-motion.
In vision and robotics tasks, high levels of pose uncertainty may occur due to potentially adversarial conditions that arise in real-world scenarios.
A principled approach to uncertainty quantiﬁcation allows for better execution of planning and situation-awareness tasks such as grasping, tracking, and motion estimation.
When representing uncertainties over poses, the position can be modeled using a Gaussian distribution.
This approach is well-motivated by the Central Limit Theorem and widely used in probabilistic deep learning models.
However, this paradigm cannot be as easily applied to modeling periodic quantities, such as the orientation of an object.
Therefore, Gaussian models become unsuitable particularly in learning regimes involving high uncertainties where one cannot assume local linearity of the underlying space.
In this work, we set out to develop a principled probabilistic deep learning approach capable of coping with uncertain orientations.
Currently, most deep learning approaches that predict poses or rigid-body motions suffer from at least one of three drawbacks: 1) they do not model the uncertainty at all and merely focus on the accuracy of the predicted pose, 2) they make simplifying assumptions not taking into account that the orientation is deﬁned on a periodic manifold, making the approach only suitable in low-noise regimes, or 3) even when trying to account for periodicity, no dependency is assumed between the orientation axes and usually an Euler angle-based representation is required.
To this point, there are no probabilistic deep learning models for uncertainty of orientations that take the geometry of the underlying domain into account.
In this work, we close this research gap by proposing a probabilistic deep learning model inspired by Directional Statistics (Mardia & Jupp, 1999).
We present a loss based on the Bingham distribution (Bingham, 1974), an antipodally symmetric distribution on the sphere.
With this loss, we represent uncertain orientations by representing uncertainty over unit quaternions.
Our contributions involve Bingham parameter learning using backpropagation through a Gram-Schmidt method to ensure orthonormalization, efﬁcient approximate evaluation of the normalization constant of the
Bingham distribution from a lookup table, and backpropagating through an interpolation scheme during learning.
We also discuss interpretability of the Bingham distribution parameters and establish the feasibility of the approach through extensive evaluations.
In summary, this work makes the following contributions: 1) We propose the Bingham loss, a novel loss function for deep learning-based predictions of orientations and their uncertainty.
We provide a methodology for making use of the newly proposed loss and its normalization constant computationally tractable in a deep learning pipeline.
We demonstrate multi-modal orientation prediction using a Bingham distribution variant of Mixture Density Networks.
We demonstrate how our approach outperforms the state-of-the-art on challenging pose and orientation estimation tasks.
B ACKGROUND : B INGHAM D ISTRIBUTION FOR U NCERTAIN O RIENTATIONS
Unit quaternions are a widely used representation for object orientation in 3d space.
They are more compact than rotation matrices and, unlike Euler angles, do not suffer from degeneracies such as Gimbal lock.
Additionally, quaternions provide a convenient mathematical notation where the quaternion product, q 1 (cid:12) q 2 , of two unit quaternions q 1 , q 2 ∈ H 1 results in a concatenation of the rotations represented by each of the quaternions individually.
A full introduction to this representation is given in Kuipers (1999) and notational aspects are discussed by Sommer et al.
In this work, a quaternion q 1 i + q 2 j + q 3 k + q 4 will be interpreted as a vector q ∈ R 4 .
It is important to note that the deﬁnition of unit quaternions is equivalent to the vector q being of unit length
|| q ||
= 1 .
Furthermore, the quaternions q and − q represent the same orientation.
Therefore, representing uncertain orientations using quaternions requires a probability distribution on the 4d hypersphere that exhibits antipodal symmetry, i.e. for the density function f ( · ) of this distribution f ( q ) = − f ( q ) has to hold.
A probability distribution exhibiting these properties was proposed by Bingham (1974).
It arises by conditioning a zero mean Gaussian to unit length.
The Bingham distribution is given in terms of its p.d.f.
as p ( x ; M , Z ) =
N ( MZM (cid:62) ) − 1 exp( x (cid:62) MZM (cid:62) x ) , where x ∈ R 4 with || x
||
= 1 , N ( MZM (cid:62) ) is a normalization constant, M ∈ R 4 × 4 orthogonal, and Z = diag
( z 1 , z 2 , z 3 , 0) ∈ R 4 ×diagonal, with diagonal entries
z
i given in an ascending order and the last entry being zero.
We use the notation Bingham( M , Z ) .
The restriction on the range of the diagonal entries in Z has numerical and representational convenience reasons as it can be shown that Bingham( M , Z ) =
Bingham( M , Z + c I ) for all c ∈ R with I ∈ R 4 × 4 denoting the identity matrix.
Similarly, changing the order of diagonal entries in Z has no effect on the distribution as long as the columns in M are permuted accordingly.
In the deﬁnition above, the parameters M , Z bear some similarity to the mean and variance of a Gaussian.
The density obtains its maxima at ± M : , 4 (the fourth column of M ) which can be thought of as a mean orientation respecting the manifold structure.
The diagonal entries of Z can be interpreted as dispersion parameters and the ﬁrst three columns of M as the directions of the dispersion (the Gaussian analog is the orientation of the covariance ellipsoid).
Bingham distributions allow for representation of uniform priors over individual axes or even the entire space making them superior over the use of Gaussians in any of the usual orientation representations.
One of the main challenges of using the Bingham distribution is the computation of its normalization constant
[Equation]
[Figure] which is a Hypergeometric function of matrix argument (Herz, 1955).
Evaluating these functions imposes a high computational burden and is still an area of active research (Koev & Edelman, 2006;
Kume et al., 2013; Koyama et al., 2014; Kume & Sei, 2018).
Using the transformation theorem and the fact that M is orthogonal, the normalization constant can be simpliﬁed as N ( MZM (cid:62) ) =
N ( Z ) , making it merely a function of the three parameters
z i ( i = 1 , 2 , 3 ) and motivating the use of precomputed lookup tables in practice.
Furthermore, to make the uncertainty of a Bingham Distribution more interpretable in practice, we propose the use of Expected Absolute Angular Deviation (EAAD) which is deﬁned as
[Equation]
where p ( · ) is the Bingham( I , Z ) density, I is the identity matrix, e = [0 , 0 , 0 , 1] is the vector corresponding to the unit quaternion representing the identity and θ ( q , e ) = 2 · arccos( |(cid:104) q , e (cid:105)| ) denotes the angular distance between q and e .
The EAAD describes the expected angular deviation from the “mean” orientation.
It can be loosely thought of as the orientation counterpart to the standard deviation in Euclidean space.
For the same reason as in the normalization constant, the
EAAD computation does not involve the parameter M .
D EEP O RIENTATION U NCERTAINTY L EARNING
The Bingham distribution is the main component of the proposed probabilistic framework for representing deep learned uncertain orientations.
Drawing inspiration from Mixture Density Networks (Bishop, 1994), we propose using the Bingham distribution’s negative log-likelihood as a loss function
[Equation]
with M , Z as deﬁned above and y being the orientation label given in the training data.
We use a neural network to learn M and Z , end-to-end, directly from the input data (e.g. RGB images).
From this prediction the point estimate of y is obtained as ˆ y =
M : , 4 as the last column corresponds to the highest diagonal entry of Z and thus represents one of the modes of the distribution (the other being − ˆ y due to antipodal symmetry).
No costly evaluation of the normalization constant is required and no major computational challenges arise in the special case where the dispersion parameter Z is known and not predicted by [Figure] a neural network.
However, as our goal is the modeling of uncertainty, we propose methods for modeling M and Z as well as backpropagating through N ( Z ) .
3. M ODELING OF DISTRIBUTION PARAMETERS
In order to obtain predictions ˆ M and ˆ Z , we require a 19 dimensional output ( o ∈ R 19 ) of the predictor network (3 outputs for Z , 16 outputs for M ).
On its own, these outputs do not satisfy the above-mentioned constraints on the Bingham distribution parameters.
Thus, we deﬁne the differentiable transforms T M : R 16 → R 4 × 4 and T Z : R 3 → R 4 × 4 that transform these outputs such that the constraints are satisﬁed.
The transform T Z is obtained as T Z ( o 1 , o 2 , o 3 ) =
diag(ˆ z 1 , ˆ z 2 , ˆ z 3 , 0) with ˆ z i = − (cid:80) 4 − i k
=1 exp( o k ) , which ensures that ˆ z 1 < ˆ z 2 < ˆ z 3 .
For computing ˆ M , we ﬁrst subdivide o 4 , . . .
, o 19 into four vectors v i ∈ R 4 ( i = 1 , . . .
Then, we apply the Gram-Schmidt orthonormalization method to these vectors according to ˆ m i =
Normalize( v i − (cid:80) i − 1 k =1 (cid:104) ˆ
m k , v i (cid:105) · ˆ m k ) with i ∈ { 1 , 2 , 3 , 4 } and Normalize( x ) = x / || x || .
Finally, the prediction ˆ M is obtained as T M ( o 3 , . . .
, o 19 ) =
[ ˆ m 1 , . . .
, ˆ m 4 ] , and ˆ M is orthogonal by construction.
3. B ACKPROPAGATION THROUGH THE B INGHAM N ORMALIZATION C ONSTANT
As mentioned earlier, computation of the Bingham normalization constant is numerically burdensome.
This is also true for its derivatives which can be shown to be proportional to the normalization constant of Bingham distributions of higher dimension (Kume & Wood, 2007).
Thus, a forwardbackward pass for one single data point requires 4 evaluations of hypergeometric functions, which is very computationally expensive.
We avoid this by precomputing a lookup table for N ( Z ) at L different locations t i (with Z i = diag([ t (cid:62) i , 0] (cid:62) ).
This table is then used to build an interpolator f N ( z )
= (cid:80) Li − 1 w i φ ( || z − t i || ) with z ∈ R 3 and φ denoting a radial basis function.
The weights w i can also be precomputed during generation of the interpolator.
Thus, we can approximate N ( Z )
≈ f N ( z )
and ∇ z N ( Z )
≈ ∇ z f N ( z ) .
To the best of our knowledge, this is the ﬁrst time that a lookup table based interpolation mechanism has been included into the computation graph of a neural network. 3. M ULTI -M ODAL P REDICTION
A Bingham variant of Mixture Density Networks can be used to obtain multi-modal predictions.
However, MDNs are hard to train even in the Gaussian case.
Following the discussion in Makansi et al.
(2019), we separate the training in two stages.
In the ﬁrst stage, we only learn to predict M and assume the dispersion to be ﬁxed with Z = diag( − a, − a, − a, 0) . In practice a ∈ R + can usually be set to 1 as it is merely a factor in front of the cost term. In the second stage, we train to predict M and Z jointly.
Our evaluation will show that in high uncertainty regimes, this training method is also helpful for the unimodal case.
E XPERIMENTS
In this section we evaluate the proposed Bingham loss on its ability to learn calibrated uncertainty estimates for orientations.
This goes beyond comparing point estimates of orientations; we evaluate how well the estimated distribution of orientations can explain the data.
We will also show that the Bingham distribution representation is capable of capturing ambiguity and uncertainty in SO(3) better than state-of-the-art approaches.
We investigate characteristics and behavior by training neural networks on two head-pose datasets,
IDIAP (Odobez, 2003) and UPNA (Ariz et al., 2016), as well as the object pose dataset TLESS (Hodaˇn et al., 2017).
We show the capability of calibrated uncertainty estimation by applying artiﬁcial label-noise to IDIAP and UPNA and observe that the Bingham parametrization allows for accurate prediction of uncertainty.
While calibrated uncertainty estimation is important, we demonstrate advanced capabilities in the face of object orientation ambiguity on the T-LESS dataset by visualizing the predicted distributions for different orientation ambiguous objects, e.g. symmetric, and comparing to objects with clear orientation.
4. A RCHITECTURE AND E XPERIMENTAL S ETUP
We seek to estimate the Bingham distribution parameters directly from image data.
Our pipeline is shown in Figure 3 and begins with an image input into a convolutional encoder, in our case a standard ResNet-18 network followed by a fully connected layer, populating the entries of o 1 , o 2 , o 3 and v 1 , v 2 , v 3 , v 4 .
Subsequently, Z is computed by constrained diagonalization of o 1 , o 2 , o 3 , and GramSchmidt orthonormalization of v 1 , v 2 , v 3 , v 4 yields M , as described in Section 3.1.
To evaluate the
Bingham loss, the normalizer N ( Z ) needs to be queried from the RBF lookup table, Section 3.2.
Automatic differentiation of the lookup table enables us to back-propagate through the entire pipeline.
All models were implemented in PyTorch and optimized with the Adam optimizer.
We create the lookup table by numerical integration.
More precisely, we use Scipy’s tplquad method to compute a triple integral for each Z in the table.
We set the relative error tolerance to 1eand the absolute error tolerance to 1e-7.
The actual computed integral is
N (
[Equation]
=
[Equation]
with
[Equation]
 to account for a transformation of coordinates from unit quaternions to 4d spherical coordinates.
Because we use the Bingham log likelihood as our optimization objective, we compute the logarithm before the interpolation to avoid failure at locations where the interpolator wrongly outputs negative values.
4. B ASELINES
We compare our work with the approach proposed by Prokudin et al.
It also uses a loss based on directional statistics, more precisely, based on the Von Mises distribution which is a circular analog of the Normal distribution.
In order to apply this approach to our setting, orientations are modeled with Euler angles.
The loss then consists of the sum of log-likelihoods for each angle.
While this approach can properly account for periodicity of the underlying data, we expect it to fail in cases where the underlying uncertainty is not axis aligned due to not considering the dependencies between uncertain rotation axes.
Furthermore, we also evaluate several different representations of the parameter matrix M .
We consider the classical Gram-Schmidt (CGS), modiﬁed Gram-Schmidt (MGS), and the matrix rep- resentation of the quaternion (QM) used by Birdal et al.
Finally, we also include two nonprobabilistic orientation prediction baselines.
The ﬁrst one is based on a Mean Square Error (MSE) between the predicted and ground truth quaternion.
The second one is based on a cosine loss applied to each angle’s biternion as discussed by Prokudin et al.
4. E VALUATION M ETRICS
To evaluate error metrics over predicted orientations, it is unsuitable to compute the RMSE over angles, since these do not sufﬁciently consider the spherical nature of the underlying data.
Instead, we make use of the Mean Absolute Angular Deviation (MAAD) which has also been used by Prokudin et al.
It is based on the angular distance between two angles deﬁned above and provides an intuition following the axis-angle representation.
We also compute the EAAD to provide an intuition of the quality of the results.
Additionally, the difference between EAAD and MAAD serves as an indicator of the quality of the predicted uncertainty.
The acceptable difference in practice is application dependent.
For the cases of the Von Mises distribution parameters, EAAD computation is carried out in a similar way as for the Bingham deﬁned above.
For the purposes of evaluation, EAAD is computed over the average of the concentration parameters.
The quality of the respective model is measured in terms of log-likelihood to indicate the goodness of an individual ﬁt.
For MDNs, we additionally report a Mean Minimum Absolute Angular Deviation (MMAAD), which uses the component closest to ground-truth for absolute angular deviation computation.
The
MAAD and EAAD for MDNs are computed in a per-component fashion and then weighted using the predicted mixture weights.
4. C ALIBRATED U NCERTAINTY E STIMATION
We evaluate the distribution ﬁt on the head pose datasets UPNA and IDIAP, which consist of head images from a video of several people inside a room.
Each image is annotated with head orientation given by pan, tilt and roll angles.
We use these datasets as they provide accurate labels and allow for carrying out experiments involving artiﬁcial label noise.
[Table]
The results for the raw dataset are shown in Table 1. They demonstrate that the general performance for point estimates, indicated by MAAD, of the Bingham distribution remains on a similar level as the Von Mises distribution and the non-probabilistic approaches.
In this setting, most motions of the subjects’ heads are aligned with the gravity axis allowing both distributions to successfully capture the noise.
However, the Bingham still attains a higher log-likelihood and a smaller gap between MAAD and EAAD.
Similarly, the parametrization of the parameter matrix M has a relatively small impact on the estimation performance.
Although MGS has stronger robustness guarantees than CGS (the latter has a quadratic dependency on the condition number of the input matrix, see Giraud et al.
for a discussion of both), the condition of the input is not poor enough to impact performance.
While the quaternion matrix approach is easier to train, it also loses some of the expressiveness of the
Bingham distribution because the underlying mapping (from quaternions to the space of orthogonal matrices) is not surjective.
To estimate how well the predicted uncertainties are calibrated, we add artiﬁcial noise by drawing random perturbations from the Bingham distribution with varying z 1 , z 2 and z 3 parameters.
Both
UPNA and IDIAP contain only negligible amounts of noise, such that the amount of label noise should be reﬂected in the mean estimated noise to high accuracy.
An evaluation of uncertainty and label noise is shown in Table 2.
The Bingham uncertainty parameters closely match the label noise parameters.
For the case of no noise, the Bingham uncertainty parameters approximate the highest [Table] certainty levels represented in the lookup table.
Thus, the maximum and minimum values in the lookup table automatically become the bounds of what certainty levels can be represented by the proposed loss.
4. H ANDLING A MBIGUOUS D ATA
We use the T-LESS dataset for evaluating the proposed model using ambiguous data.
It contains images of 30 different textureless objects taken from different cameras. We use the Kinect RGB single-object images all of which are split into training, test [Table]
and validation sets. At a coarse scale most of the objects in the dataset exhibit rotational or other symmetries. At a ﬁner scale some of these ambiguities disappear due to smaller structures.
On the one hand, we expect those to be more challenging to learn.
On the other hand, capturing these structures allows for very precise orientation estimation. To be able to disregard these structures, we create a variant of T-LESS where we add blur to each image using a uniform 10px × 10px kernel.
[Table]
We carry out two sets of experiments.
In the ﬁrst set of experiments, we train orientation estimation models for 5 epochs using the Bingham loss (BD-5) and the Von Mises loss (VM-5) on the blurred and original set of images. This allows to investigate the uncertainty estimation properties before the network captures the ﬁner grained structures.
In the second set of experiments, we use the original set of images to evaluate multi-modal orientation prediction using the two-stage training approach for models with(BD-MDN-1), 2 (BD-MDN-2), and 4 (BD-MDN-4) mixture components. Each stage is carried out for 30 epochs. The comparison methods use Von Mises (VM), Mean Square Error (MSE), and
Cosine losses with an overall training duration of 60 epochs (or until convergence if that is earlier).
[Table] The results for the ﬁrst set of experiments are visualized in Table 3.
As expected, both approaches are on average far off in terms of the true orientation.
While Von Mises performs better on the MAAD, we observe that there is a larger difference between the MAAD and EAAD values for the Von Mises distribution than the Bingham distribution.
This indicates that the uncertainty estimates of the Von Mises distribution may be overconﬁdent.
On the other hand the Bingham distribution better captures the uncertainty over individual axes.
One interesting insight is that allowing for uniform distributions over individual non-aligned periodic axes can make it hard for the learning method to pick up on the proper pose and thus may require pre-training on the pure pose estimation task in such regimes.
In the second set of experiments, as visualized in Table 4, we use this training strategy for all
Bingham MDN models resulting in robust convergence behavior.
However, the unimodal Bingham (BD-MDN-1) converges slower than von Mises (VM) thus achieving a higher MAAD, which is adequately captured by the Bingham’s EAAD.
For multiple mixture components, we obtain a very low MAAD and can observe again the phenomena of the lookup table limitations in the EAAD. Thus, the MAAD during the ﬁrst training stage can not only be used for inspecting the network’s accuracy but also for determining the minimum Z parameter values stored in the lookup table. Another interesting phenomena can be observed in the EAAD and MAAD of the VM loss. As the representation required by Von Mises assumes that each axis is independent, EAAD is computed per rotation axis.
This results in an overapproximation of the uncertainty overall.
For the nonprobabilistic losses, the cosine loss achieves better performance which is probably due to better consideration of the underlying geometry.
In summary, while the proposed Bingham loss shares the general challenges of training Mixture Density Networks, it better captures the underlying noise structure by explicitly modeling dependencies between rotation axes.
Quantifying and representing uncertainty by and in neural networks has been a subject of extensive research initially focused on modeling probability distribution parameters (Nix & Weigend, 1994) and mixture distributions (Bishop, 1994) as neural network outputs.
More recent approaches focus on improving understanding of the underlying uncertainties (Kendall & Gal, 2017), providing scalable techniques for estimating predictive uncertainty (Lakshminarayanan et al., 2017), and stabilizing training to avoid mode collapse (Makansi et al., 2019).
The present work is orthogonal to these approaches in the sense that it focuses on proper modeling of the underlying geometric domain and coping with a computationally demanding normalization constant.
Handling of poses and orientations has been extensively studied in the context of Bayesian ﬁltering for applications such as spacecraft attitude estimation Crassidis & Markley (2003) and ego-motion estimation Bloesch et al.
(2015), where one can often assume the underlying uncertainties to be small.
This allows for leveraging local-linearity and using the Gaussian distribution.
Recently, methods based on directional statistics enabled modeling of high uncertainty levels for inferring orientations (Gilitschenski et al., 2016) and full poses (Glover et al., 2011; Glover & Kaelbling, 2014; Srivatsan et al., 2016) by using the Bingham distribution.
Drawing inspiration from these results, this work extends the applicability of these approaches to probabilistic deep learning models.
Particularly in computer vision, deep learning has been applied to spherical regression and pose estimation problems (Liao et al., 2019; Huang et al., 2018).
These applications involve inferring object (Brachmann et al., 2014; Hodaˇn et al., 2018;
Li et al., 2018b;a; Manhardt et al., 2019; Sundermeyer et al., 2018; Tekin et al., 2018; Wang et al., 2019b;a), body (Yang et al., 2019), and camera poses (Clark et al., 2017; Sattler et al., 2019; Wang et al., 2017; 2018).
In all of these scenarios there is a multitude of sources for potentially high uncertainties such as the use of low-resolution data (e.g. tracking pose of distant pedestrians), absence of textures (e.g. when operating on depth data), or motion blur (e.g. due to high speeds in ego-motion estimation).
However, most of the existing approaches merely focus on inferring the pose but do not account for the underlying uncertainty.
The representation proposed in our work closes this gap by allowing for neural networks to output well-calibrated orientation uncertainty estimates.
Only few approaches consider modeling the uncertainty of orientations for deep learning based pose estimation.
PoseRBPF by Deng et al.
(2019) discretizes the orientation space into over 190bins and learns a codebook to allow for tractable inference.
In contrast to that approach, we do not require an a priori discretization and can directly obtain interpretable estimates.
Similarly to us, Prokudin et al.
(2018) propose a loss based on directional statistics.
By making use of the Von Mises distribution, their work can properly account for periodicity of circular data.
However, as we have shown in our evaluations, this approach cannot properly take dependencies between different axes into account and, thus, struggles when the underlying uncertainty is not axis aligned.
C ONCLUSION
In this work, we introduced the Bingham loss, a loss function based on the Bingham distribution that enables neural networks to predict uncertainty over unit quaternions and thus predict uncertain orientations.
It enables the use of highly symmetric objects and ambiguous sensor data in the context of pose and orientation estimation with uncertainty predictions.
In addition, we demonstrate how to cope with intractable likelihoods in deep learning pipelines by using non-linear interpolation and lookup tables as part of the computation graph.
The presented approach is directly usable in existing probabilistic deep learning techniques. Particularly, we demonstrate its applicability for mixture density models.
The choice of parametrization representation remains one of the main design decisions in pose and orientation estimation pipelines.
Our work supports the case for using quaternions over other parametrizations for deep learning.
It also motivates further research on how to properly model dependencies between uncertain periodic and non-periodic quantities.