DNBP: D IFFERENTIABLE N ONPARAMETRIC B ELIEF P ROPAGATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a differentiable approach to learn the probabilistic factors used for inference by a nonparametric belief propagation algorithm.
Existing nonparametric belief propagation methods rely on domain-speciﬁc features encoded in the probabilistic factors of a graphical model.
In this work, we replace each crafted factor with a differentiable neural network enabling the factors to be learned using an efﬁcient optimization routine from labeled data.
By combining differentiable neural networks with an efﬁcient belief propagation algorithm, our method learns to maintain a set of marginal posterior samples using end-to-end training.
We evaluate our differentiable nonparametric belief propagation (DNBP) method on a set of articulated pose tracking tasks and compare performance with learned baselines.
Results from these experiments demonstrate the effectiveness of using learned factors for tracking and suggest the practical advantage over hand-crafted approaches.
The project webpage is available at: https://sites.google.com/view/diff-nbp.
[Figure]
[Table]
[Figure]
I NTRODUCTION
A signiﬁcant challenge for robotic applications is the ability to estimate the pose of articulated objects in high noise environments.
Nonparametric belief propagation (NBP) algorithms (Sudderth et al., 2003; Isard, 2003) have proven effective for inference in visual perception tasks such as human pose tracking (Sigal et al., 2004) and articulated object tracking in robotic perception (Desingh et al., 2019; Pavlasek et al., 2020).
Moreover, these algorithms are able to account for uncertainty in their estimates when environmental noise is high and show promising computational properties in practice (Desingh et al., 2019; Ortiz et al., 2021).
Their adaptability to new applications, however, is limited by the need to deﬁne hand-crafted functions that describe the distinct statistical relationships in a particular dataset.
Current methods that utilize NBP rely on extensive domain knowledge to parameterize these relationships.
Reducing the domain knowledge required by NBP methods would enable their use in a broader range of applications.
As a form of probabilistic graphical model inference, NBP algorithms leverage domain knowledge encoded in graph-based representations, such as the Markov random ﬁeld (MRF).
Their capacity to perform inference using arbitrary graphs sets them apart from other algorithms such as the recursive
Bayes ﬁlter (Thrun et al., 2005)
(e.g. particle ﬁlter (Godsill, 2019)) and has been shown to be important in computational perception because it allows for modeling of non-causal relationships (Sudderth et al., 2003).
Data-driven approaches are an alternative for computational perception (Xiang et al., 2018; Tremblay et al., 2018).
These methods generally avoid the need for extensive domain knowledge by learning from large amounts of labelled data.
Data-driven approaches, however, are prone to noisy estimates and have limited capacity to represent uncertainty inherent in their estimates.
In robotic applications, both of these limitations negatively impact the ability for a robot to operate effectively in unstructured environments.
In this paper, we present a differentiable nonparametric belief propagation (DNBP) method, a hybrid approach which leverages neural networks to parameterize the NBP algorithm.
Through differentiable inference, DNBP leverages the explainability and robustness of probabilistic inference techniques and capitalizes on the efﬁciency and generalizability of data-driven approaches.
Inspired by the differentiable particle ﬁlter (DPF) from Jonschkowski et al.
and the pull message passing for nonparametric belief propagation (PMPNBP) algorithm
(Desingh et al., 2019), we develop a differentiable nonparametric belief propagation algorithm.
DNBP performs end-to-end learning of each probabilistic factor required for graphical model inference.
The effectiveness of DNBP is demonstrated on two simulated articulated tracking tasks and on a real-world hand pose tracking tasks in challenging noisy environments.
An analysis of the learned probabilistic factors and resulting tracking performance is used to validate the approach.
Results show that our approach can leverage the graph structure to report uncertainty about its estimates while signiﬁcantly reducing the need for prior domain knowledge required by previous NBP methods.
DNBP performs competitively in comparison to traditional learning-based approaches on the tracking tasks.
Collectively, these results indicate that DNBP has the potential to be successfully applied to robotic perception tasks, where a notion of uncertainty in the inference is inevitable.
R ELATED W ORK
Belief Propagation:
In the context of graphical models, inference refers to the process in which information about observed variables is used to derive the posterior distribution(s) of unobserved variables.
Belief propagation (BP) is a message passing algorithm for inference on graphical models.
BP computes exact marginal distributions on trees (Pearl, 1988), and has demonstrated empirical success on loopy graphs (Murphy et al., 1999; Sun et al., 2003; Lee et al., 2008; Lan et al., 2006).
In order to apply inference techniques such as BP and LBP, the parameters of a graphical model (e.g. the probabilistic factors) must be fully speciﬁed.
Maximum likelihood estimation (MLE) has been shown to be an effective approach for learning the parameters of a graphical model from data (Murphy, 2012; Koller & Friedman, 2009; Ping & Ihler, 2017).
In contrast, this current study focuses on parameter learning for use with inference of continuous random variables.
Nonparametric Belief Propagation:
For continuous spaces, such as six degrees-of-freedom object pose, exact integrals called for in BP and LBP become intractable and approximate methods for inference have been considered.
Nonparametric belief propagation (NBP) methods (Isard, 2003;
Sudderth et al., 2003), have been proposed which represent the inferred marginal distributions using mixtures of Gaussians and deﬁne efﬁcient message passing approximations for inference.
Isard (2003) demonstrated the effectiveness of PAMPAS using a set of synthetic visual datasets each modeled with hand-crafted factors.
Sudderth et al.
(2003) applied their NBP method successfully to a visual parts-based face localization task as well as a human hand tracking task (Sudderth et al., 2004).
In both applications, NBP relied on factor models which were chosen based on task-level domain knowledge (e.g. valid conﬁgurations of human hands).
Sigal et al.
(2004) extended these
NBP methods to human pose estimation and tracking using factors which were each trained separate from the inference algorithm using independent training objectives.
Ihler & McAllester (2009) described a conceptual theory of particle belief propagation, where messages being sent to inform the marginal of a particular variable could be generated using a shared proposal distribution.
Following the work of Ihler and McAllester, Desingh et al.
(2019) presented an efﬁcient “pull” message passing algorithm (PMPNBP) which uses a weighted particle set to approximate messages between random variables.
PMPNBP was shown to be effective on robot pose estimation tasks using hand-crafted factors.
Using a similar approximation of belief propagation,
Pavlasek et al.
(2020) took a step toward neural network-based potential functions by introducing a pre-trained image segmentation network to the unary factors.
An important limitation of these works is they assume the probabilistic factors expressed in the graph are provided as input or rely on domain knowledge to separately model and train each function.
The potential for neural networks to learn the parameters used by alternative inference techniques has been demonstrated (Do & Arti`eres, 2010; Tompson et al., 2014).
The recent work of Xiong & Ruozzi (2020) demonstrated promising inference performance on discrete classiﬁcation and synthetic datasets using learned probabilistic factors with a Bethe free energy approximation.
In contrast, this paper focuses on inference in high dimensional continuous state spaces using learned probabilistic factors with the particle-based inference process of NBP.
Differentiable Bayes Filtering:
In the context of robot state estimation, many approaches have recently been proposed that incorporate neural networks with recursive inference algorithms in an endto-end fashion.
Haarnoja et al.
(2016) introduced a differentiable Kalman ﬁlter, and Jonschkowski &
Brock (2016) proposed a differentiable, histogram-based Bayes ﬁlter algorithm.
Jonschkowski et al.
(2018) and Karkus et al.
both proposed differentiable particle ﬁlter algorithms for modeling continuous state spaces.
Kloss et al.
(2021) evaluate recent differentiable ﬁltering techniques.
Yi et al.
(2021) propose an end-to-end learning method for inference over factor-graph models.
In contrast to these methods, which model a single object body using variants of the Bayes ﬁlter, this work sets out to study the potential for NBP to be used as an algorithmic prior for modeling multi-part articulated objects.
Recently, this line of research on differentiable state estimation algorithms has extended into the planning domain (Karkus et al., 2019; Wang et al., 2020; Anderson et al., 2019).
Exploration of embedding DNBP within a differentiable planning system is left as future work.
B ELIEF P ROPAGATION
Consider a Markov Random Field (MRF) deﬁned by the undirected graph G = {V , E} , where V de- notes a set of nodes and
E denotes a set of edges.
An example MRF model is shown in Fig. 2b.
Each node in V represents an observed (grey) or unobserved (white) random variable, while each edge in E represents a pairwise relationship between two random variables in V .
The joint probability distribution for G is:
[Equation]
where X =
X d | d ∈ V} is the set of unobserved variables and Y
= { Y d | d ∈ V} is the set of cor- responding observed variables.
The scalar Z is a normalizing constant.
For each node, the function φ d ( · ) is the unary potential , describing the compatibility of X d with a corresponding observed variable Y d .
For each edge, the function ψ sd ( · ) is the pairwise potential , describing the compatibility of neighboring variables
X s and X d .
This work considers MRF models limited to pairwise clique potentials.
Given the factorization of the joint distribution deﬁned in Eq.
(1), BP provides an algorithm for inference of the marginal posterior distributions, know as the beliefs, bel d ( X d ) .
BP deﬁnes a message passing scheme for calculation of the beliefs as follows:
[Equation]
where ρ ( s ) denotes the set of neighboring nodes of s .
A message from node s to d is deﬁned as:
[Equation]
Performing inference of random variables in continuous space causes the integral in Eq.
(3) to become intractable.
This motivates the use of efﬁcient algorithms that approximate the message passing scheme of Eq.
(2) and Eq.
3. N ONPARAMETRIC B ELIEF P ROPAGATION
Nonparametric belief propagation (NBP) (Sudderth et al., 2003) uses Gaussian mixtures to represent the beliefs and messages for continuous random variables.
Later works, including Ihler &
McAllester (2009) and Desingh et al.
(2019), further improve upon the tractibility of approximate nonparametric inference by representing beliefs and messages with sets of weighted particles.
These particle-based NBP methods infer an approximation of the beliefs using an iterative message passing algorithm, in which beliefs and messages are updated at each iteration t .
In particular, Desingh et al.
(2019) avoid the expensive message generation of NBP by approximating Eq.
(3) with a “pull” strategy.
A message, m ts → d , outgoing from s to d , is generated by ﬁrst sampling M independent samples from bel t − 1 d ( X d ) then reweighting and resampling from this set.
D IFFERENTIABLE N ONPARAMETRIC B ELIEF P ROPAGATION
We propose a differentiable nonparametric belief propagation (DNBP) method.
DNBP maintains a representation of the uncertainty in the estimate by efﬁciently approximating the marginal posterior distributions encoded in an MRF.
Our method avoids the need to deﬁne hand-crafted functions for each domain by modeling the potentials needed for the computation of the distributions with neural networks that are trained end-to-end.
This hybrid generative-discriminative approach leverages the strengths of both NBP and neural networks.
DNBP uses an iterative, differentiable message passing scheme to infer the beliefs over hidden variables in an MRF.
DNBP approximates the belief and messages in Eq.
(2) and Eq.
(3) at iteration t by sets of N and M weighted particles respectively:
[Equation]
DNBP relies on a “pull” message passing strategy similar to the one presented by Desingh et al.
In this strategy, each iteration of the algorithm is deﬁned in terms of a message update step and a belief update step.
The message update generates a new set of message particles as a reweighted set of samples from the previous iteration’s belief.
Crucially, the weights associated with these updated message samples result from learned probabilistic factors as opposed to handcrafted ones.
Following a message update, the belief update combines information that is incoming to each node from the newly generated messages.
Pseudocode of DNBP’s message and belief update schemes is included in Appendix A.1.
The following sections describe the networks used to compute the message and belief updates.
Unary Potential Functions:
According to the factorization of the MRF joint distribution in Eq.
(1), each unobserved variable X d , for d ∈ V , is related to a corresponding observed variable Y d by the unary potential function φ d ( X d , Y d ) .
DNBP models each unary function with a feedforward neural network.
The unary potential for a particle, x d , given an observed image, y d , is:
[Equation]
6) where f d is a convolutional neural network, l d is a fully connected neural network, and the symbol ⊕ denotes concatenation of feature vectors.
Details of network architectures are given in Appendix A.2, Table 1.
Pairwise Potential Functions: For any pair of hidden variables, X s and X d , which are connected by an edge in E , a pairwise potential function, ψ sd ( X s , X d ) , represents the probabilistic relationship between the two variables.
DNBP models each pairwise potential using a pair of feedforward, fully connected neural networks, ψ sd ( X s , X d ) =
{ ψ ρsd ( · ) , ψ ∼ sd ( · ) } .
The pairwise density network, ψ ρsd ( · ) , evaluates the unnormalized potential for a pair of particles.
The pairwise sampling network, ψ ∼ sd ( · ) , is used to form samples of node s conditioned on node d and vice versa.
Details of network architectures are given in the Appendix A.2, Table 1.
The weight computation is detailed in the pseudocode in Appendix A.1.
Particle Diffusion: DNBP uses a learned particle diffusion model for each hidden variable, modeled as distinct feedforward neural networks, τ ∼ d ( · ) for d ∈ V .
This diffusion model replaces the Gaussian diffusion models typically used by particle-based inference methods.
At the outset of message generation at iteration t , DNBP’s belief particles from iteration t − 1 are resampled then passed through the diffusion model at the beginning of iteration t to form the messages used to update the distributions at iteration t .
Y t
Y t Y t
X t X t
X t
Base Joint
Middle Joint
End Effector (a)
Y t
Y t Y t
X t X t
X t(b)
Y t Y t
X t
X t X t
Y t Y t
Y t 2 X t X t 2 Y t
Y t
Y t
X t 0 X t
X t
Y t Y t
X t
X t(c)
Y t Y t
X t
X t X t
Y t Y t
Y t 2 X t X t 2 Y t
Y t
Y t
X t 0 X t
X t
Y t Y t
X t
X t(d)
Particle Resampling: The ﬁnal operation of the belief update algorithm in NBP is a weighted resampling of belief particles.
This resampling operation is non-differentiable (Karkus et al., 2018;
Jonschkowski et al., 2018).
It follows that the iterative belief update algorithm is non-differentiable due to the resampling step.
DNBP addresses the non-differentiability of the belief update algorithm by relocating the resampling and diffusion operations to the beginning of the message update algorithm.
With this modiﬁcation, the belief update returns a weighted set of particles approximating the marginal beliefs.
The resulting belief density estimate is differentiable up to the beginning of the message update, when particles from the previous iteration were resampled.
The resulting algorithm is differentiable through one belief update and message passing updates.
4. S UPERVISED T RAINING
DNBP’s training approach is inspired by the work of Jonschkowski et al.
(2018) with modiﬁcations to enable learning the potential functions distinct to DNBP.
During training, DNBP uses a set of observation sequences, and a corresponding set of ground truth sequences.
Using the observation sequences, DNBP estimates belief of each unobserved variable at each sequence step.
Then, by maximizing estimated belief at the ground truth label of each unobserved variable, DNBP learns its network parameters by maximum likelihood estimation.
Further details regarding the implementation of the training procedure are discussed in Appendix A.2.
Objective Function:
Given a set of weighted particles representing the belief of X d produced by the inference procedure at iteration t , the density of the belief can be expressed as a mixture of
Gaussians, with a component centered at each particle.
The density of a sample x d can be computed as follows:
[Equation]
DNBP deﬁnes a loss function one each hidden node d ∈ G as:
[Equation]
where x t, ∗ d denotes the ground truth label for node d at sequence step t .
The loss for each hidden node is computed and optimized separately.
At each sequence step during training, DNBP iterates through the nodes of the graph, updating each node’s incoming messages and belief followed by a single optimization step of Eq.
(8) using stochastic gradient descent.
R ESULTS
The capability of DNBP is demonstrated on three challenging articulated tracking tasks.
The ﬁrst two tasks involve visually tracking the articulated joints of simulated articulated structures, as illustrated in Fig.
To increase the difﬁculty of these tasks, simulated clutter 1 in the form of static and dynamic geometric shapes are rendered into the image sequences.
In the second task, we evaluate
DNBP on its ability to track the articulated pose of human hands.
In both experiments, DNBP is directly compared to learned baseline approaches that are not NBP.
5. D ATASETS
Simulated Double Pendulum: To characterize DNBP’s tracking performance under chaotic motion, the double pendulum task was chosen as an initial evaluation.
The double pendulum structure consists of two revolute joints connected to two rigid-body links in series (see Fig.
2a for illustration), which are acted on by gravity.
The pose of the double pendulum is modeled by the 2dimensional position of its two revolute joints, rendered as yellow circles, and one end effector.
The training set on this task consists of 1024 total sequences with 20 frames per sequence while the validation set consists of 150 total sequences with 20 frames per sequence.
Both training and validation sequences are split evenly among three bins of clutter ratio: none, 0 to 0 . 04 and 0 .
04 to 0 .
Of the training and validation sequences with any amount of clutter, half contain static clutter and the other half contain dynamic clutter.
The held-out test set is evenly split among clutter ratio deciles fromto 0 .
95 , thus contains a shift in distribution from the training set, which was limited to clutter ratios below 0 .
Each decile contains 50 sequences with 100 frames per sequence.
For test sequences with any amount of clutter, half contain static clutter and the other half contain dynamic clutter.
Simulated Articulated Spider: The spider task was chosen to further characterize DNBP’s performance using a structure with added articulations and a larger graphical model.
As depicted in
Fig.
2c, the spider is comprised of three revolute-prismatic joints, three purely revolute joints, and six rigid-body links.
An example of the spider is shown in Fig.
2c, in which the joints are rendered as yellow circles and the rigid-body links are rendered as coloured rectangles.
Unlike the double pendulum, which contained a stationary base joint, the spider is not tethered to any position and can move freely throughout the image under simulated joint control.
The training, validation and test set for this task follow the same respective distributions of clutter as were used in the double pendulum datasets.
The training set consists of 2 , 048 total sequences and the validation set consists of 300 sequences.
The training and validation sequences are split evenly among ﬁve bins of clutter ratio: none, 0 to 0 . 04 and 0 .
04 to 0 .
1 to 0 .
2 and 0 .
2 to 0 .
There are 20 frames per sequence in each of the spider datasets.
Both simulated tasks use images of size 128 × 128 pixels.
Ground truth keypoint locations are represented as continuous valued coordinates scaled to range of [ − 1 , +1] .
First-Person Hand Action Benchmark: The FPHAB dataset (Garcia-Hernando et al., 2018) consists of RGB-D image sequences taken from the ﬁrst-person perspective.
Thus, the dataset captures the pose and motion of human hands as they perform typical actions.
This is a challenging dataset with extreme occlusions where complete observations of all the ﬁnger joints are rare.
In total, there are 1175 distinct sequences and 105459 individual image frames.
Each image is labeled with the 3 D position of 21 hand joints (illustration of joint relations shown in center column of Fig. 1).
The best-performing hand pose estimation baseline proposed by Garcia-Hernando et al.
(2018) is used for comparison in the current study.
Just like Garcia-Hernando et al.
, DNBP uses only depth observations.
To ensure fair comparisons with the FPHAB baseline, this study follows the 1 :cross-subject training protocol as described in FPHAB.
5. I MPLEMENTATION D ETAILS
On all three tasks, Adam (Kingma & Ba, 2015) is used for network optimization with a batch size of 6 and models are trained until convergence of the validation loss.
The graphs used by DNBP are shown in Figs.
1, 2b and 2d.
While DNBP uses tree-structured graphs in these experiments, the inference strategy is compatible with graphs containing cycles.
DNBP is trained using 100 particles per message and tested using 200 particles per message.
During training, one message update is performed at each sequence step, while two message updates are used at test time.
The pairwise density, pairwise sampling and diffusion sampling processes of DNBP are deﬁned over the relative translations between neighboring nodes.
The maximum weighted particle from each marginal belief set of DNBP is used during evaluation for comparison with the ground truth.
On both simulated tasks, DNBP is compared to an LSTM recurrent neural network (Hochreiter &
Schmidhuber, 1997).
Both models use image inputs that are normalized channel-wise based on training set statistics.
The total number of trainable parameters between LSTM and DNBP were
A v e r a g e E u c li d ea n E rr o r
( P i x e l s )
[Figure] [Figure]
Ratio
A v e r a g e E u c li d ea n E rr o r
( P i x e l s ) chosen to be similar. For hand tracking, the preprocessing protocol of Xiong et al. (2019), is followed. Notably, preprocessing on the hand tracking task assumes ground truth bounding boxes to ensure fair comparison with the baseline method published by Garcia-Hernando et al.
Similarly, the feature extractor used by DNBP in the following experiments was designed to emulate the feature extractor of compared baseline.
Details of network parameters and inspection of learned relationships are included in the Appendices A.2 and A.6.
5. P ERFORMANCE METRICS
As a quantitative measure of tracking error, average Euclidean error is used.
On the simulated tasks,
Euclidean error is averaged over all images in the test set.
On the hand tracking task, Euclidean error is averaged over all joints per frame then used to calculate the percent of frames satisfying variable error thresholds as used by Garcia-Hernando et al.
Discrete entropy (Shannon, 1948) is used as a quantitative measure of uncertainty estimated by
DNBP.
Discrete entropy is calculated by binning samples from each marginal belief set.
For qualitative analysis of the uncertainty estimated by DNBP, samples from an approximation of the joint posterior distribution (i.e. for collection of all unobserved variables) are formed using a sequential
Monte Carlo sampling approach (Naesseth et al., 2014).
Visualization of these samples are formed by plotting a rendered link between each pair of keypoint samples.
5. D OUBLE P ENDULUM T RACKING R ESULTS
As shown in Fig.
3, the keypoint tracking error of DNBP is directly compared to that of the LSTM baseline on the held-out test set for each keypoint type (base, middle and end effector) across the full range of clutter ratios.
Results from this comparison show that DNBP’s average keypoint tracking error is comparable to the LSTM’s corresponding error for both the mid joint and end effector keypoints, independent of clutter ratio.
For the base joint keypoint, which is stationary at the center position of every image, the LSTM was able to memorize the correct position.
DNBP, which diffuses particles based on the message passing scheme, does not memorize the base joint position and registers a consistently larger error which increased with clutter ratio.
DNBP provides measures of uncertainty associated with its predictions, which are generated according to the algorithmic prior of belief propagation.
Next tested was the hypothesis that the DNBP model would generate increased uncertainty under conditions in which an occluding object is placed into the input images such that it covers portions of the double pendulum.
This test was performed by rendering an occluding block onto a test sequence as shown in Fig.
5a-c.
Under optimal conditions, in which the pendulum is minimally occluded ( < 25% by surface area), the model’s output indicates a low level of uncertainty (see Fig.
5d,f,g.) for each keypoint and each frame.
In contrast, under conditions in which the pendulum is occluded by the superimposed object, the model’s output indicates relatively high levels of uncertainty precisely at frames in which the superimposed object
I npu t
M a r g i n a l
E n t r opy ( N a t s )
Frame
Frame Frame
Frame 54 a. b. c. f. e. d. g.
DN B P
P r e d i c ti on s
G r ound
T r u t h
L S T M
P r e d i c ti on s
Frame
Frame
DN B P
P r e d i c ti on s
Frame 20 a. b. c. d. e. f. g. h. i. occludes a portion ( > 25% ) of the double pendulum (see Fig. 5e,g.). These results demonstrate that the estimate of uncertainty produced by DNBP can identify predictions which are unreliable.
5. A RTICULATED S PIDER T RACKING R ESULTS
After having established the performance characteristics of DNBP on the relatively straightforward double pendulum task we next set out to determine DNBP’s capability for tracking more complex structures.
To this end, the 3-arm spider structure was used as a more challenging articulated pose tracking task.
Each model’s performance was quantitatively assessed on the held-out test set of the articulated spider tracking task using the same approach as described for the double pendulum experiment by varying clutter ratio (Fig. 4).
Similar to the results of the double pendulum experiment, average error on the spider task increases as a function of clutter ratio for both the LSTM and for DNBP.
For clutter ratios between 0 and 0 . 25 , average error for both models remains near 6 pixels then increases consistently with clutter ratio, reaching above 30 pixels of average error for clutter ratios above 0 .
As in the case of the double pendulum experiment, these results demonstrate comparable performance between LSTM and DNBP on an articulated pose tracking task.
Next, a qualitative example of tracking performance under conditions of clutter is shown in Fig.
In Fig.
6(a-c), the ground truth pose is shown amidst distracting shapes across selected frames of a test sequence with clutter ratio of 0 . 25 .
Pose predictions generated by LSTM are shown in Fig.
6(df) and by DNBP in (g-i).
Qualitative assessment of the images indicates both the LSTM and DNBP place their predictions in the correct region of the image.
Additionally, each model is shown to correctly predict the relative positions of the three arms.
Over the sequence, both models track the motion of each keypoint, however appear to struggle with certain keypoint predictions.
5. H UMAN H AND T RACKING R ESULTS
To evaluate DNBP’s capability for application to real-world tasks, the algorithm’s state estimation and tracking performance was evaluated on the FPHAB dataset.
This is a challenging dataset with extreme occlusions where complete observations of all the ﬁnger joints are rare.
Firstly, Euclidean
Ground Truth Estimate Thumb Uncertainty Index Uncertainty Middle Uncertainty Ring Uncertainty Pinky Uncertainty (1) (2) (3) (4) (5) (6) (7) (8)
[Figure]
[Figure]
(mm) <latexit sha1_base64="pxCsljL5+1ZnkmpQIdJjuiELoIU=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkqMeiF48V7Ae0oWy2k3bpZhN3N0IJ/RNePCji1b/jzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHssHM0nQj+hQ8pAzaqzU6WGiuYhlv1xxq+4cZJV4OalAjka//NUbxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bH7vlJxZZUDCWNmShszV3xMZjbSeRIHtjKgZ6WVvJv7ndVMTXvsZl0lqULLFojAVxMRk9jwZcIXMiIkllClubyVsRBVlxkZUsiF4yy+vktZF1bus1u5rlfpNHkcRTuAUzsGDK6jDHTSgCQwEPMMrvDmPzovz7nwsWgtOPnMMf+B8/gBPv5Ap</latexit> ✏
P e r ce n t a g e o f F r a m e s w it
h e rr
{+o r < <latexit sha1_base64="pxCsljL
+}
+1ZnkmpQIdJjuiELoIU=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkqMeiF48V7Ae0oWy2k3bpZhN3N0IJ/RNePCji1b/jzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b
[Figure]
NUbxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bH7vlJxZZUDCWNmShszV3xMZjbSeRIHtjKgZ6WVvJv7ndVMTXvsZl0lqULLFojAVxMRk9jwZcIXMiIkllClubyVsRBVlxkZUsiF4yy
[Figure]
vktZF1bus1u5rlfpNHkcRTuAUzsGDK6jDHTSgCQwEPMMrvDmPzovz7nwsWgtOPnMMf [Figure] error between the estimated and ground truth pose is measured for every frame in the test set.
For this ﬁrst evaluation, DNBP is applied as a frame-by-frame estimator without maintaining its belief over time.
The quantitative results from this experiment, are included in Fig.
8 with direct comparison to a pure neural network baseline.
The results from this experiment indicate that for error thresholds below 50 mm, DNBP will consistently have an accuracy of 95% and above.
Following the comparison against a state of the art baseline, it was hypothesized that DNBP’s performance would improve when applied as a tracking method which maintains belief over time.
To perform this test, DNBP was applied sequentially to each test sequence and evaluated under the same error metric.
The result from this test, as shown in Fig. 8, demonstrates that DNBP does improve in terms of frame error when allowed to track its uncertainty over time.
Qualitative examples (on frames from randomly chosen sequences) showing DNBP’s tracking performance are shown in Fig.
7 and Appendix A.7.
The tracking videos showing the DNBP’s estimates and belief are included in the supplementary material and project webpage: https://sites.google.com/view/diff-nbp.
C ONCLUSION
In this work, we proposed a novel formulation of belief propagation which is differentiable and uses a nonparametric representation of belief.
It was hypothesized that combining maximum likelihood estimation with the nonparametric inference approach would enable end-to-end learning of the probabilistic factors needed for inference.
Results on both qualitative and quantitative experiments demonstrate successful application of this approach and highlight the capability of DNBP to estimate useful measures of uncertainty, which are crucial for applications where incorrect estimates lead to catastrophic decisions, such as robotics.
The current approach is limited by its use of non-differentiable resampling and its demand for a graph model as input.
Exploration of methods to overcome these limitations, such as by incorporating a soft-resampling strategy Karkus et al. (2018), are left as future work.
A A PPENDIX A. A LGORITHM P SEUDOCODE
In this section, pseudo code of the DNBP message passing algorithm is given for reference.
As discussed in Section 4, this algorithm is a differentiable variant of the PMPNBP algorithm Desingh et al.
Algorithm input : Belief set bel n − 1 d ( X d ) =
{ ( w ( i ) d , µ ( i ) d )
[Equation]
Ti =
Incoming messages m n − 1 u → s ( X s ) =
{ ( w ( i ) us , µ ( i ) us ) } Mi =1 for each node u ∈ ρ ( s ) \ d output: Outgoing messages, m ns → d ( X d ) =
( µ ( i ) sd , w ( i ) sd ) }
Mi =
Draw (1 − γ n − 1 ) · M independent samples from bel n − 1 d ( X d ) { µ ( i ) sd ← bel n − 1 d ( X d ) } (1 − γ n − 1 ) · M i = ; 2 Apply particle diffusion to each sampled particle µ ( i ) sd = µ ( i ) sd + τ d ( (cid:15) ) ;
; 4 foreach { do for
] do
Sample
; end
U ; foreach u ∈ ρ
\ d do
W (
) u = M (cid:80)
j = w ( j )
× w (
) u where w ( ij ) u
= ψ sd ( X s = µ ( j ) us , X d = µ ( i ) sd )
; end
[Table]
Results in this work were generated with U set to 10 , while past related work (Desingh et al., 2019) used U = 1 .
It was observed that this modiﬁcation improved training stability during preliminary development.
Note that γ is a hyperparameter that controls the resampling strategy and is set to
in our experiments. γ is used only during training; during evaluation, all M are
A. N ETWORK A RCHITECTURE & T RAINING
For both simulated articulated tracking tasks, the network architecture for each sub-network described in Section 4 is summarized in Table 1.
For the hand tracking task, each network follows the same structure as those in Table 1, with two exceptions: (1) the feature extractor, f s ( · ) , used for hand tracking is based on the architecture used by the FPHAB baseline that was introduced by Ye et al.
(2) each node likelihood network, l s ( · ) , has one additional feature reduction layer of [fc(64, BatchNorm,ReLU)] preceeding the layers of the corresponding network in Table 1.
The following sections describe speciﬁc implementation details used in the supervised training of
DNBP.
To ensure independence from spatial location, the pairwise density, pairwise sampling and diffusion sampling processes of DNBP are deﬁned over the space of transformations between variables.
Specifically, each of these networks takes as input or produces as output a translation between samples of their corresponding random variables.
Gradient Decoupling: The belief weight, w t, ( i ) d , of particle i is proportional to the product of component weights, w t, ( i ) unary
d × w t, ( i ) unary s × w t, ( i ) neigh s , where s is the neighbor of node d from which particle i originated (see Algorithm 2).
Since each of these component weights is produced by a separate potential network (either φ d , ψ ∼ sd , or ψ ρsd respectively), direct optimization of the belief density will lead to interdependence of the potential network gradients during training.
In the context of DNBP, interdependence between different potential functions is inconsistent with the factorization given in
G .
Tompson et al.
(2014) describe a similar phenomenon they refer to as gradient coupling which was addressed by expressing a product of features in log-space which “decouples” the gradients.
To avoid interdependence between potential functions during training, we consider the partial -belief densities which are deﬁned for each node d ∈ V as mixtures of Gaussian density functions:
[Equation] [Equation]
Using these deﬁnitions, direct interaction between the potential networks’ gradients is avoided by maximizing the product of partial-beliefs at the ground truth of each node in log space.
The product of partial-beliefs is deﬁned:
[Equation]
Unary Potentials: During training of DNBP, only those gradients derived from the belief update of each node are used to update the corresponding node’s unary potential network parameters.
Any gradients derived from the outgoing messages of a particular node are manually stopped from propagating to that node’s unary network.
This is done to avoid confounding the objective functions of neighboring nodes, which each rely on the others’ unary network during message passing.
This approach can be implemented with standard deep learning frameworks by dynamically stopping the parameter update of each unary network depending on where in the algorithm its forward pass was registered.
Pairwise Density Networks:
To speed up and stabilize the training of pairwise density potential networks, the following substitution is made during training.
While calculating w ( i ) sd for outgoing message i from node s to d , the summation over incoming messages from u ∈ ρ ( s ) to s is replaced by a single evaluation of:
[Equation]
13) where x ∗ s is the ground truth label of sender node s .
This change improves inference time and reduces memory demands by removing a summation over M particles while also providing more stable training feedback to the network.
This substitution is removed at test time after training is complete.
Pairwise Sampling Networks: The pairwise sampling networks, ψ ∼ s,d , take a random sample of Gaussian noise as input and generate conditional samples using the following rule: [Equation] [Equation]
15) where x s | d is the sample of variable X s conditioned on neighboring sample x d and where (cid:15) is a noise vector sampled from a zero-mean, unit variance multivariate Gaussian distribution with dim ( (cid:15) )
=
Similarly, for sampling in the opposite conditioning direction (node d conditioned on s ), memory efﬁciency is gained by reusing the ψ ∼ s,d network but negating the sampled translation.
[Equation] [Figure] As summarized in Section 5.1, the double pendulum dataset was generated using a modiﬁed version of the OpenAI Brockman et al.
Acrobot environment.
Synthetic geometric shapes are rendered into each image of the dataset to simulate noisy, cluttered environments.
All simulated clutter on the double pendulum task is generated according to the following parameters: 50% of clutter is rendered visually beneath the pendulum while the remaining 50% is rendered on top of the pendulum.
For dynamic clutter, each geometry simulates motion using a random, constant position velocity ( ˙ x , ˙ y ) and orientation velocity ( ˙ θ ).
Position velocities are sampled from N (0 , 0 . 025) .
Orientation velocities are sampled from N (0 , 0 . 05) .
Clutter is simulated as either rectangles with 80% probability or circles with 20% probability.
Clutter rectangles are sized randomly with length of max (0 , l ∼ N (0 . 2 , 0 . 05)) and height of max (0 , h ∼ N (0 . 8 , 0 . 2)) .
Color of clutter rectangles is randomly chosen with RGB of (0 , 204 , 204) or (245 , 87 , 77) .
Clutter circles are sized randomly with radius of max (0 , r ∼ N (0 . 1 , 0 . 1)) and colored randomly with RGB of (204 , 204 , 0) or (96 , 217 , 63) .
Size and color distributions were chosen to ensure clutter visually resembles the double pendulum parts.
The position of each clutter geometry was randomly initialized within 1 .
5 x the extent of the image boundary.
The training and validation datasets were distributed evenly among clutter ratios of [ 0 , 0 − 0 . 04 , and 0 .
04 − 0 .
For the training/validation sequences that included clutter, the number of clutters rendered beneath and on top of the double pendulum was individually randomly sampled from independent Binomial distributions using n = 15 , p = 0 .
To generate the test set, which was uniformly distributed among clutter ratios as described in Section 5.1, rejection sampling was used with variable numbers of rendered geometries.
A. A RTICULATED S PIDER
[Table] Data for the articulated spider tracking task of Section 5.5 was simulated using the Pillow Clark (2015) image processing library.
Three revolute-prismatic joints are all centrally located and treated as the root of the spider’s kinematic tree.
The remaining three revolute joints are attached to pairs of links, forming three distinct ’arms’ of the spider.
Each joint is rendered as a yellow circle while the six rigid-body links are rendered as distinct red, green or blue rectangles respectively.
Size parameters that follow are with respect to rendered image size of 500 x 500 px.
The three inner revoluteprismatic joints include rotational constraints limiting each to a non-overlapping 120 ◦ range of articulation as well as prismatic constraints limiting the extension to within [20 , 80] pixels of translation.
The three purely revolute joints are constrained to rotations between ± 35 ◦ with respect to their local origins.
Each rigid-body link has width of 20 px and height of 80 px pixels while each joint has radius of 10 px.
For every simulated sequence, the spider is initialized with uniformly random root position within the central 180 x 180 px window and uniformly random root orientation from [ 0 , 2 π ].
Furthermore, each joint state is initialized uniformly at random within its particular articulation constraints.
The spider is simulated with dynamics using randomized, constant root, and joint velocities with respect to a time step ( dt ) of 0 .
The root’s position velocities ( ˙ x , ˙ y ) are each sampled from an equally weighted 2 -component Gaussian mixture with means ( +24 , − 24 ) and standard deviations ( 15 , 15 ).
Whereas, the root’s orientation velocity ( ˙ θ ) is sampled from an equally weighted 2 -component Gaussian mixture with means ( +0 . 3 , − 0 . 3 ) and standard deviations ( 0 . 1 , 0 . 1 ).
Each rotational joint’s velocity is sampled from an equally weighted 2 -component Gaussian mixture with means ( +0 . 3 , − 0 . 3 ) and standard deviations ( 0 . 1 , 0 . 1 ).
Similarly, each prismatic joint’s velocity is sampled from an equally weighted 2 -component Gaussian mixture with means ( +500 , − 500 ) and standard deviations ( 60 , 60 ).
Note that if any joint reaches an articulation limit during simulation, the direction of its velocity is reversed.
A. A RTICULATED S PIDER C LUTTER
Clutter generation for the articulated spider tracking task follows a similar generation process as was used for the double pendulum task.
Clutter parameters that follow are with respect to rendered image size of 500 x 500 px and time step ( dt ) of 0 .
50% of clutter is rendered beneath and 50% is rendered on top of the spider.
For dynamic clutter, each geometry simulates motion using a random, constant position velocity ( ˙ x , ˙ y ) and orientation velocity ( ˙ θ ).
Position velocities are sampled from N (0 , 3) while orientation velocities are sampled from N (0 , 0 . 05) .
Clutter is simulated as either a rectangle with 70% probability or a circle with 30% probability.
Clutter rectangles are sized randomly with length of max (0 , l ∼ N (20 , 3)) and height of max (0 , h ∼ N (80 , 5)) .
The color of clutter rectangles is chosen uniformly at random from the same colors as were used for the spider arms.
Clutter circles are sized randomly with a radius of max (0 , r ∼ N (10 , 3)) and colored yellow to match the color of the spider’s joints.
The position of each clutter geometry was randomly initialized within the image boundary.
For the training/validation sequences that included clutter, the number of clutter shapes rendered beneath and on top of the double pendulum was each randomly sampled from independent Binomial distributions using n = 10 , p = 0 .
The test set was generated with uniformly distributed clutter ratios, as described in Section 5.1, using rejection sampling with variable numbers of rendered geometries.
A. L EARNED P AIRWISE I NSPECTION
As further validation of DNBP, the learned pairwise potentials are inspected in Fig. The normalized histogram of pairwise translations computed from the training set for X 1 − X 0 (top) and X 2 − X 1 (bottom) are shown in the left column of Fig. The middle column shows the normalized histogram of samples from learned pairwise sampler networks, ψ ∼ 0 , 1 ( · ) and ψ ∼ 1 , 2 ( · ) .
Finally, the right column shows output from the learned pairwise density networks, ψ ρ 0 , 1 ( · ) and ψ ρ 1 , 2 ( · ) , gen- erated with 100 x 100 uniform samples across pairwise translation space. The qualitative similarity between each learned potential model and the corresponding true distribution of pairwise translations indicates that DNBP is successful in learning to model each pairwise potential factor. The circular pairwise relationships are explained by the fact that each pair of double pendulum keypoints is related by a revolute joint. The effect of simulated gravity in the double pendulum experiment can be observed by the bias of each pairwise potential in favor of the lower half of each plot as indicated by increased likelihood.
[Table]
L i k e li hood L i k e li hood
L i k e li hood L i k e li hood
P o t e n ti a l P o t e n ti a l
P i x e l s P i x e l s
P i x e l s P i x e l s
P i x e l s 1 , 2 ( X| X 1 ) <latexit sha1_base64="XIfKsvstBC6Hg3cHy/Mm8VxmAgA=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARKkhJSkHdFd24rGDbQBPCZDpth04ezNwIJdSNv+LGhSJu/Qt3/o3TNgttPTBwOOdc7twTJIIrsKxvo7Cyura+UdwsbW3v7O6Z+wdtFaeSshaNRSydgCgmeMRawEEwJ5GMhIFgnWB0M/U7D0wqHkf3ME6YF5JBxPucEtCSbx65ieJ+Zp/XJhXHr7k6C9jx7TPfLFtVawa8TOyclFGOpm9+ub2YpiGLgAqiVNe2EvAyIoFTwSYlN1UsIXREBqyraURCprxsdsEEn2qlh
/ux1C8CPFN/T2QkVGocBjoZEhiqRW8q/ud1U+hfehmPkhRYROeL+qnAEONpHbjHJaMgxpoQKrn+K6ZDIgkFXVpJl2AvnrxM2rWqXa9e3dXLjeu8jiI6Rieogmx0gRroFjVRC1H0iJ7RK3oznowX4934mEcLRj5ziP7A+PwBY+WVlA==</latexit>
[Table]
<latexit sha1_
[Table] ="iyEl2D6qVVjeZt1/4B3nDNJhmOo=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARKkhJpKDuim5cVrBtoAlhMp20QycPZm6EEurGX3HjQhG3/oU7/8Zpm4VWDwwczjmXO/cEqeAKLOvLKC0tr6yuldcrG5tb2zvm7l5HJZmkrE0TkUgnIIoJHrM2cBDMSSUjUSBYNxhdT/3uPZOKJ/EdjFPmRWQQ85BTAlryzQM3VdzPrVN7UnN829VZwI5vnfhm1apbM+C/xC5IFRVo+ean209oFrEYqCBK9WwrBS8nEjgVbFJxM8VSQkdkwHqaxiRiystnF0zwsVb6OEykfjHgmfpzIieRUuMo0MmIwFAtelPxP6+XQXjh5TxOM2AxnS8KM4EhwdM6cJ9LRkGMNSFUcv1XTIdEEgq6tIouwV48+S/pnNXtRv3ytlFtXhV1lNEhOkI1ZKNz1EQ3qIXaiKIH9IRe0KvxaDwbb8b7PFoyipl99AvGxzddqpWQ</latexit>
[Figure]
[Figure]
<latexit sha1_base64="+DzxC0LkQFVn9eRdAqhOeVQ6wFU=">AAACAXicbVDLSsNAFL2pr1pfUTeCm8EiVJCSSEDdFd24rGDbQBPCZDpth04ezEyEEurGX3HjQhG3/oU7/8Zpm4VWDwwczjmXO/eEKWdSWdaXUVpaXlldK69XNja3tnfM3b22TDJBaIskPBFuiCXlLKYtxRSnbioojkJOO+Hoeup37qmQLInv1DilfoQHMeszgpWWAvPASyULcvvUmdTcwPF0ViE3sE8Cs2rVrRnQX2IXpAoFmoH56fUSkkU0VoRjKbu2lSo/x0Ixwumk4mWSppiM8IB2NY1xRKWfzy6YoGOt9FA/EfrFCs3UnxM5jqQcR6FORlgN5aI3Ff/zupnqX/g5i9NM0ZjMF/UzjlSCpnWgHhOUKD7WBBPB9F8RGWKBidKlVXQJ9uLJf0n7rG479ctbp9q4KuoowyEcQQ1sOIcG3EATWkDgAZ7gBV6NR+PZeDPe59GSUczswy8YH99qJ5WY</latexit>
[Table]
L i
[Table]
L i k
[Table]
L i k e li
[Table]
L i k e li
[Table]
P o t e n ti a l P o t e n ti a l
P i x e l s P i x e l s
P i x e l s P i x e l s
P i x e l s 0 , 1 ( X| X 0 ) <latexit sha1_base64="iyEl2D6qVVjeZt1/4B3nDNJhmOo=">AAACAXicbVDLSsNAFJ3UV62vqBvBzWARKkhJpKDuim5cVrBtoAlhMp20QycPZm6EEurGX3HjQhG3/oU7/8Zpm4VWDwwczjmXO/cEqeAKLOvLKC0tr6yuldcrG5tb2zvm7l5HJZmkrE0TkUgnIIoJHrM2cBDMSSUjUSBYNxhdT/
[Table]
+C/xC5IFRVo+ean209oFrEYqCBK9WwrBS8nEjgVbFJxM8VSQkdkwHqaxiRiystnF0zwsVb6OEykfjHgmfpzIieRUuMo0MmIwFAtelPxP6+XQXjh5TxOM2AxnS8KM4EhwdM6cJ9LRkGMNSFUcv1XTIdEEgq6tIouwV48+S/pnNXtRv3ytlFtXhV1lNEhOkI1ZKNz1EQ3qIXaiKIH9IRe0KvxaDwbb8b7PFoyipl99AvGxzddqpWQ</latexit>
The pairwise potential functions learned by DNBP in the spider tracking task are visualized as was done in the double pendulum task.
Fig.
10 shows qualitative output from two of the six models.
Only two are shown to avoid redundancy; chosen results are representative of remaining four potential functions.
The left column of Fig.
10 shows the normalized histogram of pairwise translations as computed from the training set for X 1 − X 0 (top) and X 4 − X 1 (bottom).
The middle column of Fig.
10 shows the normalized histogram of samples from learned pairwise sampler networks, ψ ∼ 0 , 1 ( · ) and ψ ∼ 1 , 4 ( · ) .
Finally in the right column of Fig.
10, uniformly sampled output ( 100 x 100 samples across pixel space) of the learned pairwise density networks is shown.
Once again, the visual similarity between each learned potential function and the corresponding true distribution of pairwise translations is an indicator that DNBP is successful in learning to model each pairwise factor.
Observe that the learned potential functions for ψ 1 , 4 ( · ) , which correspond to a revolute articulation, show no bias in favor of the downward conﬁguration.
This result is notably different from the potential functions learned on the double pendulum task and can be explained by the absence of gravity in the spider simulation.
Similarly, the learned models for ψ 0 , 1 ( · ) on the spider task exhibit a torus shape due to the effect of prismatic motion associated with the corresponding joint’s articulation type and constraint.
A. H AND T RACKING R ESULTS
Ground
Truth
Estimate
Thumb
Uncertainty
Index
Uncertainty
Middle
Uncertainty
Ring
Uncertainty
Pinky
Uncertainty (1) (2) (3) (4) (5) (6) (7) (8)
Ground
Truth Estimate Thumb Uncertainty Index Uncertainty Middle Uncertainty Ring Uncertainty Pinky Uncertainty