Original: PRISM: Probabilistic Real-Time Inference in Spatial World Models
Revised: PRISM: Probabilistic Real-Time Inference in Spatial World Models

Original: Abstract: We introduce PRISM, a method for real-time ﬁltering in a probabilistic generative model of agent motion and visual perception.
Revised: Abstract: We introduce PRISM, a method for real-time ﬁltering in a probabilistic generative model of agent motion and visual perception.

Original: Previous approaches either lack uncertainty estimates for the map and agent state, do not run in real-time, do not have a dense scene representation or do not model agent dynamics.
Revised: Previous approaches either lack uncertainty estimates for the map and agent state, do not run in real-time, do not have a dense scene representation or do not model agent dynamics.

Original: Our solution reconciles all of these aspects.
Revised: Our solution reconciles all of these aspects.

Original: We start from a predeﬁned state-space model which combines differentiable rendering and 6-DoF dynamics.
Revised: We start from a predeﬁned state-space model which combines differentiable rendering and 6-DoF dynamics.

Original: Probabilistic inference in this model amounts to simultaneous localisation and mapping (SLAM) and is intractable.
Revised: Probabilistic inference in this model amounts to simultaneous localisation and mapping (SLAM) and is intractable.

Original: We use a series of approximations to Bayesian inference to arrive at probabilistic map and state estimates.
Revised: We use a series of approximations to Bayesian inference to arrive at probabilistic map and state estimates.

Original: We take advantage of well-established methods and closed-form updates, preserving accuracy and enabling real-time capability.
Revised: We take advantage of well-established methods and closed-form updates, preserving accuracy and enabling real-time capability.

Original: The proposed solution runs at 10Hz real-time and is similarly accurate to state-of-the-art SLAM in small to medium-sized indoor environments, with highspeed UAV and handheld camera agents (Blackbird, EuRoC and TUM-RGBD).
Revised: The proposed solution runs at 10Hz real-time and is similarly accurate to state-of-the-art SLAM in small to medium-sized indoor environments, with highspeed UAV and handheld camera agents (Blackbird, EuRoC and TUM-RGBD).

Original:  Introduction
Revised:  Introduction

Original: Moving agents perceive streams of information, typically a mix of RGB images, depth and inertial measurements.
Revised: Moving agents perceive streams of information, typically a mix of RGB images, depth and inertial measurements.

Original: Probabilistic generative models [1] are a principled way to formalise the synthesis of this data, and from these models inference can be derived through Bayes’ rule.
Revised: Probabilistic generative models [1] are a principled way to formalise the synthesis of this data, and from these models inference can be derived through Bayes’ rule.

Original: We focus on exactly such inference and target the agent states and the scene map, a problem known as simultaneous localisation and mapping (SLAM).
Revised: We focus on exactly such inference and target the agent states and the scene map, a problem known as simultaneous localisation and mapping (SLAM).

Original: We treat it as a posterior approximation for a given state-space model, such that the combination is useful for model-based control: the posterior inference serves as a state estimator and the predictive state-space model as a simulator with which to plan ahead [2].
Revised: We treat it as a posterior approximation for a given state-space model, such that the combination is useful for model-based control: the posterior inference serves as a state estimator and the predictive state-space model as a simulator with which to plan ahead [2].

Original: To pave the way towards decision making, we believe an inference method should have:
Revised: To pave the way towards decision making, we believe an inference method should have:

Original: • a compatible predictive model for both RGB-D images and 6-DoF dynamics; • principled state and map uncertainty; • real-time performance on commodity hardware; • state-of-the-art localisation accuracy.
Revised: • a compatible predictive model for both RGB-D images and 6-DoF dynamics; • principled state and map uncertainty; • real-time performance on commodity hardware; • state-of-the-art localisation accuracy.

Original: We motivate these requirements further in appendix J. Prominent methods like LSD-SLAM [3],
Revised: We motivate these requirements further in appendix J. Prominent methods like LSD-SLAM [3],

Original: ORB-SLAM [4], DSO [5] have propelled visual SLAM forward, with heavy focus on large-scale localisation.
Revised: ORB-SLAM [4], DSO [5] have propelled visual SLAM forward, with heavy focus on large-scale localisation.

Original: The core of modern large-scale SLAM is maximum a-posteriori (MAP) smoothing in a probabilistic factor graph [6, 7].
Revised: The core of modern large-scale SLAM is maximum a-posteriori (MAP) smoothing in a probabilistic factor graph [6, 7].

Original: At present this demands sparsity assumptions for computational feasibility, which obstructs the tight integration of dense maps and rendering.
Revised: At present this demands sparsity assumptions for computational feasibility, which obstructs the tight integration of dense maps and rendering.

Original: Nonetheless, for smaller scenes the recent popularity of neural models (e.g. NERF [8]) has sparked interest in inference through a renderer (e.g. [9, 10, 11]), but dynamics modelling and uncertainty have remained out of scope.
Revised: Nonetheless, for smaller scenes the recent popularity of neural models (e.g. NERF [8]) has sparked interest in inference through a renderer (e.g. [9, 10, 11]), but dynamics modelling and uncertainty have remained out of scope.

Original: Conversely, classical ﬁltering comes with dynamics and uncertainty in real-time (e.g. [12, 13, 14]), but over time has given way to large-scale smoothing [6] and to our knowledge has not been well explored for the integration of dense differentiable rendering and dynamics on a moderate scale.
Revised: Conversely, classical ﬁltering comes with dynamics and uncertainty in real-time (e.g. [12, 13, 14]), but over time has given way to large-scale smoothing [6] and to our knowledge has not been well explored for the integration of dense differentiable rendering and dynamics on a moderate scale.

Original: Overall, we ﬁnd there is a need for a cohesive inference solution that satisﬁes our requirements.
Revised: Overall, we ﬁnd there is a need for a cohesive inference solution that satisﬁes our requirements.

Original: We thus contribute by meeting all the above goals, emphasising the link to a predictive model (ﬁg. 1).
Revised: We thus contribute by meeting all the above goals, emphasising the link to a predictive model (ﬁg. 1).

Original: We start from the generative model of Mirchev et al.
Revised: We start from the generative model of Mirchev et al.

Original: [15], who combine differentiable rendering and agent dynamics in a probabilistic framework.
Revised: [15], who combine differentiable rendering and agent dynamics in a probabilistic framework.

Original: The authors considered stochastic variational inference for this model, applying it off-line with runtime orders of magnitude too long for on-line use.
Revised: The authors considered stochastic variational inference for this model, applying it off-line with runtime orders of magnitude too long for on-line use.

Original: We pursue an alternative route for real-time inference: from the generative assumptions we derive approximations to the true marginal ﬁlters over the last state and map [16].
Revised: We pursue an alternative route for real-time inference: from the generative assumptions we derive approximations to the true marginal ﬁlters over the last state and map [16].

Original: By focusing on recursive ﬁltering updates, we identify where established probabilistic inference and computer vision techniques can be used, putting emphasis on fast closed-form updates.
Revised: By focusing on recursive ﬁltering updates, we identify where established probabilistic inference and computer vision techniques can be used, putting emphasis on fast closed-form updates.

Original: We ﬁnd this divide-and-conquer strategy is a good compromise for achieving the aforementioned objectives under computational constraints.
Revised: We ﬁnd this divide-and-conquer strategy is a good compromise for achieving the aforementioned objectives under computational constraints.

Original: We evaluate the proposed solution on two unmanned aerial vehicle (UAV) data sets [17, 18] and on
Revised: We evaluate the proposed solution on two unmanned aerial vehicle (UAV) data sets [17, 18] and on

Original: TUM-RGBD [19].
Revised: TUM-RGBD [19].

Original: Our method PRISM runs at 10 Hz real-time with similar localisation accuracy to state-of-the-art SLAM in moderately-sized indoor environments.
Revised: Our method PRISM runs at 10 Hz real-time with similar localisation accuracy to state-of-the-art SLAM in moderately-sized indoor environments.

Original: It provides uncertainty estimates and features a predictive distribution that can both render images and forecast the agent’s movement.
Revised: It provides uncertainty estimates and features a predictive distribution that can both render images and forecast the agent’s movement.

Original:  Related Work
Revised:  Related Work

Original: Generative models Generative state-space models simulate the formation of observed data over time in a Markov chain [1, 12, 20, 21, 22, 23], serving as world models [24, 25].
Revised: Generative models Generative state-space models simulate the formation of observed data over time in a Markov chain [1, 12, 20, 21, 22, 23], serving as world models [24, 25].

Original: With their agent dynamics and state-to-observation emission models we can imagine future rollouts for planning [2, 26, 27, 28, 29, 30, 31, 32, 33].
Revised: With their agent dynamics and state-to-observation emission models we can imagine future rollouts for planning [2, 26, 27, 28, 29, 30, 31, 32, 33].

Original: We abide by this framework and design a posterior inference for a spatial state-space model, to enable on-line control.
Revised: We abide by this framework and design a posterior inference for a spatial state-space model, to enable on-line control.

Original: Among such models (e.g. [34, 35, 36, 37, 38, 39, 40]), we tailor our inference to the model of Mirchev et al.
Revised: Among such models (e.g. [34, 35, 36, 37, 38, 39, 40]), we tailor our inference to the model of Mirchev et al.

Original: It scales to 3D with rendering and 6-DoF dynamics.
Revised: It scales to 3D with rendering and 6-DoF dynamics.

Original: We contribute a real-time inference that ﬁts its probabilistic formulation.
Revised: We contribute a real-time inference that ﬁts its probabilistic formulation.

Original: SLAM through image synthesis
Revised: SLAM through image synthesis

Original: The assumed generative model renders RGB-D images, which is related to SLAM through full-image synthesis.
Revised: The assumed generative model renders RGB-D images, which is related to SLAM through full-image synthesis.

Original: Traditional methods feature varied maps, from volumetric to surfels (e.g. [41, 42, 43, 44, 45, 46, 47, 48, 49]), and commonly estimate new camera poses by aligning new observations to a rendered image with variants of point-to-plane ICP with photometric consistency [50, 51, 52, 53, 44].
Revised: Traditional methods feature varied maps, from volumetric to surfels (e.g. [41, 42, 43, 44, 45, 46, 47, 48, 49]), and commonly estimate new camera poses by aligning new observations to a rendered image with variants of point-to-plane ICP with photometric consistency [50, 51, 52, 53, 44].

Original: We extend this optimisation with dynamics in our approximate state ﬁlter [54].
Revised: We extend this optimisation with dynamics in our approximate state ﬁlter [54].

Original: A recent trend is to use implicit scene representations like NERF (e.g. [8, 55, 56]) with high rendering ﬁdelity.
Revised: A recent trend is to use implicit scene representations like NERF (e.g. [8, 55, 56]) with high rendering ﬁdelity.

Original: Gradient-based pose inference through NERF-like rendering has received attention [57, 58], with iMAP [11] and NICE-SLAM [9] being two real-time solutions.
Revised: Gradient-based pose inference through NERF-like rendering has received attention [57, 58], with iMAP [11] and NICE-SLAM [9] being two real-time solutions.

Original: The mapping runtime of such methods is weighed down by optimisation through the renderer.
Revised: The mapping runtime of such methods is weighed down by optimisation through the renderer.

Original: Rendering can be sped up by decomposing parameters over space, e.g. by using voxels or primitives [59, 60, 61, 62, 63], but how to update neural maps in closed form remains unclear.
Revised: Rendering can be sped up by decomposing parameters over space, e.g. by using voxels or primitives [59, 60, 61, 62, 63], but how to update neural maps in closed form remains unclear.

Original: Therefore, we rely on vanilla voxel grid maps [15, 64], as their probabilistic treatment and closed-form updates are straightforward, leaving implicit representations for future work.
Revised: Therefore, we rely on vanilla voxel grid maps [15, 64], as their probabilistic treatment and closed-form updates are straightforward, leaving implicit representations for future work.

Original: We note that none of the aforementioned methods incorporate dynamics and uncertainty, which distinguishes our approach.
Revised: We note that none of the aforementioned methods incorporate dynamics and uncertainty, which distinguishes our approach.

Original: Probabilistic SLAM inference SLAM ﬁlters are thoroughly explored for ﬂat 2D modelling [65, 12, 66, 13, 67, 68, 69], but have been superseded by MAP smoothing in modern visual SLAM (e.g. [3, 4, 5, 70, 71, 72, 73]), primarily due to scalability concerns [6, 74].
Revised: Probabilistic SLAM inference SLAM ﬁlters are thoroughly explored for ﬂat 2D modelling [65, 12, 66, 13, 67, 68, 69], but have been superseded by MAP smoothing in modern visual SLAM (e.g. [3, 4, 5, 70, 71, 72, 73]), primarily due to scalability concerns [6, 74].

Original: However, as of now smoothing is not computationally feasible without sparsity assumptions.
Revised: However, as of now smoothing is not computationally feasible without sparsity assumptions.

Original: We therefore reexamine ﬁltering for differentiable rendering, as we aim to obtain a dense map posterior with uncertainty in real-time (see appendix J for further motivation).
Revised: We therefore reexamine ﬁltering for differentiable rendering, as we aim to obtain a dense map posterior with uncertainty in real-time (see appendix J for further motivation).

Original: Filters may beneﬁt from the dense modelling of observations [74], which aligns with our objective, and we will demonstrate they can be a feasible solution for moderately-sized indoor environments.
Revised: Filters may beneﬁt from the dense modelling of observations [74], which aligns with our objective, and we will demonstrate they can be a feasible solution for moderately-sized indoor environments.

Original: For the states, we use a Laplace approximation [75] and velocity updates similar to those in extended Kalman ﬁlters [12].
Revised: For the states, we use a Laplace approximation [75] and velocity updates similar to those in extended Kalman ﬁlters [12].

Original: For the map, occupancy grids are a common probabilistic choice [64, 66, 76] and closed-form mapping has been used in that context [69].
Revised: For the map, occupancy grids are a common probabilistic choice [64, 66, 76] and closed-form mapping has been used in that context [69].

Original: To enable rendering we provide a similar derivation, but for a signed distance function (SDF), which is related.
Revised: To enable rendering we provide a similar derivation, but for a signed distance function (SDF), which is related.

Original: Probabilistic SDF mapping dates back to Curless and Levoy [41], and SDF updates have a well-known probabilistic interpretation [77, 78].
Revised: Probabilistic SDF mapping dates back to Curless and Levoy [41], and SDF updates have a well-known probabilistic interpretation [77, 78].

Original: We use these approximations to arrive at a holistic probabilistic solution that scales to dense 3D modelling in real-time.
Revised: We use these approximations to arrive at a holistic probabilistic solution that scales to dense 3D modelling in real-time.

Original:  Overview
Revised:  Overview

Original: We approach on-line SLAM inference with two aims in mind.
Revised: We approach on-line SLAM inference with two aims in mind.

Original: First, we want to harmonise our map and state estimation with a predictive model.
Revised: First, we want to harmonise our map and state estimation with a predictive model.

Original: Second, we want to quantify uncertainty: estimates and predictions should account for modelling inaccuracies as well as measurement and process noise.
Revised: Second, we want to quantify uncertainty: estimates and predictions should account for modelling inaccuracies as well as measurement and process noise.

Original: Both are important for autonomous decision making.
Revised: Both are important for autonomous decision making.

Original: To achieve this, we derive a Bayesian posterior in the probabilistic model of Mirchev et al.
Revised: To achieve this, we derive a Bayesian posterior in the probabilistic model of Mirchev et al.

Original: [15], to ensure that inference matches the forward model.
Revised: [15], to ensure that inference matches the forward model.

Original: Before we delve into our proposed solution, we present a practical summary.
Revised: Before we delve into our proposed solution, we present a practical summary.

Original: At every time step:
Revised: At every time step:

Original: we point-estimate the agent’s pose using gradient descent, involving
Revised: we point-estimate the agent’s pose using gradient descent, involving

Original: geometry and dynamics
Revised: geometry and dynamics

Original: we extend the pose with a Gaussian covariance matrix through a Laplace approximation.
Revised: we extend the pose with a Gaussian covariance matrix through a Laplace approximation.

Original: with the pose, we estimate the agent’s current velocity in closed form.
Revised: with the pose, we estimate the agent’s current velocity in closed form.

Original: with the pose and the current observation, we update the map in closed form.
Revised: with the pose and the current observation, we update the map in closed form.

Original: We use well-established methods for the above.
Revised: We use well-established methods for the above.

Original: In 1.
Revised: In 1.

Original: we combine assumed density ﬁltering [79], point-to-plane ICP [50] and photometric alignment [51, 52].
Revised: we combine assumed density ﬁltering [79], point-to-plane ICP [50] and photometric alignment [51, 52].

Original: In 2.
Revised: In 2.

Original: we use a Laplace approximation [75, 80].
Revised: we use a Laplace approximation [75, 80].

Original: In 3.
Revised: In 3.

Original: we use linear-Gaussian updates, akin to Kalman ﬁlters [12].
Revised: we use linear-Gaussian updates, akin to Kalman ﬁlters [12].

Original: In 4.
Revised: In 4.

Original: we ﬁrst derive generic closed-form map updates, which boil down to SDF updates [41] for our generative assumptions.
Revised: we ﬁrst derive generic closed-form map updates, which boil down to SDF updates [41] for our generative assumptions.

Original: We contribute by deriving a holistic Bayesian inference from the generative model we started with.
Revised: We contribute by deriving a holistic Bayesian inference from the generative model we started with.

Original: In doing so, we identify where traditional techniques are applicable to make a practical algorithm.
Revised: In doing so, we identify where traditional techniques are applicable to make a practical algorithm.

Original:  Methods
Revised:  Methods

Original: In the following we will denote generative distributions, true posterior distributions and conditionals with p ( · ) .
Revised: In the following we will denote generative distributions, true posterior distributions and conditionals with p ( · ) .

Original: Respectively, approximate distributions will be denoted with q ( · ) .
Revised: Respectively, approximate distributions will be denoted with q ( · ) .

Original: Approximation steps will be indicated by ≈ in equations.
Revised: Approximation steps will be indicated by ≈ in equations.

Original: We use q φ ( · ) to subsume estimated distribution parameters into φ .
Revised: We use q φ ( · ) to subsume estimated distribution parameters into φ .

Original: A subscript · t indicates that a variable or a distribution is different at every time step.
Revised: A subscript · t indicates that a variable or a distribution is different at every time step.

Original: 4.1 Background
Revised: 4.1 Background

Original: We start with an overview of the generative model of Mirchev et al.
Revised: We start with an overview of the generative model of Mirchev et al.

Original: [15] from which we will derive the inference.
Revised: [15] from which we will derive the inference.

Original: We assume a sequence of RGB-D observations x 1: T and a sequence of agent states z 1: T driven by controls u 1: T − 1 form a Markovian state-space model.
Revised: We assume a sequence of RGB-D observations x 1: T and a sequence of agent states z 1: T driven by controls u 1: T − 1 form a Markovian state-space model.

Original: Each observation is constructed from a respective state with a rendering emission model p ( x t | M , z t ) , where M is a global latent random variable for a dense map.
Revised: Each observation is constructed from a respective state with a rendering emission model p ( x t | M , z t ) , where M is a global latent random variable for a dense map.

Original: A transition model p ( z t | z t − 1 , u t − 1 ) accounts for the agent dynamics, where u t are known acceleration controls.
Revised: A transition model p ( z t | z t − 1 , u t − 1 ) accounts for the agent dynamics, where u t are known acceleration controls.

Original: Assuming z 1 is given, the joint distribution is: p
Revised: Assuming z 1 is given, the joint distribution is: p

Original: [Equation]
Revised: [Equation]

Original: , z
Revised: , z

Original: [Equation]
Revised: [Equation]

Original: : T , x
Revised: : T , x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The map is a 3D voxel grid of occupancy and color–each cell contains four values.
Revised: The map is a 3D voxel grid of occupancy and color–each cell contains four values.

Original: The emission is fully-differentiable and performs volumetric raymarching, searching for a unique hit position at a surface along each ray [81].
Revised: The emission is fully-differentiable and performs volumetric raymarching, searching for a unique hit position at a surface along each ray [81].

Original: The transition performs Euler integration, using the acceleration controls and maintained velocity from the latent state.
Revised: The transition performs Euler integration, using the acceleration controls and maintained velocity from the latent state.

Original: Appendix B and the original paper have the details.
Revised: Appendix B and the original paper have the details.

Original: 4.2 Posterior Choice
Revised: 4.2 Posterior Choice

Original: First we need to choose which posterior to approximate.
Revised: First we need to choose which posterior to approximate.

Original: For example, Mirchev et al.
Revised: For example, Mirchev et al.

Original: [15] approximate the full posterior over the map and all states p ( M , z 2: T | x 1: T , u 1: T − 1 , z 1 ) with variational inference [82].
Revised: [15] approximate the full posterior over the map and all states p ( M , z 2: T | x 1: T , u 1: T − 1 , z 1 ) with variational inference [82].

Original: While generic, this approach is slowed down by rendering at every optimisation step [54], and the inevitable stochastic optimisation demands multiple steps until convergence.
Revised: While generic, this approach is slowed down by rendering at every optimisation step [54], and the inevitable stochastic optimisation demands multiple steps until convergence.

Original: In addition, estimating the posterior over all states scatters the optimisation budget across the whole trajectory.
Revised: In addition, estimating the posterior over all states scatters the optimisation budget across the whole trajectory.

Original: To enable real-time inference we target an alternative posterior, the ﬁlter p ( M , z t | x 1: t , u 1: t − 1 , z 1 ) , as the last state belief is enough for planning ahead [2].
Revised: To enable real-time inference we target an alternative posterior, the ﬁlter p ( M , z t | x 1: t , u 1: t − 1 , z 1 ) , as the last state belief is enough for planning ahead [2].

Original: Since ﬁlters can be updated recursively [16, 80], we can use closed-form updates for fast inference.
Revised: Since ﬁlters can be updated recursively [16, 80], we can use closed-form updates for fast inference.

Original: Still, maintaining the joint distribution is too costly because of the large dense 3D map M . 

Revised: Still, maintaining the joint distribution is too costly because of the large dense 3D map M .

Original: Instead, we approximate the two marginal ﬁlters:
Revised: Instead, we approximate the two marginal ﬁlters:

Original: [Equation]
Revised: [Equation]

Original: where H t = x 1: t , u 1: t − 1 , z 1 .
Revised: where H t = x 1: t , u 1: t − 1 , z 1 .

Original: More details about this modelling choice can be found in appendix A.
Revised: More details about this modelling choice can be found in appendix A.

Original: We draw attention to the shorthand notation p ( · | H t ) , which will appear again in the following.
Revised: We draw attention to the shorthand notation p ( · | H t ) , which will appear again in the following.

Original: 4.3 Approximate Filtering
Revised: 4.3 Approximate Filtering

Original: For both marginal ﬁlters
Revised: For both marginal ﬁlters

Original: [Equation]
Revised: [Equation]

Original: we will arrive at adequate approximations by reusing the following equation:
Revised: we will arrive at adequate approximations by reusing the following equation:

Original: [Equation]
Revised: [Equation]

Original: This is a classic recursive expression of the Bayes ﬁlter [16].
Revised: This is a classic recursive expression of the Bayes ﬁlter [16].

Original: Starting from each true marginal posterior, we will ﬁrst expand the joint, then use eq.
Revised: Starting from each true marginal posterior, we will ﬁrst expand the joint, then use eq.

Original: (1) and apply a set of approximations.
Revised: (1) and apply a set of approximations.

Original: Next we will discuss our ﬁnal result, we defer the detailed derivation of both ﬁlters to appendices C and E.
Revised: Next we will discuss our ﬁnal result, we defer the detailed derivation of both ﬁlters to appendices C and E.

Original: 4.3.1 Marginal Map Filter
Revised: 4.3.1 Marginal Map Filter

Original: We begin with the map approximation, starting from the true marginal Bayes ﬁlter:
Revised: We begin with the map approximation, starting from the true marginal Bayes ﬁlter:

Original: [Equation]
Revised: [Equation]

Original: Equations (2) and (3) hide a few approximations detailed in appendix C.
Revised: Equations (2) and (3) hide a few approximations detailed in appendix C.

Original: The resulting solution takes a nominal state sample ˆ z t , with which a map update q ( M | x t , ˆ z t ) is applied to the previous map belief q φ t − 1 ( M ) .
Revised: The resulting solution takes a nominal state sample ˆ z t , with which a map update q ( M | x t , ˆ z t ) is applied to the previous map belief q φ t − 1 ( M ) .

Original: We set ˆ z t to the mean of the current state belief q φ t ( z t ) .
Revised: We set ˆ z t to the mean of the current state belief q φ t ( z t ) .

Original: Accepting some bias, we do this for speed as it is our best guess for z t without extra computation.
Revised: Accepting some bias, we do this for speed as it is our best guess for z t without extra computation.

Original: Intuitively, the map update q ( M | x t , ˆ z t ) populates the map such that the observation x t can be reconstructed.
Revised: Intuitively, the map update q ( M | x t , ˆ z t ) populates the map such that the observation x t can be reconstructed.

Original: Our derivation of the updates is similar to the one by Grisetti et al.
Revised: Our derivation of the updates is similar to the one by Grisetti et al.

Original: [69] for 2D occupancy maps, but now applied to 3D.
Revised: [69] for 2D occupancy maps, but now applied to 3D.

Original: The above approximation is generic, agnostic to the speciﬁc map and rendering assumptions.
Revised: The above approximation is generic, agnostic to the speciﬁc map and rendering assumptions.

Original: In practice, we need a closed-form map update q ( M | x t , ˆ z t ) that is faithful to the emission p ( x t | ˆ z t , M ) .
Revised: In practice, we need a closed-form map update q ( M | x t , ˆ z t ) that is faithful to the emission p ( x t | ˆ z t , M ) .

Original: In this work, we follow Mirchev et al.
Revised: In this work, we follow Mirchev et al.

Original: [15] and use a Gaussian map that factorises over voxels:
Revised: [15] and use a Gaussian map that factorises over voxels:

Original: [Equation]
Revised: [Equation]

Original: Here the indices ijk run over voxels in a 3D grid.
Revised: Here the indices ijk run over voxels in a 3D grid.

Original: For this speciﬁc representation and the assumed surface-based rendering, we identify that the map update q ( M | x t , ˆ z t ) can be implemented as a probabilistic signed distance function (SDF) update [41].
Revised: For this speciﬁc representation and the assumed surface-based rendering, we identify that the map update q ( M | x t , ˆ z t ) can be implemented as a probabilistic signed distance function (SDF) update [41].

Original: We provide the technical details in appendix D. SDF updates for voxel maps are a traditional concept in computer vision, and prior work has considered their probabilistic interpretation before [77, 78].
Revised: We provide the technical details in appendix D. SDF updates for voxel maps are a traditional concept in computer vision, and prior work has considered their probabilistic interpretation before [77, 78].

Original: We contribute by identifying the place of such updates in a probabilistic ﬁlter that follows the generative model of [15].
Revised: We contribute by identifying the place of such updates in a probabilistic ﬁlter that follows the generative model of [15].

Original: A detailed discussion of how the above relates to classical SDF update equations can be found in appendix D.
Revised: A detailed discussion of how the above relates to classical SDF update equations can be found in appendix D.

Original: The above approximations are motivated by the real-time constraint.
Revised: The above approximations are motivated by the real-time constraint.

Original: For example, one could optimise eq.
Revised: For example, one could optimise eq.

Original: (2) directly with gradient descent through the renderer, but evaluating the emission is expensive and hinders accurate convergence on a budget.
Revised: (2) directly with gradient descent through the renderer, but evaluating the emission is expensive and hinders accurate convergence on a budget.

Original: This is particularly true when uncertainty estimates are desirable, as optimisation would then be stochastic and gradients noisy [83].
Revised: This is particularly true when uncertainty estimates are desirable, as optimisation would then be stochastic and gradients noisy [83].

Original: In contrast, the derived one-shot map updates are meant to have a cost similar to emitting just once, while capturing uncertainty as well.
Revised: In contrast, the derived one-shot map updates are meant to have a cost similar to emitting just once, while capturing uncertainty as well.

Original: We show some of the differences between the two approaches in section 5.3.
Revised: We show some of the differences between the two approaches in section 5.3.

Original: 4.3.2 Marginal State Filter
Revised: 4.3.2 Marginal State Filter

Original: Similarly, for the state ﬁlter we start from the true marginal and arrive at approximations via eq.
Revised: Similarly, for the state ﬁlter we start from the true marginal and arrive at approximations via eq.

Original: [Equation]
Revised: [Equation]

Original: We detail all the approximations that lead to eq.
Revised: We detail all the approximations that lead to eq.

Original: (4) in appendix E. In eq.
Revised: (4) in appendix E. In eq.

Original: (4) we have three terms: an image reconstruction likelihood, a Gaussian pose prior and a linear Gaussian velocity conditional given a pose.
Revised: (4) we have three terms: an image reconstruction likelihood, a Gaussian pose prior and a linear Gaussian velocity conditional given a pose.

Original: The latter two we obtain analytically with a linear approximation of the transition model and the previous Gaussian belief q φ t − 1 ( z t − 1 ) (c.f. appendix E).
Revised: The latter two we obtain analytically with a linear approximation of the transition model and the previous Gaussian belief q φ t − 1 ( z t − 1 ) (c.f. appendix E).

Original: First, using the ﬁrst two terms of eq.
Revised: First, using the ﬁrst two terms of eq.

Original: (4) we deﬁne a maximum a-posteriori (MAP) objective for pose optimisation:
Revised: (4) we deﬁne a maximum a-posteriori (MAP) objective for pose optimisation:

Original: [Equation]
Revised: [Equation]

Original: Here, ˆ M is a nominal map sample set to the mean of the previous map belief q φ t − 1 ( M ) .
Revised: Here, ˆ M is a nominal map sample set to the mean of the previous map belief q φ t − 1 ( M ) .

Original: The term log q t ( z pose t | u t − 1 , H t − 1 ) is an approximate dynamics prior over the current pose, it makes the pose respect the transition model.
Revised: The term log q t ( z pose t | u t − 1 , H t − 1 ) is an approximate dynamics prior over the current pose, it makes the pose respect the transition model.

Original: The term log p ( x t | ˆ M , z pose t ) represents reconstructing the current observation, optimising it for the current pose will align the observation to the map.
Revised: The term log p ( x t | ˆ M , z pose t ) represents reconstructing the current observation, optimising it for the current pose will align the observation to the map.

Original: However, evaluating this rendering term in every gradient step is inefﬁcient.
Revised: However, evaluating this rendering term in every gradient step is inefﬁcient.

Original: Because of this, we replace it with the prediction-to-observation objective used by Kayalibay et al.
Revised: Because of this, we replace it with the prediction-to-observation objective used by Kayalibay et al.

Original: [54], Nießner et al.
Revised: [54], Nießner et al.

Original: [45], Newcombe et al.
Revised: [45], Newcombe et al.

Original: We refer to [54] for further motivation and we list the technical details in appendix E.
Revised: We refer to [54] for further motivation and we list the technical details in appendix E.

Original: The above optimisation gives us a MAP pose estimate, which we denote with µ pose t .
Revised: The above optimisation gives us a MAP pose estimate, which we denote with µ pose t .

Original: Next, we apply a Laplace approximation [75] around it to obtain a full covariance matrix Σ pose t which captures the curvature of the objective.
Revised: Next, we apply a Laplace approximation [75] around it to obtain a full covariance matrix Σ pose t which captures the curvature of the objective.

Original: This leaves us with a full Gaussian belief over the current pose:
Revised: This leaves us with a full Gaussian belief over the current pose:

Original: [Equation]
Revised: [Equation]

Original: Appendix discusses further.
Revised: Appendix discusses further.

Original: [Figure]
Revised: [Figure]

Original: can combinethis
Revised: can combine

Original: Gaussian with the
Revised: Gaussian with the

Original: Gaussian velocity conditional q t
Revised: Gaussian velocity conditional q t

Original: [Equation]
Revised: [Equation]

Revised: z vel

Revised: [Equation]

Revised: pose

Revised: [Equation]

Original: , u
Revised: , u

Original: [Equation]
Revised: [Equation]

Original: , H
Revised: , H

Original: [Equation]
Revised: [Equation]

Original: ) (the third term in eq.
Revised: ) (the third term in eq

Original: (4)) into a full-state belief in closed form:
Revised: (4)) into a full-state belief in closed form

Original: [Equation]
Revised: [Equation]

Original: This is approximate, as q t ( z vel t | z pose t , u t − 1 , H t − 1 ) is not informed of the latest observation x t .

Original: We do it for speed and ﬁnd it does not harm localisation in practice.
Revised: This is approximate, we do it for speed and ﬁnd it does not harm localisation in practice.

Original: Appendix E has the details.
Revised: Appendix E describes how the linear Gaussian terms come to be in more detail.

Original:  Experiments
Revised:  Experiments

Original: Originally we set out with a few goals: the inference method should be faithful to the generative assumptions, it should quantify uncertainty and it should run in real-time.
Revised: Originally we set out with a few goals: the inference method should be faithful to the generative assumptions, it should quantify uncertainty and it should run in real-time.

Original: What follows is an empirical analysis of these aspects.
Revised: What follows is an empirical analysis of these aspects.

Original: We evaluate on the EuRoC [17], Blackbird [18] and TUM-RGBD [19] data sets.
Revised: We evaluate on the EuRoC [17], Blackbird [18] and TUM-RGBD [19] data sets.

Original: The agent in the former two is an unmanned aerial vehicle (UAV), with speed of up to 4 m/s. For Blackbird, we use Semi-Global Block Matching (SGBM) for stereo depth estimation [85].
Revised: The agent in the former two is an unmanned aerial vehicle (UAV), with speed of up to 4 m/s. For Blackbird, we use Semi-Global Block Matching (SGBM) for stereo depth estimation [85].

Original: For EuRoC, we use the ground-truth Leica MS50 depth readings provided by [10].
Revised: For EuRoC, we use the ground-truth Leica MS50 depth readings provided by [10].

Original: We pretend the
Revised: We pretend the

Original: IMU readings from these data sets are our control inputs.
Revised: IMU readings from these data sets are our control inputs.

Original: For TUM-RGBD we do not feed in any controls and assume a constant-velocity transition.
Revised: For TUM-RGBD we do not feed in any controls and assume a constant-velocity transition.

Original: All experimental details are in appendix G.
Revised: All experimental details are in appendix G.

Original: 5.1 Inference Through a Probabilistic Generative Model
Revised: 5.1 Inference Through a Probabilistic Generative Model

Original: First we look into the synergy between the inference and the generative assumptions.
Revised: First we look into the synergy between the inference and the generative assumptions.

Original: In ﬁg. 2a we see mapping and localisation examples.
Revised: In ﬁg. 2a we see mapping and localisation examples.

Original: The inferred scenes are consistent, with no dramatic offsets in geometry.
Revised: The inferred scenes are consistent, with no dramatic offsets in geometry.

Original: More importantly, rendering from the inferred map using the emission p ( x t | z t , M ) works as expected (see middle row), indicating that map updates are consistent with the generative assumptions.
Revised: More importantly, rendering from the inferred map using the emission p ( x t | z t , M ) works as expected (see middle row), indicating that map updates are consistent with the generative assumptions.

Original: This is evident from the accuracy of the inferred state trajectories as well (last row), as the pose optimisation objective from section 4.3.2 uses rendered images at every ﬁltering step.
Revised: This is evident from the accuracy of the inferred state trajectories as well (last row), as the pose optimisation objective from section 4.3.2 uses rendered images at every ﬁltering step.

Original: A potential discrepancy between the inference and the generative assumptions would lead to errors that would accumulate over time, which is not the case.
Revised: A potential discrepancy between the inference and the generative assumptions would lead to errors that would accumulate over time, which is not the case.

Original: [Figure]
Revised: [Figure]

Original: Map uncertainty The inferred map uncertainty is determined by the map updates.
Revised: Map uncertainty The inferred map uncertainty is determined by the map updates.

Original: show its interpretable effects in ﬁgs.
Revised: show its interpretable effects in ﬁgs.

Original: 2b and 2c for two examples, one from Blackbird and another from EuRoC.
Revised: 2b and 2c for two examples, one from Blackbird and another from EuRoC.

Original: Our map updates are akin to traditional SDF updates and the main factor that decides whether a map region is certain is how often it was observed.
Revised: Our map updates are akin to traditional SDF updates and the main factor that decides whether a map region is certain is how often it was observed.

Original: Regions that were occluded by objects, are behind walls or were rarely in view remain uncertain, e.g. as seen in ﬁg. 2c.
Revised: Regions that were occluded by objects, are behind walls or were rarely in view remain uncertain, e.g. as seen in ﬁg. 2c.

Original: In contrast, if the agent spends a lot of time looking at a certain map region, the uncertainty there decreases, as seen in ﬁg. 2b.
Revised: In contrast, if the agent spends a lot of time looking at a certain map region, the uncertainty there decreases, as seen in ﬁg. 2b.

Original: State uncertainty In ﬁg. 3 we analyse state uncertainty by looking at the variance for individual dimensions.
Revised: State uncertainty In ﬁg. 3 we analyse state uncertainty by looking at the variance for individual dimensions.

Original: We notice that state uncertainty changes along the trajectory.
Revised: We notice that state uncertainty changes along the trajectory.

Original: Uncertainty is determined by what the agent currently sees, based on the geometric relationship between the agent movement and the observed scene (e.g. ﬁg. 3a and ﬁg. 3b).
Revised: Uncertainty is determined by what the agent currently sees, based on the geometric relationship between the agent movement and the observed scene (e.g. ﬁg. 3a and ﬁg. 3b).

Original: This effect can be explained if we examine the Laplace approximation used to estimate pose covariances.
Revised: This effect can be explained if we examine the Laplace approximation used to estimate pose covariances.

Original: At any given time step, we set the covariance to Σ pose t ≈ − H − 1 ≈ − (cid:0) 2 J T J (cid:1) − 1 .
Revised: At any given time step, we set the covariance to Σ pose t ≈ − H − 1 ≈ − (cid:0) 2 J T J (cid:1) − 1 .

Original: Here H is the Hessian of the tracking objective at the mean pose estimate and J is the Jacobian.
Revised: Here H is the Hessian of the tracking objective at the mean pose estimate and J is the Jacobian.

Original: The Jacobian connects the pose to all image pixel errors.
Revised: The Jacobian connects the pose to all image pixel errors.

Original: The more consistent Jacobian entries are for a given pose dimension, the smaller the variance for that dimension will be.
Revised: The more consistent Jacobian entries are for a given pose dimension, the smaller the variance for that dimension will be.

Original: We refer
Revised: We refer

Original: 5.2 Localisation Accuracy
Revised: 5.2 Localisation Accuracy

Original: We compare PRISM’s localisation to state-of-the-art methods in moderately-sized indoor environments.
Revised: We compare PRISM’s localisation to state-of-the-art methods in moderately-sized indoor environments.

Original: We consider both baselines with dense maps (TANDEM [10], VSSM-LM [15], iMAP [11],
Revised: We consider both baselines with dense maps (TANDEM [10], VSSM-LM [15], iMAP [11],

Original: NICE-SLAM [9], CodeVIO [86]) and sparse methods without rendering (ORB-SLAM2 [4], VINS [71], VIMO [70]).
Revised: NICE-SLAM [9], CodeVIO [86]) and sparse methods without rendering (ORB-SLAM2 [4], VINS [71], VIMO [70]).

Original: The results are in table 1.
Revised: The results are in table 1.

Original: For the considered trajectories accuracy is comparable to the baselines, with differences of a few centimeters.
Revised: For the considered trajectories accuracy is comparable to the baselines, with differences of a few centimeters.

Original: At the same time, our inference boasts a predictive state-space model with both rendering and dynamics as well as uncertainty estimates, which is not common in the dense visual SLAM literature.
Revised: At the same time, our inference boasts a predictive state-space model with both rendering and dynamics as well as uncertainty estimates, which is not common in the dense visual SLAM literature.

Original: Finally, in ﬁg. 4 we see example inferred agent velocities, noting the uncertainty bands.
Revised: Finally, in ﬁg. 4 we see example inferred agent velocities, noting the uncertainty bands.

Original: This is possible because we model the agent dynamics.
Revised: This is possible because we model the agent dynamics.

Original: Our localisation accuracy on Blackbird is better than the off-line variational inference results of
Revised: Our localisation accuracy on Blackbird is better than the off-line variational inference results of

Original: VSSM-LM presented by Mirchev et al.
Revised: VSSM-LM presented by Mirchev et al.

Original: [15], and at the same time our solution runs in real-time and also captures uncertainty.
Revised: [15], and at the same time our solution runs in real-time and also captures uncertainty.

Original: This shows the advantages of the proposed divide-and-conquer ﬁltering.
Revised: This shows the advantages of the proposed divide-and-conquer ﬁltering.

Original: 5.3 Approximations for Runtime Improvement
Revised: 5.3 Approximations for Runtime Improvement

Original: All of our approximations are motivated by the real-time constraint, dictating the need for closed-form map updates, a Laplace approximation, linearisation assumptions and a surrogate pose optimisation objective.
Revised: All of our approximations are motivated by the real-time constraint, dictating the need for closed-form map updates, a Laplace approximation, linearisation assumptions and a surrogate pose optimisation objective.

Original: Figure 5 shows a runtime breakdown for different image resolutions, measured on an
Revised: Figure 5 shows a runtime breakdown for different image resolutions, measured on an

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: ofﬁce
Revised: ofﬁce

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: NVIDIA 

Revised: NVIDIA

Original: Ti GPU and an Intel(R) Xeon(R)
Revised: Ti GPU and an Intel(R) Xeon(R)

Original: W-2123 CPU at 3.6 GHz.
Revised: W-2123 CPU at 3.6 GHz.

Original: The heaviest operations are rendering and the gradient-based pose optimisation.
Revised: The heaviest operations are rendering and the gradient-based pose optimisation.

Original: Based on movement speed, rendering can happen periodically, whenever a new anchor image prediction for pose optimisation is needed.
Revised: Based on movement speed, rendering can happen periodically, whenever a new anchor image prediction for pose optimisation is needed.

Original: This leaves us with a total runtime of 10 Hz to 15 Hz, updating the map and state at every data step.
Revised: This leaves us with a total runtime of 10 Hz to 15 Hz, updating the map and state at every data step.

Original: In ﬁg. 6 we also compare closed-form map updates to map inference via gradient-descent (e
Revised: In ﬁg. 6 we also compare closed-form map updates to map inference via gradient-descent (e

Original: [Figure]
Revised: [Figure]

Original: g. as in [15, 8, 11, 9]).
Revised: g. as in [15, 8, 11, 9]).

Original: While gradient-descent is more accurate on a bigger budget, it is much more expensive.
Revised: While gradient-descent is more accurate on a bigger budget, it is much more expensive.

Original: For example, to match the accuracy of the closed-form updates, which take less than 10 ms, one would need ca.
Revised: For example, to match the accuracy of the closed-form updates, which take less than 10 ms, one would need ca.

Original: 250 ms of optimisation, which is impractical.
Revised: 250 ms of optimisation, which is impractical.

Original: These runtimes are for a voxel grid that is signiﬁcantly faster than neural representations [54], which would only exacerbate the problem.
Revised: These runtimes are for a voxel grid that is signiﬁcantly faster than neural representations [54], which would only exacerbate the problem.

Original:  Limitations and Conclusion
Revised:  Limitations and Conclusion

Original: SDF voxel grids allow for closed-form updates, but their memory footprint limits the maximum resolution and scene size.
Revised: SDF voxel grids allow for closed-form updates, but their memory footprint limits the maximum resolution and scene size.

Original: Voxel hashing [45] or octrees [87] can directly replace them for memory efﬁciency.
Revised: Voxel hashing [45] or octrees [87] can directly replace them for memory efﬁciency.

Original: Neural maps and dynamically changing maps have remained out of our scope.
Revised: Neural maps and dynamically changing maps have remained out of our scope.

Original: Their probabilistic formulation and closed-form updates require further investigation.
Revised: Their probabilistic formulation and closed-form updates require further investigation.

Original: Our map factorises over voxels with no inter-region correlation, which could also be improved.
Revised: Our map factorises over voxels with no inter-region correlation, which could also be improved.

Original: PRISM provides interpretable uncertainty in real-time, but estimation is approximate.
Revised: PRISM provides interpretable uncertainty in real-time, but estimation is approximate.

Original: Obtaining perfectly calibrated uncertainty on a budget remains an open question (see appendix H).
Revised: Obtaining perfectly calibrated uncertainty on a budget remains an open question (see appendix H).

Original: While ﬁltering works for our generative assumptions indoors, ﬁlters cannot revisit past errors and can drift in large scenes with high levels of exploration [6].
Revised: While ﬁltering works for our generative assumptions indoors, ﬁlters cannot revisit past errors and can drift in large scenes with high levels of exploration [6].

Original: We leave large-scale inference considerations for future work.
Revised: We leave large-scale inference considerations for future work.

Original: We have introduced PRISM, a method for probabilistic ﬁltering in a predeﬁned spatial state-space model.
Revised: We have introduced PRISM, a method for probabilistic ﬁltering in a predeﬁned spatial state-space model.

Original: Our solution runs in real-time, provides state and map uncertainty, and infers a dense map and a 6-DoF state trajectory with velocities.
Revised: Our solution runs in real-time, provides state and map uncertainty, and infers a dense map and a 6-DoF state trajectory with velocities.

Original: It is comparably accurate to state-of-the-art SLAM in indoor environments.
Revised: It is comparably accurate to state-of-the-art SLAM in indoor environments.

Original: To the best of our knowledge this is the ﬁrst real-time fully-probabilistic solution for
Revised: To the best of our knowledge this is the ﬁrst real-time fully-probabilistic solution for

Original: SLAM that combines differentiable rendering and agent dynamics.
Revised: SLAM that combines differentiable rendering and agent dynamics.

Original: We validated our method on three challenging data sets, featuring unmanned aerial vehicles and a handheld camera.
Revised: We validated our method on three challenging data sets, featuring unmanned aerial vehicles and a handheld camera.

Original: The results are promising, establishing PRISM as a viable state estimator for downstream model-based control.
Revised: The results are promising, establishing PRISM as a viable state estimator for downstream model-based control.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank our reviewers for the thoughtful discussion, it helped us to better position our contribution.
Revised: We thank our reviewers for the thoughtful discussion, it helped us to better position our contribution.