Original: PRISM: Probabilistic Real-Time Inference in Spatial  ModelsAnonymous Author(s)
Revised: PRISM: Probabilistic Real-Time Inference in Spatial World Models

Original: AfﬁliationAddress email Abstract: We introduce PRISM, a method for real-time ﬁltering in a probabilisticgenerative model of agent motion and visual perception.
Revised: Abstract: We introduce PRISM, a method for real-time ﬁltering in a probabilistic generative model of agent motion and visual perception.

Original: Previous approacheseither lack uncertainty estimates for the map and agent state, do not run in realtime, do not have a dense scene representation or do not model agent dynamics.
Revised: Previous approaches either lack uncertainty estimates for the map and agent state, do not run in real-time, do not have a dense scene representation or do not model agent dynamics.

Original: Our solution reconciles all of these aspects.
Revised: Our solution reconciles all of these aspects.

Original: We start from a predeﬁned worldmodel which combines differentiable rendering and 6-DoF dynamics.
Revised: We start from a predeﬁned state-space model which combines differentiable rendering and 6-DoF dynamics.

Original: Probabilisticinference in this model amounts to simultaneous localisation and mapping (SLAM)and is intractable.
Revised: Probabilistic inference in this model amounts to simultaneous localisation and mapping (SLAM) and is intractable.

Original: We use a series of approximations to Bayesian inference to arriveat probabilistic map and state estimates.
Revised: We use a series of approximations to Bayesian inference to arrive at probabilistic map and state estimates.

Original: We take advantage of well-establishedmethods and closed-form updates, preserving accuracy and enabling real-timecapability.
Revised: We take advantage of well-established methods and closed-form updates, preserving accuracy and enabling real-time capability.

Original: The proposed solution runs at 10Hz real-time and is competitive withstate-of-the-art SLAM on three challenging data sets, with  UAV and handheldcamera agents, Blackbird, EuRoC and TUM-RGBD.
Revised: The proposed solution runs at 10Hz real-time and is similarly accurate to state-of-the-art SLAM in small to medium-sized indoor environments, with highspeed UAV and handheld camera agents (Blackbird, EuRoC and TUM-RGBD).

Original:  Introduction
Revised:  Introduction

Original: Moving agents perceive streams oflocal sensory information, typically a combination of RGBimages, depth and inertial measurements.
Revised: Moving agents perceive streams of information, typically a mix of RGB images, depth and inertial measurements.

Original: Probabilistic generative models [1] are a principled wayof formalising the synthesis of this data, and from these models inference can be derived through Bayes’ rule.
Revised: Probabilistic generative models [1] are a principled way to formalise the synthesis of this data, and from these models inference can be derived through Bayes’ rule.

Original: We focus on exactly such inference, targeting the agent states and the map of the scene,a problem known as simultaneous localisation and mapping (SLAM).
Revised: We focus on exactly such inference and target the agent states and the scene map, a problem known as simultaneous localisation and mapping (SLAM).

Original: Our motivation for obtainingthe inference procedure asthe inversion of a well-deﬁned forward model is that their combinationlends itself to model-based control, where the former serves as a state estimator and the latter as asimulator with which to plan ahead [2].
Revised: We treat it as a posterior approximation for a given state-space model, such that the combination is useful for model-based control: the posterior inference serves as a state estimator and the predictive state-space model as a simulator with which to plan ahead [2].

Original: To pave the way towards decision making, we believe an inference  should have:
Revised: To pave the way towards decision making, we believe an inference method should have:

Original: • a compatible predictive model for both RGB-D images and 6-DoF dynamics; • principled state and map uncertainty; • real-time performance on commodity hardware; • state-of-the-art localisation accuracy.
Revised: • a compatible predictive model for both RGB-D images and 6-DoF dynamics; • principled state and map uncertainty; • real-time performance on commodity hardware; • state-of-the-art localisation accuracy.

Original:  Prominent methods like LSD-SLAM [3], ORB-SLAM [4], DSO [5] have propelled the SLAMﬁeld forward, and for years focus has gravitated towards large-scale localisation accuracy.
Revised: We motivate these requirements further in appendix J. Prominent methods like LSD-SLAM [3], ORB-SLAM [4], DSO [5] have propelled visual SLAM forward, with heavy focus on large-scale localisation.

Revised: The core of modern large-scale SLAM is maximum a-posteriori (MAP) smoothing in a probabilistic factor graph [6, 7].

Original: The implication is that map and state inference has become an aim in itself, usually disconnected from predictivemodelling and control considerations.
Revised: At present this demands sparsity assumptions for computational feasibility, which obstructs the tight integration of dense maps and rendering.

Original: With the rise in popularity of implicit scene models like NERF [6], some approaches have shifted towards inference through predictive image synthesis (e.g.[7, 8, 9]), but dynamics modelling and uncertaintyestimation have remained out oftheir scope.
Revised: Nonetheless, for smaller scenes the recent popularity of neural models (e.g. NERF [8]) has sparked interest in inference through a renderer (e.g. [9, 10, 11]), but dynamics modelling and uncertainty have remained out of scope.

Original: On the ﬂip side, classical probabilistic methods often feature dynamics and run in real-time, but are limited to ﬂat 2D range observations, image features or landmarks, and lack rendering (e.g. [10, 11])
Revised: Conversely, classical ﬁltering comes with dynamics and uncertainty in real-time (e.g. [12, 13, 14]), but over time has given way to large-scale smoothing [6] and to our knowledge has not been well explored for the integration of dense differentiable rendering and dynamics on a moderate scale. Overall, we ﬁnd there is a need for a cohesive inference solution that satisﬁes our requirements.

Original: We  contribute by meeting all the above goals, emphasising the link to a predictive model (ﬁg. 1).
Revised: We thus contribute by meeting all the above goals, emphasising the link to a predictive model (ﬁg. 1).

Original: We start from the generative model of Mirchev et al.
Revised: We start from the generative model of Mirchev et al.

Original: [12], who combine differentiable renderingand agent dynamics in a probabilistic framework.
Revised: [15], who combine differentiable rendering and agent dynamics in a probabilistic framework.

Original: The authors considered stochastic variationalinference for this model, applying it off-line with runtime orders of magnitude too long for on-line use.
Revised: The authors considered stochastic variational inference for this model, applying it off-line with runtime orders of magnitude too long for on-line use.

Original: We pursue an alternative route for real-time inference: from the generative assumptions we deriveapproximations to the true marginal ﬁlters over the last state and map [13].
Revised: We pursue an alternative route for real-time inference: from the generative assumptions we derive approximations to the true marginal ﬁlters over the last state and map [16].

Original: By focusing on recursiveﬁltering updates, we identify where established probabilistic inference and computer vision techniquescan be used, putting emphasis on fast closed-form updates.
Revised: By focusing on recursive ﬁltering updates, we identify where established probabilistic inference and computer vision techniques can be used, putting emphasis on fast closed-form updates.

Original: We ﬁnd this divide-and-conquer strategyis a good compromise for achieving the aforementioned objectives under computational constraints.
Revised: We ﬁnd this divide-and-conquer strategy is a good compromise for achieving the aforementioned objectives under computational constraints.

Original: We evaluate the proposed solution on two unmanned aerial vehicle (UAV) data sets [14, 15] andon TUM-RGBD [16].
Revised: We evaluate the proposed solution on two unmanned aerial vehicle (UAV) data sets [17, 18] and on TUM-RGBD [19].

Original: In these setups, our method PRISM runs at 10 Hz real-time with similarlocalisation accuracy to state-of-the-art SLAM.
Revised: Our method PRISM runs at 10 Hz real-time with similar localisation accuracy to state-of-the-art SLAM in moderately-sized indoor environments.

Original: It provides uncertainty estimates during inferenceand features a predictive distribution that can both render images and forecast the agent’s movement.
Revised: It provides uncertainty estimates and features a predictive distribution that can both render images and forecast the agent’s movement.

Original:  Related Work
Revised:  Related Work

Original: Generative models Generative state-space models simulate the formation of observed data overtime in a Markov chain [1, 17, 18, 19, 20, 21].
Revised: Generative models Generative state-space models simulate the formation of observed data over time in a Markov chain [1, 12, 20, 21, 22, 23], serving as world models [24, 25].

Original: They specify models for the agent dynamics andemission (state-to-observation mapping), with which we can plan ahead by imagining future rollouts[2, 22, 23, 24, 25, 26, 27, 28, 29].
Revised: With their agent dynamics and state-to-observation emission models we can imagine future rollouts for planning [2, 26, 27, 28, 29, 30, 31, 32, 33].

Original: We abide by this framework and design a posterior inference fora spatial world model with the aim to enable on-line control downstream.
Revised: We abide by this framework and design a posterior inference for a spatial state-space model, to enable on-line control.

Original: There are a number ofrecently proposed generative spatial models (e.g. [30, 31, 32, 33, 34, 35, 36]). We have selected themodel of Mirchev et al.[12] to tailor our inference to.
Revised: Among such models (e.g. [34, 35, 36, 37, 38, 39, 40]), we tailor our inference to the model of Mirchev et al.

Original: It has been shown to reliably scale to 3D,featuring a dense map representation, a renderingemission and 6-DoFagent dynamics.
Revised: It scales to 3D with rendering and 6-DoF dynamics.

Original: We contributeby designing a real-time inferencefor this model that respects its probabilistic formulation.
Revised: We contribute a real-time inference that ﬁts its probabilistic formulation.

Original: SLAM through image synthesis
Revised: SLAM through image synthesis

Original: The assumed generative model renders RGB-D images, whichrelates our work to SLAM through image synthesis.
Revised: The assumed generative model renders RGB-D images, which is related to SLAM through full-image synthesis.

Original: Traditional methods feature varied maps, fromvolumetric to surfels (e.g. [37, 38, 39, 40, 41, 42, 43, 44, 45]), and commonly estimate new cameraposes by aligning new observations to a rendered image with variants of point-to-plane ICP withphotometric consistency [46, 47, 48, 49, 40].
Revised: Traditional methods feature varied maps, from volumetric to surfels (e.g. [41, 42, 43, 44, 45, 46, 47, 48, 49]), and commonly estimate new camera poses by aligning new observations to a rendered image with variants of point-to-plane ICP with photometric consistency [50, 51, 52, 53, 44].

Original: We extend this optimisation with dynamics in ourapproximate state ﬁlter [50].
Revised: We extend this optimisation with dynamics in our approximate state ﬁlter [54].

Original: A recent trend is to use implicit scene representations like NERF(e.g. [6, 51, 52]) with high rendering ﬁdelity.
Revised: A recent trend is to use implicit scene representations like NERF (e.g. [8, 55, 56]) with high rendering ﬁdelity.

Original: Gradient-based pose inference through NERF-likerendering has received attention [53, 54], with iMAP [9] and NICE-SLAM [7] being two real-timesolutions.
Revised: Gradient-based pose inference through NERF-like rendering has received attention [57, 58], with iMAP [11] and NICE-SLAM [9] being two real-time solutions.

Original: The mapping runtime of such methods is weighed down by optimisation through therenderer.
Revised: The mapping runtime of such methods is weighed down by optimisation through the renderer.

Original: Rendering can be sped up by decomposing parameters over space, e.g. by using voxelsor primitives [55, 56, 57, 58, 59], but how to update neural maps in closed form remains unclear.
Revised: Rendering can be sped up by decomposing parameters over space, e.g. by using voxels or primitives [59, 60, 61, 62, 63], but how to update neural maps in closed form remains unclear.

Original: Therefore, we rely on vanilla voxel grid maps [12, 60], as their probabilistic treatment and closed-formupdates are straightforward, leaving implicit representations for future work.
Revised: Therefore, we rely on vanilla voxel grid maps [15, 64], as their probabilistic treatment and closed-form updates are straightforward, leaving implicit representations for future work.

Original: We note that none of theaforementioned methods incorporate dynamics and uncertainty, which distinguishes our approach.
Revised: We note that none of the aforementioned methods incorporate dynamics and uncertainty, which distinguishes our approach.

Original: Probabilistic SLAM inferenceWe approach visual dense SLAM with approximate probabilisticﬁltering. Probabilistic SLAM is a traditional concept [61, 17, 62, 10, 63, 64, 65], but historicallyit has remained prevalent for ﬂat 2D modelling, and modern visual SLAM has drifted away fromthe probabilistic paradigm.
Revised: Probabilistic SLAM inference SLAM ﬁlters are thoroughly explored for ﬂat 2D modelling [65, 12, 66, 13, 67, 68, 69], but have been superseded by MAP smoothing in modern visual SLAM (e.g. [3, 4, 5, 70, 71, 72, 73]), primarily due to scalability concerns [6, 74]. However, as of now smoothing is not computationally feasible without sparsity assumptions.

Original: Our motivation is to reconcile both ﬁelds, as predictive uncertainty isimportant for control (e.g. collision avoidance) and vision is important for scene understanding–tworequirements for spatial decision-making.
Revised: We therefore reexamine ﬁltering for differentiable rendering, as we aim to obtain a dense map posterior with uncertainty in real-time (see appendix J for further motivation). Filters may beneﬁt from the dense modelling of observations [74], which aligns with our objective, and we will demonstrate they can be a feasible solution for moderately-sized indoor environments.

Original: For the states, we use  updates similar to those in extended Kalman ﬁlters [17] and a Laplace approximation [66].
Revised: For the states, we use a Laplace approximation [75] and velocity updates similar to those in extended Kalman ﬁlters [12].

Original: For the maps, occupancy grids would be anatural probabilistic choice [60, 62, 67], and closed-form mapping has been used in that context [65].
Revised: For the map, occupancy grids are a common probabilistic choice [64, 66, 76] and closed-form mapping has been used in that context [69].

Original: To enable rendering, we provide a similar derivation, but for a signed distance function (SDF), whichis closely related.
Revised: To enable rendering we provide a similar derivation, but for a signed distance function (SDF), which is related.

Original: Probabilistic SDF mapping dates back to Curless and Levoy [37], and SDF updateshave a well-known probabilistic interpretation [68, 69].
Revised: Probabilistic SDF mapping dates back to Curless and Levoy [41], and SDF updates have a well-known probabilistic interpretation [77, 78].

Original: We introduce all of these approximations toarrive at a holistic probabilistic solution that scales to  3D  in real-time.
Revised: We use these approximations to arrive at a holistic probabilistic solution that scales to dense 3D modelling in real-time.

Original:  Overview
Revised:  Overview

Original: We approach on-line SLAM inference with two aims in mind.
Revised: We approach on-line SLAM inference with two aims in mind.

Original: First, we want to harmonise our mapand state estimation with a predictive model.
Revised: First, we want to harmonise our map and state estimation with a predictive model.

Original: Second, we want to quantify uncertainty: estimates andpredictions should account for modelling inaccuracies as well as measurement and process noise.
Revised: Second, we want to quantify uncertainty: estimates and predictions should account for modelling inaccuracies as well as measurement and process noise.

Original: Bothaspects are important for autonomous decision making.
Revised: Both are important for autonomous decision making.

Original: To achieve this, we derive our solutionas Bayesian inference in the probabilistic model proposed by Mirchev et al.
Revised: To achieve this, we derive a Bayesian posterior in the probabilistic model of Mirchev et al.

Original: [12], by which we ensureinference matches the forward model.
Revised: [15], to ensure that inference matches the forward model.

Original: Before we delve into our proposed solution, we present apractical summary.
Revised: Before we delve into our proposed solution, we present a practical summary.

Original: At every time step:
Revised: At every time step:

Original: we point-estimate the agent’s pose using gradient descent, involving geometry and dynamics.2.
Revised: we point-estimate the agent’s pose using gradient descent, involving geometry and dynamics

Original: we extend the pose with a Gaussian covariance matrix through a Laplace approximation.
Revised: we extend the pose with a Gaussian covariance matrix through a Laplace approximation.

Original: with the pose, we estimate the agent’s current velocity in closed form.4.
Revised: with the pose, we estimate the agent’s current velocity in closed form.

Original: with the pose and the current observation, we update the map in closed form.
Revised: with the pose and the current observation, we update the map in closed form.

Original: In practice, we implement the above as a combination of well-established methods.
Revised: We use well-established methods for the above.

Original: The objective in1. is inspired by assumed density ﬁltering [70], point-to-plane ICP [46] and photometric reprojection[47, 48].
Revised: In 1. we combine assumed density ﬁltering [79], point-to-plane ICP [50] and photometric alignment [51, 52].

Original: For 2.
Revised: In 2.

Original: we use a Laplace approximation [66, 71].
Revised: we use a Laplace approximation [75, 80].

Original: For 3.
Revised: In 3.

Original: we use linear-Gaussian updates,similar tothe ones used in Kalman ﬁlters [17].
Revised: we use linear-Gaussian updates, akin to Kalman ﬁlters [12].

Original: For 4.
Revised: In 4.

Original: we ﬁrst derive generic closed-form map updates,which then end up nearly equivalent totraditional SDF updates [37] for our generative assumptions.
Revised: we ﬁrst derive generic closed-form map updates, which boil down to SDF updates [41] for our generative assumptions.

Original: We contribute by deriving a holistic Bayesian inference from the generative requirements we posited.
Revised: We contribute by deriving a holistic Bayesian inference from the generative model we started with.

Original: In doing so, we identify where traditional techniques are applicable to make a practical algorithm.
Revised: In doing so, we identify where traditional techniques are applicable to make a practical algorithm.

Original:  Methods
Revised:  Methods

Original: In the following we will denote generative distributions, true posterior distributions and conditionalswith p ( · ) .
Revised: In the following we will denote generative distributions, true posterior distributions and conditionals with p ( · ) .

Original: Respectively, approximate distributions will be denoted with q ( · ) .
Revised: Respectively, approximate distributions will be denoted with q ( · ) .

Original: Approximation stepswill be indicated by ≈ in equations.
Revised: Approximation steps will be indicated by ≈ in equations.

Original: We use q φ ( · ) to subsume estimated distribution parameters into108 φ .
Revised: We use q φ ( · ) to subsume estimated distribution parameters into φ .

Original: A subscript · t indicates that a variable or a distribution is different at every time step.
Revised: A subscript · t indicates that a variable or a distribution is different at every time step.

Original: 4.1 Background
Revised: 4.1 Background

Original: We start with an overview of the generative model of Mirchev et al.
Revised: We start with an overview of the generative model of Mirchev et al.

Original: [12] from which we will derivethe inference.
Revised: [15] from which we will derive the inference.

Original: We assume a sequence of RGB-D observations x 1: T and a sequence of agent states112 z 1: T driven by controls u 1: T − 1 form a Markovian state-space model.
Revised: We assume a sequence of RGB-D observations x 1: T and a sequence of agent states z 1: T driven by controls u 1: T − 1 form a Markovian state-space model.

Original: Each observation is constructedfrom a respective state with a rendering emission model p ( x t | M , z t ) , where M is a global latentrandom variable for a dense map.
Revised: Each observation is constructed from a respective state with a rendering emission model p ( x t | M , z t ) , where M is a global latent random variable for a dense map.

Original: A transition model p ( z t | z t − 1 , u t − 1 ) accounts for the agentdynamics, where u t are known acceleration controls.
Revised: A transition model p ( z t | z t − 1 , u t − 1 ) accounts for the agent dynamics, where u t are known acceleration controls.

Original: Assuming z 1 is given, the joint distribution is:p ( M , z 2: T , x 1: T | u 1: T − 1 , z 1 ) = p ( M ) p ( x 1 | M , z 1 )
Revised: Assuming z 1 is given, the joint distribution is: p [Equation] , z [Equation]

Original:  T 
Revised: : T , x [Equation]

Original: [Equation]
Revised: [Equation]

Original: The map is a 3D voxel grid of occupancy and color.
Revised: The map is a 3D voxel grid of occupancy and color–each cell contains four values.

Original: The emission is fully-differentiable and performsvolumetric raymarching, searching for a unique hit position at a surface along each ray [72].
Revised: The emission is fully-differentiable and performs volumetric raymarching, searching for a unique hit position at a surface along each ray [81].

Original: Thetransition performs Euler integration, using the acceleration controls and maintained velocity fromthe latent state.
Revised: The transition performs Euler integration, using the acceleration controls and maintained velocity from the latent state.

Original: Appendix B and the original paper have the details.
Revised: Appendix B and the original paper have the details.

Original: 4.2 Posterior Choice
Revised: 4.2 Posterior Choice

Original: Our ﬁrst consideration is the choice of which posterior to approximate.
Revised: First we need to choose which posterior to approximate.

Original: For example, Mirchevet al.
Revised: For example, Mirchev et al.

Original: [12] approximate the full posterior over  map and all states p ( M , z 2: T | x 1: T , u 1: T − 1 , z 1 ) withvariational inference using the evidence lower bound [73].
Revised: [15] approximate the full posterior over the map and all states p ( M , z 2: T | x 1: T , u 1: T − 1 , z 1 ) with variational inference [82].

Original: While generic, this approach’s runtime isweighed down by the cost of propagating gradients through the renderer at every optimisation step[50], and the inevitable stochastic optimisation demands multiple steps until convergence.
Revised: While generic, this approach is slowed down by rendering at every optimisation step [54], and the inevitable stochastic optimisation demands multiple steps until convergence.

Original: In addition,estimating the posterior over all states scatters the optimisation budget across the whole trajectory.
Revised: In addition, estimating the posterior over all states scatters the optimisation budget across the whole trajectory.

Original: To enableinference in real-time  we thus focus on an alternative posterior, the ﬁlterp ( M , z t | x 1: t , u 1: t − 1 , z 1 ) , asonly the belief over the last state  is needed for planning ahead [2].
Revised: To enable real-time inference we target an alternative posterior, the ﬁlter p ( M , z t | x 1: t , u 1: t − 1 , z 1 ) , as the last state belief is enough for planning ahead [2].

Original: Since ﬁlters can be updated recursively using Bayes’ rule [13, 71], we canconcentrate our computational budget on this smaller problem and make useof closed-form updates for fast inference.
Revised: Since ﬁlters can be updated recursively [16, 80], we can use closed-form updates for fast inference.

Original: A main issue is how to maintain the joint distributionof map and state. This is due to our dense 3Dmap, as the large number of parameters in Mmakes full-covariance Gaussian or Rao-Blackwellisedrepresentations [10, 11] infeasible 1 . {+
+}
Revised: Still, maintaining the joint distribution is too costly because of the large dense 3D map M . {+
+}

Original: Thus we approximate the two marginal ﬁlters instead:
Revised: Instead, we approximate the two marginal ﬁlters:

Original: [Equation]
Revised: [Equation]

Original: where H t = x 1: t , u 1: t − 1 , z 1 .
Revised: where H t = x 1: t , u 1: t − 1 , z 1 .

Original: We note that this does not mean we approximate the joint with theproduct of the two marginals, i.e. we do not assume a mean-ﬁeld approximation [71].

Original: Rather, wechoose not to explicitly approximate the joint distribution p ( z t , M | x 1: t , u 1: t − 1 , z 1 ) , and leave itimplicitly deﬁned by the two marginals.

Original: More details on this aspect can be found in appendix A. Wedraw attention to the shorthand notation p ( · | H t ) , which will appear again in the following.
Revised: More details about this modelling choice can be found in appendix A. We draw attention to the shorthand notation p ( · | H t ) , which will appear again in the following.

Original: 4.3 Approximate Filtering
Revised: 4.3 Approximate Filtering

Original: For both marginal ﬁlters, we will arrive at adequate approximations by reusing the following equation:
Revised: For both marginal ﬁlters [Equation] we will arrive at adequate approximations by reusing the following equation:

Original: [Equation]
Revised: [Equation]

Original: This is a classic recursive expression of the Bayes ﬁlter [13].
Revised: This is a classic recursive expression of the Bayes ﬁlter [16].

Original: Starting from each true marginalposterior, we will ﬁrst expand the joint, then use eq.
Revised: Starting from each true marginal posterior, we will ﬁrst expand the joint, then use eq.

Original: (1) andthen apply a set of approximations toobtain a practical solution.
Revised: (1) and apply a set of approximations.

Original: In the following we  discuss our ﬁnal result,and we defer the detailedderivation of bothmarginal ﬁlters to appendices C and E.
Revised: Next we will discuss our ﬁnal result, we defer the detailed derivation of both ﬁlters to appendices C and E.

Original: 4.3.1 Marginal Map Filter
Revised: 4.3.1 Marginal Map Filter

Original: We begin with the map approximation, starting from the true marginal Bayes ﬁlter:
Revised: We begin with the map approximation, starting from the true marginal Bayes ﬁlter:

Original: [Equation]
Revised: [Equation]

Original: Equations (2) and (3) hide a number of approximations, detailed in appendix C.
Revised: Equations (2) and (3) hide a few approximations detailed in appendix C.

Original: The resulting solutiontakes a nominalcurrent state sample ˆ z t , with which a map update q ( M | x t , ˆ z t ) is applied to theprevious map belief q φ t − 1 ( M ) .
Revised: The resulting solution takes a nominal state sample ˆ z t , with which a map update q ( M | x t , ˆ z t ) is applied to the previous map belief q φ t − 1 ( M ) .

Original: We set ˆ z t to the mean of the currentapproximate state ﬁlter q φ t ( z t ) .
Revised: We set ˆ z t to the mean of the current state belief q φ t ( z t ) .

Original: Accepting some bias, we do this for speed, as this is our best guess for z t without extra computation.
Revised: Accepting some bias, we do this for speed as it is our best guess for z t without extra computation.

Original: Intuitively, the map update q ( M | x t , ˆ z t ) approximately inverts the emission p ( x t | ˆ z t , M ) andpopulates the map with the content necessary for reconstructing the observation x t .
Revised: Intuitively, the map update q ( M | x t , ˆ z t ) populates the map such that the observation x t can be reconstructed.

Original: Our derivation ofthe is similar to the one  Grisetti  al.
Revised: Our derivation of the updates is similar to the one by Grisetti et al.

Original: [ for 2D occupancy maps, but now applied to 3D.
Revised: [69] for 2D occupancy maps, but now applied to 3D.

Original: Note that the above approximation is generic, agnostic to the speciﬁc parameterisation of the mapor the rendering equations.
Revised: The above approximation is generic, agnostic to the speciﬁc map and rendering assumptions.

Original: In practice, we need to formulate the map update q ( M | x t , ˆ z t )inclosed-form, such thatit is faithful to the emission p ( x t | ˆ z t , M ) .
Revised: In practice, we need a closed-form map update q ( M | x t , ˆ z t ) that is faithful to the emission p ( x t | ˆ z t , M ) .

Original: In this work, we focus on the mapparameterisation of Mirchev et al.
Revised: In this work, we follow Mirchev et al.

Original: [12], a Gaussian  that factorises overoccupancy and color voxels:
Revised: [15] and use a Gaussian map that factorises over voxels:

Original: [Equation]
Revised: [Equation]

Original: Here the indices ijk run over voxels in a 3D grid.
Revised: Here the indices ijk run over voxels in a 3D grid.

Original: For this speciﬁc representation and the assumedsurface-based rendering, we identify that the map update q ( M | x t , ˆ z t ) can be implemented asa probabilistic signed distance function (SDF) update [37].
Revised: For this speciﬁc representation and the assumed surface-based rendering, we identify that the map update q ( M | x t , ˆ z t ) can be implemented as a probabilistic signed distance function (SDF) update [41].

Original: We provide the technical details inappendix D. SDF updates for voxel maps are a traditional concept in computer vision, and prior workhas considered their probabilistic interpretation before [68, 69].
Revised: We provide the technical details in appendix D. SDF updates for voxel maps are a traditional concept in computer vision, and prior work has considered their probabilistic interpretation before [77, 78].

Original: We contribute by identifying theplace of such updates in a probabilistic ﬁlter that follows the generative model of [12].
Revised: We contribute by identifying the place of such updates in a probabilistic ﬁlter that follows the generative model of [15].

Original: A detaileddiscussion of how the above relates to classical SDF update equations can be found in appendix D.
Revised: A detailed discussion of how the above relates to classical SDF update equations can be found in appendix D.

Original: All of the above approximations are motivated by the real-time constraint.
Revised: The above approximations are motivated by the real-time constraint.

Original: For example, one couldconsider optimising eq.
Revised: For example, one could optimise eq.

Original: (2) directly with gradient descent through the renderer, but evaluating theemission is expensive, which hinders accurate convergence on a budget.
Revised: (2) directly with gradient descent through the renderer, but evaluating the emission is expensive and hinders accurate convergence on a budget.

Original: This is particularly truewhen uncertainty estimates are desirable, as optimisation would then be stochastic and gradientsnoisy [74].
Revised: This is particularly true when uncertainty estimates are desirable, as optimisation would then be stochastic and gradients noisy [83].

Original: In contrast, the derived  map updates are meant to be applied in a one-shot manner, withcost similar to emitting just once, while capturing uncertainty as well.
Revised: In contrast, the derived one-shot map updates are meant to have a cost similar to emitting just once, while capturing uncertainty as well.

Original: We experimentally quantifythe runtime and approximation differences between the two approaches in section 5.4.
Revised: We show some of the differences between the two approaches in section 5.3.

Original: 4.3.2 Marginal State Filter
Revised: 4.3.2 Marginal State Filter

Original: For the state ﬁlter, weproceed similarly. We start out with the true marginalBayes ﬁlter and througheq.
Revised: Similarly, for the state ﬁlter we start from the true marginal and arrive at approximations via eq.

Original: (1) we gradually arrive at a combination of well-established methods:

Original: [Equation]
Revised: [Equation]

Original: As with the map ﬁlter, we detail all the approximations behind eq.
Revised: We detail all the approximations that lead to eq.

Original: (4) in appendix E. 
Revised: (4) in appendix E. In eq.

Original: To understandhow state beliefs are formed, we need to explain the step from eq.

Original: (4)to eq. In eq. (4) we havethree terms: an image reconstruction likelihood, a Gaussian pose prior and a linear Gaussian velocityconditional given a pose.
Revised: (4) we have three terms: an image reconstruction likelihood, a Gaussian pose prior and a linear Gaussian velocity conditional given a pose.

Original: The latter two we obtain analytically with a linear approximation of thetransition model and the previous Gaussian belief q φ t − 1 ( z t − 1 ) (c.f. appendix E).
Revised: The latter two we obtain analytically with a linear approximation of the transition model and the previous Gaussian belief q φ t − 1 ( z t − 1 ) (c.f. appendix E).

Original: First, using the ﬁrsttwo terms of eq.
Revised: First, using the ﬁrst two terms of eq.

Original: (4) we deﬁne a maximum a-posteriori (MAP) objective for pose optimisation:
Revised: (4) we deﬁne a maximum a-posteriori (MAP) objective for pose optimisation:

Original: [Equation]
Revised: [Equation]

Original: Here, ˆ M is a nominal map sample set to the mean of the previous map belief q φ t − 1 ( M ) .
Revised: Here, ˆ M is a nominal map sample set to the mean of the previous map belief q φ t − 1 ( M ) .

Original: Theterm log q t ( z pose t | u t − 1 , H t − 1 ) is an approximate dynamics prior over the current pose, it makesthe pose respect the transition model.
Revised: The term log q t ( z pose t | u t − 1 , H t − 1 ) is an approximate dynamics prior over the current pose, it makes the pose respect the transition model.

Original: The term log p ( x t | ˆ M , z pose t ) represents reconstructing thecurrent observation, optimising it for the current pose will align the observation to the map.
Revised: The term log p ( x t | ˆ M , z pose t ) represents reconstructing the current observation, optimising it for the current pose will align the observation to the map.

Original: However,evaluating this rendering term in every gradient step is inefﬁcient.
Revised: However, evaluating this rendering term in every gradient step is inefﬁcient.

Original: Because of this, we replace it withthe prediction-to-observation objective used by Kayalibay et al.
Revised: Because of this, we replace it with the prediction-to-observation objective used by Kayalibay et al.

Original: [50], Nießner et al.
Revised: [54], Nießner et al.

Original: [41], Newcombeet al.
Revised: [45], Newcombe et al.

Original: We refer to [50] for a discussion of why this surrogate is meaningful and we list thetechnical details in appendix E.
Revised: We refer to [54] for further motivation and we list the technical details in appendix E.

Original: The above optimisation gives us a MAP pose estimate, which we denote with µ pose t .
Revised: The above optimisation gives us a MAP pose estimate, which we denote with µ pose t .

Original: Next, we applya Laplace approximation [66] around it to obtain a full covariance matrix Σ pose t which captures thecurvature of the objective.
Revised: Next, we apply a Laplace approximation [75] around it to obtain a full covariance matrix Σ pose t which captures the curvature of the objective.

Original: This leaves us with a full Gaussian belief over the current pose:
Revised: This leaves us with a full Gaussian belief over the current pose:

Original: [Equation]
Revised: [Equation]

Revised: Appendix discusses further.

Original: [Figure]
Revised: [Figure]

Original: Finally, we can combine this
Revised: can combine this

Original: Gaussian with the
Revised: Gaussian with the

Original: Gaussian velocity conditional 
Revised: Gaussian velocity conditional q t

Original: [Equation]
Revised: [Equation]

Original: , ut
Revised: , u

Original: [Equation]
Revised: [Equation]

Original: , H
Revised: , H

Original: [Equation]
Revised: [Equation]

Original: ) (the third term in eq.
Revised: ) (the third term in eq.

Original: (5)) into a full-state belief in closed form:
Revised: (4)) into a full-state belief in closed form:

Original: [Equation]
Revised: [Equation]

Original: Thislast step is approximate, asthe velocity conditional q t ( z vel t | z pose t , u t − 1 , H t − 1 ) has not beeninformed of the latest observation x t , we do it for speed andin practice we ﬁnd it does not harmlocalisation.
Revised: This is approximate, as q t ( z vel t | z pose t , u t − 1 , H t − 1 ) is not informed of the latest observation x t . We do it for speed and ﬁnd it does not harm localisation in practice.

Original: We refer to appendix E for the details.
Revised: Appendix E has the details.

Original:  Experiments
Revised:  Experiments

Original: When we deﬁned the proposed inference, we set out with a few goals in mind: it should be faithfulto the generative assumptions, it should quantify uncertainty and it should run in real-time.
Revised: Originally we set out with a few goals: the inference method should be faithful to the generative assumptions, it should quantify uncertainty and it should run in real-time.

Original: Whatfollows is an empirical analysis of these aspects.
Revised: What follows is an empirical analysis of these aspects.

Original: We evaluate on the EuRoC [14], Blackbird [15] and TUM-RGBD [16] data sets.
Revised: We evaluate on the EuRoC [17], Blackbird [18] and TUM-RGBD [19] data sets.

Original: The agent in the former two is an unmanned aerial vehicle (UAV), withspeed of up to 4 m/s. For Blackbird, we use Semi-Global Block Matching (SGBM) for stereo depthestimation [76].
Revised: The agent in the former two is an unmanned aerial vehicle (UAV), with speed of up to 4 m/s. For Blackbird, we use Semi-Global Block Matching (SGBM) for stereo depth estimation [85].

Original: For EuRoC, we use the ground-truth Leica MS50 depth readings provided by [8],and since they are sparse we downscale them to a resolution of 60 × 80 pixels to densify.
Revised: For EuRoC, we use the ground-truth Leica MS50 depth readings provided by [10]. We pretend the

Original: We usethe IMU readings from these data sets as our control inputs.
Revised: IMU readings from these data sets are our control inputs.

Original: For TUM-RGBD we do not feed in anycontrols and assume a constant-velocity transition.
Revised: For TUM-RGBD we do not feed in any controls and assume a constant-velocity transition.

Original: All experimental details are in appendix G.
Revised: All experimental details are in appendix G.

Original: 5.1 Inference Through a  Generative Model
Revised: 5.1 Inference Through a Probabilistic Generative Model

Original: First we look into the synergy between the inference and the generative assumptions.
Revised: First we look into the synergy between the inference and the generative assumptions.

Original: In ﬁg. 2a wesee mapping and localisation examples.
Revised: In ﬁg. 2a we see mapping and localisation examples.

Original: The inferred scenes are consistent, with no dramatic offsetsin geometry.
Revised: The inferred scenes are consistent, with no dramatic offsets in geometry.

Original: More importantly, rendering from the inferred map using the emission p ( x t | z t , M )works as expected (see middle row), indicating that map updates are consistent with the generativeassumptions.
Revised: More importantly, rendering from the inferred map using the emission p ( x t | z t , M ) works as expected (see middle row), indicating that map updates are consistent with the generative assumptions.

Original: This is evident from the accuracy of the inferred state trajectories as well (last row), asthe pose optimisation objective from section 4.3.2 uses rendered images at every ﬁltering step.
Revised: This is evident from the accuracy of the inferred state trajectories as well (last row), as the pose optimisation objective from section 4.3.2 uses rendered images at every ﬁltering step.

Original: Apotential discrepancy between the inference and the generative assumptions would lead to errors thatwould accumulate over time, which is not the case.
Revised: A potential discrepancy between the inference and the generative assumptions would lead to errors that would accumulate over time, which is not the case.

Original: [Figure]
Revised: [Figure]

Original: 5.2 Uncertainty Analysis

Original: Map uncertainty The inferred map uncertainty is determined by the map updates.
Revised: Map uncertainty The inferred map uncertainty is determined by the map updates.

Original: We show itsinterpretable effects in ﬁgs.
Revised: show its interpretable effects in ﬁgs.

Original: 2b and 2c for two examples, one from Blackbird and another from EuRoC.
Revised: 2b and 2c for two examples, one from Blackbird and another from EuRoC.

Original: Our map updates are akin to traditional SDF updates and the main factor that decides whether a mapregion is certain is how often it was observed.
Revised: Our map updates are akin to traditional SDF updates and the main factor that decides whether a map region is certain is how often it was observed.

Original: Regions that were occluded by objects, are behindwalls or were rarely in view remain uncertain, e.g. as seen in ﬁg. 2c.
Revised: Regions that were occluded by objects, are behind walls or were rarely in view remain uncertain, e.g. as seen in ﬁg. 2c.

Original: In contrast, if the agent spends alot of time looking at a certain map region, the uncertainty there decreases, as seen in ﬁg. 2b.
Revised: In contrast, if the agent spends a lot of time looking at a certain map region, the uncertainty there decreases, as seen in ﬁg. 2b.

Original: State uncertainty In ﬁg. 3 we analyse state uncertainty by looking at the variance for individualdimensions.
Revised: State uncertainty In ﬁg. 3 we analyse state uncertainty by looking at the variance for individual dimensions.

Original: We notice that state uncertainty changes along the trajectory.
Revised: We notice that state uncertainty changes along the trajectory.

Original: Uncertainty is determinedby what the agent currently sees, based on the geometric relationship between the agent movement andthe observed scene (e.g. ﬁg. 3a and ﬁg. 3b).
Revised: Uncertainty is determined by what the agent currently sees, based on the geometric relationship between the agent movement and the observed scene (e.g. ﬁg. 3a and ﬁg. 3b).

Original: This effect can be explained if we examine the Laplaceapproximation used to estimate pose covariances.
Revised: This effect can be explained if we examine the Laplace approximation used to estimate pose covariances.

Original: At any given time step, we set the covariance to: [Equation]
Revised: At any given time step, we set the covariance to Σ pose t ≈ − H − 1 ≈ − (cid:0) 2 J T J (cid:1) − 1 .

Original: Here H is the Hessian of the tracking objective at the mean pose estimate and J is the Jacobianthereof.
Revised: Here H is the Hessian of the tracking objective at the mean pose estimate and J is the Jacobian.

Original: The Jacobian connects the pose to all image pixel errors.
Revised: The Jacobian connects the pose to all image pixel errors.

Original: The more consistent Jacobianentries are for a given pose dimension, the smaller the variance for thatpose dimension will be.
Revised: The more consistent Jacobian entries are for a given pose dimension, the smaller the variance for that dimension will be.

Revised: We refer

Original: 5.3 Localisation Accuracy
Revised: 5.2 Localisation Accuracy

Original: We compare PRISM’s localisation to state-of-the-art SLAM solutions.
Revised: We compare PRISM’s localisation to state-of-the-art methods in moderately-sized indoor environments.

Original: We consider both baselineswith dense mapslike ours (TANDEM [8], VSSM-LM [12], iMAP [9], NICE-SLAM [7], CodeVIO[77]), and sparse SLAM without rendering (ORB-SLAM2 [4], VINS [78], VIMO [79]).
Revised: We consider both baselines with dense maps (TANDEM [10], VSSM-LM [15], iMAP [11], NICE-SLAM [9], CodeVIO [86]) and sparse methods without rendering (ORB-SLAM2 [4], VINS [71], VIMO [70]).

Original: We summarise the results  in table 1.
Revised: The results are in table 1.

Original: Overall, localisation accuracy is comparable to modern SLAM methods,with differences of a few centimeters.
Revised: For the considered trajectories accuracy is comparable to the baselines, with differences of a few centimeters.

Original: At the same time, our inference boasts an explicitly deﬁnedprobabilistic generative model, with both rendering and dynamics, as well as uncertainty estimateswhich are not common in the  visual SLAM literature.
Revised: At the same time, our inference boasts a predictive state-space model with both rendering and dynamics as well as uncertainty estimates, which is not common in the dense visual SLAM literature.

Original: Finally, in ﬁg. 4 we seean example for inferredagent velocities, noting the uncertainty bands.
Revised: Finally, in ﬁg. 4 we see example inferred agent velocities, noting the uncertainty bands.

Original: This is possible because we model the agent dynamics.
Revised: This is possible because we model the agent dynamics.

Original: Our localisation accuracy on Blackbird is better than the off-line variational inference results of
Revised: Our localisation accuracy on Blackbird is better than the off-line variational inference results of

Original: VSSM-LM presented by Mirchev et al.
Revised: VSSM-LM presented by Mirchev et al.

Original: [12], and at the same time our solution runs in real-time andalso captures uncertainty.
Revised: [15], and at the same time our solution runs in real-time and also captures uncertainty.

Original: This shows the advantages of the proposed divide-and-conquer ﬁltering,which concentrates computation on solving one ﬁltering step at a time well.
Revised: This shows the advantages of the proposed divide-and-conquer ﬁltering.

Original: 5.4 Approximations for Runtime Improvement
Revised: 5.3 Approximations for Runtime Improvement

Original: All of our approximations are motivated by the real-time constraint, dictating the need for closed-formmap updates, a Laplace approximation, linearisation assumptions and a surrogate pose optimisationobjective.
Revised: All of our approximations are motivated by the real-time constraint, dictating the need for closed-form map updates, a Laplace approximation, linearisation assumptions and a surrogate pose optimisation objective.

Original: Figure 5 shows a runtime breakdown for different image resolutions, measured on an
Revised: Figure 5 shows a runtime breakdown for different image resolutions, measured on an

Original: NVIDIA 1080 TI GPU and an Intel(R) Xeon(R)

Original: W-2123 CPU at 3.6 GHz.

Original: The heaviest operations

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: ofﬁce

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: {+NVIDIA 
 Ti GPU and an Intel(R) Xeon(R) W-2123 CPU at 3.6 GHz. The heaviest operations+} are rendering and the gradient-based pose optimisation.
Revised: {+NVIDIA 
 Ti GPU and an Intel(R) Xeon(R) W-2123 CPU at 3.6 GHz. The heaviest operations+} are rendering and the gradient-based pose optimisation.

Original: Based on movement speed, rendering canhappen periodically, whenever a new anchor image prediction for pose optimisation is needed.
Revised: Based on movement speed, rendering can happen periodically, whenever a new anchor image prediction for pose optimisation is needed.

Original: Thisleaves us with a total runtime of 10 Hz to 15 Hz, updating the map and state at every data step.
Revised: This leaves us with a total runtime of 10 Hz to 15 Hz, updating the map and state at every data step.

Original: In ﬁg. 6 we  compare closed-form map updates to map inference via gradient-descent (e
Revised: In ﬁg. 6 we also compare closed-form map updates to map inference via gradient-descent (e

Original: [Figure]
Revised: [Figure]

Original: g. asused in NERF [6], VSSM-LM [12], iMAP [9], NICE-SLAM [7]).
Revised: g. as in [15, 8, 11, 9]).

Original: While gradient-descent is perfectly alignedwith the rendering and can lead to the smallest reconstruction error, it is much more expensive.
Revised: While gradient-descent is more accurate on a bigger budget, it is much more expensive.

Original: Forexample, to match the accuracy of the closed-form updates, which take less than 10 ms, one wouldneed ca.
Revised: For example, to match the accuracy of the closed-form updates, which take less than 10 ms, one would need ca.

Original: 250 ms of optimisation, which is impractical.
Revised: 250 ms of optimisation, which is impractical.

Original: These runtimes are for a voxel grid that issigniﬁcantly faster than neural representations [50], which would only exacerbate the problem.
Revised: These runtimes are for a voxel grid that is signiﬁcantly faster than neural representations [54], which would only exacerbate the problem.

Original:  Limitations 
Revised:  Limitations and Conclusion

Original: We use SDF voxel grids as they can be updated probabilistically in closed form.
Revised: SDF voxel grids allow for closed-form updates, but their memory footprint limits the maximum resolution and scene size.

Original: We note that voxelhashing [41] can directly replace them forbetter memory efﬁciency.
Revised: Voxel hashing [45] or octrees [87] can directly replace them for memory efﬁciency.

Original: Neural maps and dynamicallychanging maps have remained out of our scope.
Revised: Neural maps and dynamically changing maps have remained out of our scope.

Original: Their probabilistic formulation and closed-formupdates require further investigation.
Revised: Their probabilistic formulation and closed-form updates require further investigation.

Original: Our map factorises over voxels with no inter-region correlation,which could also be improved.
Revised: Our map factorises over voxels with no inter-region correlation, which could also be improved.

Original: PRISM provides interpretable uncertainty in real-time, but estimationis approximate.
Revised: PRISM provides interpretable uncertainty in real-time, but estimation is approximate.

Original: Obtaining perfectly calibrated uncertainty on a budget remains an open question.
Revised: Obtaining perfectly calibrated uncertainty on a budget remains an open question (see appendix H).

Original: Conclusion
Revised: While ﬁltering works for our generative assumptions indoors, ﬁlters cannot revisit past errors and can drift in large scenes with high levels of exploration [6]. We leave large-scale inference considerations for future work.

Original: We have introduced PRISM, a method for probabilistic ﬁltering in a predeﬁned spatial world model.
Revised: We have introduced PRISM, a method for probabilistic ﬁltering in a predeﬁned spatial state-space model.

Original: Our solution runs in real-time, provides state and map uncertainty, and infers a dense map and a 6-DoFstate trajectory, including velocities, with comparable accuracy to state-of-the-art SLAM algorithms.
Revised: Our solution runs in real-time, provides state and map uncertainty, and infers a dense map and a 6-DoF state trajectory with velocities. It is comparably accurate to state-of-the-art SLAM in indoor environments.

Original: To the best of our knowledge this is the ﬁrst real-time capable probabilistic solution for SLAM thatcombines differentiable rendering andaccounts for the agent dynamics.
Revised: To the best of our knowledge this is the ﬁrst real-time fully-probabilistic solution for SLAM that combines differentiable rendering and agent dynamics.

Original: We validated our method onthree challenging data sets, featuring unmanned aerial vehicles and a handheld camera.
Revised: We validated our method on three challenging data sets, featuring unmanned aerial vehicles and a handheld camera.

Original: The resultsare promising, establishing PRISM as a viable state estimator for downstream model-based control.
Revised: The results are promising, establishing PRISM as a viable state estimator for downstream model-based control.

Revised: Acknowledgments

Revised: We thank our reviewers for the thoughtful discussion, it helped us to better position our contribution.