  In this paper, we introduce a non-parametric memory representation for
spatio-temporal segmentation that captures the local space and time around an
autonomous vehicle (AV). Our representation has three important properties: (i)
it remembers what it has seen in the past, (ii) it reinforces and (iii) forgets
its past beliefs based on new evidence. Reinforcing is important as the first
time we see an element we might be uncertain, e.g, if the element is heavily
occluded or at range. Forgetting is desirable, as otherwise false positives
will make the self driving vehicle behave erratically. Our process is informed
by 3D reasoning, as occlusion is key to distinguishing between the desire to
forget and to remember. We show how our method can be used as an online
component to complement static world representations such as HD maps by
detecting and remembering changes that should be superimposed on top of this
static view due to such events.
