  We propose a novel neural network module that transforms an existing
single-frame semantic segmentation model into a video semantic segmentation
pipeline. In contrast to prior works, we strive towards a simple and general
module that can be integrated into virtually any single-frame architecture. Our
approach aggregates a rich representation of the semantic information in past
frames into a memory module. Information stored in the memory is then accessed
through an attention mechanism. This provides temporal appearance cues from
prior frames, which are then fused with an encoding of the current frame
through a second attention-based module. The segmentation decoder processes the
fused representation to predict the final semantic segmentation. We integrate
our approach into two popular semantic segmentation networks: ERFNet and
PSPNet. We observe an improvement in segmentation performance on Cityscapes by
1.7% and 2.1% in mIoU respectively, while increasing inference time of ERFNet
by only 1.5ms.
