  Event cameras are novel vision sensors that report per-pixel brightness
changes as a stream of asynchronous "events". They offer significant advantages
compared to standard cameras due to their high temporal resolution, high
dynamic range and lack of motion blur. However, events only measure the varying
component of the visual signal, which limits their ability to encode scene
context. By contrast, standard cameras measure absolute intensity frames, which
capture a much richer representation of the scene. Both sensors are thus
complementary. However, due to the asynchronous nature of events, combining
them with synchronous images remains challenging, especially for learning-based
methods. This is because traditional recurrent neural networks (RNNs) are not
designed for asynchronous and irregular data from additional sensors. To
address this challenge, we introduce Recurrent Asynchronous Multimodal (RAM)
networks, which generalize traditional RNNs to handle asynchronous and
irregular data from multiple sensors. Inspired by traditional RNNs, RAM
networks maintain a hidden state that is updated asynchronously and can be
queried at any time to generate a prediction. We apply this novel architecture
to monocular depth estimation with events and frames where we show an
improvement over state-of-the-art methods by up to 30% in terms of mean
absolute depth error. To enable further research on multimodal learning with
events, we release EventScape, a new dataset with events, intensity frames,
semantic labels, and depth maps recorded in the CARLA simulator.
