  The standard way of training video models entails sampling at each iteration
a single clip from a video and optimizing the clip prediction with respect to
the video-level label. We argue that a single clip may not have enough temporal
coverage to exhibit the label to recognize, since video datasets are often
weakly labeled with categorical information but without dense temporal
annotations. Furthermore, optimizing the model over brief clips impedes its
ability to learn long-term temporal dependencies. To overcome these
limitations, we introduce a collaborative memory mechanism that encodes
information across multiple sampled clips of a video at each training
iteration. This enables the learning of long-range dependencies beyond a single
clip. We explore different design choices for the collaborative memory to ease
the optimization difficulties. Our proposed framework is end-to-end trainable
and significantly improves the accuracy of video classification at a negligible
computational overhead. Through extensive experiments, we demonstrate that our
framework generalizes to different video architectures and tasks, outperforming
the state of the art on both action recognition (e.g., Kinetics-400 & 700,
Charades, Something-Something-V1) and action detection (e.g., AVA v2.1 & v2.2).
