  Endowing visual agents with predictive capability is a key step towards video
intelligence at scale. The predominant modeling paradigm for this is sequence
learning, mostly implemented through LSTMs. Feed-forward Transformer
architectures have replaced recurrent model designs in ML applications of
language processing and also partly in computer vision. In this paper we
investigate on the competitiveness of Transformer-style architectures for video
predictive tasks. To do so we propose HORST, a novel higher order recurrent
layer design whose core element is a spatial-temporal decomposition of
self-attention for video. HORST achieves state of the art competitive
performance on Something-Something-V2 early action recognition and
EPIC-Kitchens-55 action anticipation, without exploiting a task specific
design. We believe this is promising evidence of causal predictive capability
that we attribute to our recurrent higher order design of self-attention.
