  Current approaches to video analysis of human motion focus on raw pixels or
keypoints as the basic units of reasoning. We posit that adding higher-level
motion primitives, which can capture natural coarser units of motion such as
backswing or follow-through, can be used to improve downstream analysis tasks.
This higher level of abstraction can also capture key features, such as loops
of repeated primitives, that are currently inaccessible at lower levels of
representation. We therefore introduce Motion Programs, a neuro-symbolic,
program-like representation that expresses motions as a composition of
high-level primitives. We also present a system for automatically inducing
motion programs from videos of human motion and for leveraging motion programs
in video synthesis. Experiments show that motion programs can accurately
describe a diverse set of human motions and the inferred programs contain
semantically meaningful motion primitives, such as arm swings and jumping
jacks. Our representation also benefits downstream tasks such as video
interpolation and video prediction and outperforms off-the-shelf models. We
further demonstrate how these programs can detect diverse kinds of repetitive
motion and facilitate interactive video editing.
