  Although vital to computer vision systems, few-shot action recognition is
still not mature despite the wide research of few-shot image classification.
Popular few-shot learning algorithms extract a transferable embedding from seen
classes and reuse it on unseen classes by constructing a metric-based
classifier. One main obstacle to applying these algorithms in action
recognition is the complex structure of videos. Some existing solutions sample
frames from a video and aggregate their embeddings to form a video-level
representation, neglecting important temporal relations. Others perform an
explicit sequence matching between two videos and define their distance as
matching cost, imposing too strong restrictions on sequence ordering. In this
paper, we propose Compromised Metric via Optimal Transport (CMOT) to combine
the advantages of these two solutions. CMOT simultaneously considers semantic
and temporal information in videos under Optimal Transport framework, and is
discriminative for both content-sensitive and ordering-sensitive tasks. In
detail, given two videos, we sample segments from them and cast the calculation
of their distance as an optimal transport problem between two segment
sequences. To preserve the inherent temporal ordering information, we
additionally amend the ground cost matrix by penalizing it with the positional
distance between a pair of segments. Empirical results on benchmark datasets
demonstrate the superiority of CMOT.
