  Current video/action understanding systems have demonstrated impressive
performance on large recognition tasks. However, they might be limiting
themselves to learning to recognize spatiotemporal patterns, rather than
attempting to thoroughly understand the actions. To spur progress in the
direction of a truer, deeper understanding of videos, we introduce the task of
win-fail action recognition -- differentiating between successful and failed
attempts at various activities. We introduce a first of its kind paired
win-fail action understanding dataset with samples from the following domains:
"General Stunts," "Internet Wins-Fails," "Trick Shots," and "Party Games."
Unlike existing action recognition datasets, intra-class variation is high
making the task challenging, yet feasible. We systematically analyze the
characteristics of the win-fail task/dataset with prototypical action
recognition networks and a novel video retrieval task. While current action
recognition methods work well on our task/dataset, they still leave a large gap
to achieve high performance. We hope to motivate more work towards the true
understanding of actions/videos. Dataset will be available from
https://github.com/ParitoshParmar/Win-Fail-Action-Recognition.
