Temporally locating and classifying action segments in long untrimmed videos is of particular interest to many applications like surveillance and robotics. While traditional approaches follow a two-step pipeline, by generating framewise probabilities and then feeding them to high-level temporal models, recent approaches use temporal convolutions to directly classify the video frames. In this paper, we introduce a multi-stage architecture for the temporal action segmentation task. Each stage features a set of dilated temporal convolutions to generate an initial prediction that is refined by the next one. This architecture is trained using a combination of a classification loss and a proposed smoothing loss that penalizes over-segmentation errors. Extensive evaluation shows the effectiveness of the proposed model in capturing long-range dependencies and recognizing action segments. Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.
With the multi-stage pipeline, MS-TCN REF performs hierarchical temporal convolutions to effectively extract temporal features and achieve the state-of-the-art performance for action segmentation.
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Computer Science