  Existing methods for skeleton-based action recognition mainly focus on
improving the recognition accuracy, whereas the efficiency of the model is
rarely considered. Recently, there are some works trying to speed up the
skeleton modeling by designing light-weight modules. However, in addition to
the model size, the amount of the data involved in the calculation is also an
important factor for the running speed, especially for the skeleton data where
most of the joints are redundant or non-informative to identify a specific
skeleton. Besides, previous works usually employ one fix-sized model for all
the samples regardless of the difficulty of recognition, which wastes
computations for easy samples. To address these limitations, a novel approach,
called AdaSGN, is proposed in this paper, which can reduce the computational
cost of the inference process by adaptively controlling the input number of the
joints of the skeleton on-the-fly. Moreover, it can also adaptively select the
optimal model size for each sample to achieve a better trade-off between
accuracy and efficiency. We conduct extensive experiments on three challenging
datasets, namely, NTU-60, NTU-120 and SHREC, to verify the superiority of the
proposed approach, where AdaSGN achieves comparable or even higher performance
with much lower GFLOPs compared with the baseline method.
