  Vision-based Continuous Sign Language Recognition (CSLR) aims to recognize
unsegmented gestures from image sequences. To better train CSLR models, the
iterative training scheme is widely adopted to alleviate the overfitting of the
alignment model. Although the iterative training scheme can improve
performance, it will also increase the training time. In this work, we revisit
the overfitting problem in recent CTC-based CSLR works and attribute it to the
insufficient training of the feature extractor. To solve this problem, we
propose a Visual Alignment Constraint (VAC) to enhance the feature extractor
with more alignment supervision. Specifically, the proposed VAC is composed of
two auxiliary losses: one makes predictions based on visual features only, and
the other aligns short-term visual and long-term contextual features. Moreover,
we further propose two metrics to evaluate the contributions of the feature
extractor and the alignment model, which provide evidence for the overfitting
problem. The proposed VAC achieves competitive performance on two challenging
CSLR datasets and experimental results show its effectiveness.
