  Fingerspelling, in which words are signed letter by letter, is an important
component of American Sign Language. Most previous work on automatic
fingerspelling recognition has assumed that the boundaries of fingerspelling
regions in signing videos are known beforehand. In this paper, we consider the
task of fingerspelling detection in raw, untrimmed sign language videos. This
is an important step towards building real-world fingerspelling recognition
systems. We propose a benchmark and a suite of evaluation metrics, some of
which reflect the effect of detection on the downstream fingerspelling
recognition task. In addition, we propose a new model that learns to detect
fingerspelling via multi-task training, incorporating pose estimation and
fingerspelling recognition (transcription) along with detection, and compare
this model to several alternatives. The model outperforms all alternative
approaches across all metrics, establishing a state of the art on the
benchmark.
