  Punctuation prediction for automatic speech recognition (ASR) output
transcripts plays a crucial role for improving the readability of the ASR
transcripts and for improving the performance of downstream natural language
processing applications. However, achieving good performance on punctuation
prediction often requires large amounts of labeled speech transcripts, which is
expensive and laborious. In this paper, we propose a Discriminative
Self-Training approach with weighted loss and discriminative label smoothing to
exploit unlabeled speech transcripts. Experimental results on the English
IWSLT2011 benchmark test set and an internal Chinese spoken language dataset
demonstrate that the proposed approach achieves significant improvement on
punctuation prediction accuracy over strong baselines including BERT, RoBERTa,
and ELECTRA models. The proposed Discriminative Self-Training approach
outperforms the vanilla self-training approach. We establish a new
state-of-the-art (SOTA) on the IWSLT2011 test set, outperforming the current
SOTA model by 1.3% absolute gain on F$_1$.
