  More recently, Bidirectional Encoder Representations from Transformers (BERT)
was proposed and has achieved impressive success on many natural language
processing (NLP) tasks such as question answering and language understanding,
due mainly to its effective pre-training then fine-tuning paradigm as well as
strong local contextual modeling ability. In view of the above, this paper
presents a novel instantiation of the BERT-based contextualized language models
(LMs) for use in reranking of N-best hypotheses produced by automatic speech
recognition (ASR). To this end, we frame N-best hypothesis reranking with BERT
as a prediction problem, which aims to predict the oracle hypothesis that has
the lowest word error rate (WER) given the N-best hypotheses (denoted by
PBERT). In particular, we also explore to capitalize on task-specific global
topic information in an unsupervised manner to assist PBERT in N-best
hypothesis reranking (denoted by TPBERT). Extensive experiments conducted on
the AMI benchmark corpus demonstrate the effectiveness and feasibility of our
methods in comparison to the conventional autoregressive models like the
recurrent neural network (RNN) and a recently proposed method that employed
BERT to compute pseudo-log-likelihood (PLL) scores for N-best hypothesis
reranking.
