  In this work, we propose to improve the effectiveness of language
pre-training methods with the help of mis-predictions during pre-training.
Neglecting words in the input sentence that have conflicting semantics with
mis-predictions is likely to be the reason of generating mis-predictions at
pre-training. Therefore, we hypothesis that mis-predictions during pre-training
can act as detectors of the ill focuses of the model. If we train the model to
focus more on the conflicts with the mis-predictions while focus less on the
rest words in the input sentence, the mis-predictions can be more easily
corrected and the entire model could be better trained. Towards this end, we
introduce Focusing Less on Context of Mis-predictions(McMisP). In McMisP, we
record the co-occurrence information between words to detect the conflicting
words with mis-predictions in an unsupervised way. Then McMisP uses such
information to guide the attention modules when a mis-prediction occurs.
Specifically, several attention modules in the Transformer are optimized to
focus more on words in the input sentence that have co-occurred rarely with the
mis-predictions and vice versa. Results show that McMisP significantly
expedites BERT and ELECTRA and improves their performances on downstream tasks.
