This paper presents a new UNIfied pre-trained Language Model (UNILM) that can be finetuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling objectives: unidirectional (both left-to-right and right-to-left), bidirectional, and sequence-tosequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. We can fine-tune UNILM as a unidirectional decoder, a bidirectional encoder, or a sequence-to-sequence model to support various downstream natural language understanding and generation tasks. 1 compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, our model achieves new state-of-theart results on three natural language generation tasks, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.63 (2.16 absolute improvement), pushing the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), and the SQuAD question generation BLEU-4 to 22.88 (6.50 absolute improvement).
UNILM REF uses multiple language modeling objectives -unidirectional (both left-to-right and right-to-left), bidirectional, and sequence-to-sequence prediction -to aid generation tasks like summarization and question generation.
Unified Language Model Pre-training for Natural Language Understanding and Generation
ArXiv
ArXiv
Computer Science