C hinese WPLC : A C hinese Dataset for Evaluating Pretrained Language Models on Word Prediction Given Long-Range Context (2021)

Abstract: This paper presents a Chinese dataset for evaluating pretrained language models on Word Prediction given Long-term Context (Chinese WPLC). We propose both automatic and manual selection strategies tailored to Chinese to guarantee that target words in passages collected from over 69K novels can only be predicted with long-term context beyond the scope of sentences containing the target words. Dataset analysis reveals that the types of target words range from common nouns to Chinese 4-character idioms. We also observe that linguistic relations between target words and long-range context exhibit diversity, including lexical match, synonym, summary and reasoning. Experiment results show that the Chinese pretrained language model PanGu- \alpha is 45 points behind human in terms of top-1 word prediction accuracy, indicating that Chinese WPLC is a challenging dataset. The dataset is publicly available at https://git.openi.org.cn/PCL-Platform.Intelligence/Chinese_WPLC .

## introduction predicting a target word from previous context, especially long-range context, is a long-standing challenging problem in natural language processing. a variety of large-scale datasets such as cnn/daily mail @xcite , @xcite and cmrc-2017 @xcite have been developed to examine the capability of machines in word prediction. however, the majority of such datasets have not undergone a thorough manual testing whether a target word can only be predicted from long-range dependencies except for lambada @xcite . this dataset provides a benchmark testbed where a target word can be easily predicted with long-range context but cannot with only context words in the sentence where the target word is located. partially inspired by lambada, we create chinese wplc, a dataset for evaluating powerful pretrained language models on word prediction with long-range context. the passages used in our dataset are carefully extracted from over 69k chinese novels following a procedure mixed with automatic and manual selection. significant differences from lambda lie not only in language (english vs. chinese), but also in the following two aspects: we carry out an in-depth analysis on the built dataset, finding that the relations between target words and previous context ranges from lexical match, synonym, summary to commonsense reasoning. we conduct experiments on the built dataset to evaluate a range of state-of-the-art chinese pretrained models, including the chinese pretrained model pangu-α with up to 200 billion parameters @xcite , which achieves a top-1 accuracy of 12.1%, 45.2 points behind human dataset task data collection qa train development test language cnn/daily mail entity prediction ac 380,298/879,450 3,924/6,4835 3,198/53,182 en cbt entity prediction ac 669,343 8,000 10,000 en lambada word prediction rnnf+mc -4,869 5,153 en wsc commonsense reasoning mc --285 en winogrande commonsense reasoning mc --44,000 en wplc (ours) word prediction plmf+mc -4,827 4,474 zh in chinese, people daily (pd) & children's fairy tale (cft) @xcite corpus is the first cloze-style reading comprehension dataset in chinese. chid @xcite offers an interesting task where words to be predicted are all idioms. cluewsc2020 @xcite , a chinese version of wsc dataset, aims to test the ability of coreference resolution via word prediction. significantly different from such chinese datasets, our dataset is specifically developed for evaluating word prediction from long-range context. ## experiments we carried out experiments with a range of stateof-the-art pretrained language models on chinese wplc. as bert-large and the last layer of roberta-large are currently not available for chinese, results of these two models are not provided. top-1 and top-3 accuracy are reported. ## conclusions in this paper, we have presented the chinese wplc, a chinese word prediction dataset created from over 69k novels to examine the ability of pretrained language models on long-term context modeling. we employ both automatic and manual selection strategies to keep passages where target words can be only predicted from long-term context beyond target sentences and it is difficult for pretrained language model to predict target words. experiments with a range of state-of-the-art pretrained language models and in-depth analyse demonstrate that the created dataset is a very challenging testbed even for the very large chinese pretrained panguα, covering a variety of linguistic phenomena (e.g., lexical match, synonym, summary and reasoning).