LOT : A Story-Centric Benchmark for Evaluating C hinese Long Text Understanding and Generation (2022)

Abstract: Standard multi-task benchmarks are essential for developing pretraining models that can generalize to various downstream tasks. Existing benchmarks for natural language processing (NLP) usually focus only on understanding or generating short texts. However, long text modeling requires many distinct abilities in contrast to short texts, such as the modeling of long-range discourse and commonsense relations, and the coherence and controllability of generation. The lack of standardized benchmarks makes it difficult to assess these abilities of a model and fairly compare different models, especially Chinese models. Therefore, we propose a story-centric benchmark named LOT for evaluating Chinese long text modeling, which aggregates two understanding tasks and two generation tasks. We construct new datasets for these tasks based on human-written Chinese stories with hundreds of words. Furthermore, we release an encoder-decoder-based Chinese long text pretraining model named LongLM with up to 1 billion parameters. We pretrain LongLM on 120G Chinese novels with two generative tasks including text infilling and conditional continuation. Extensive experiments show that LongLM outperforms similar-sized pretraining models substantially on both the understanding and generation tasks in LOT.

## introduction pretrained language models have achieved significant advances in various natural language understanding (nlu) and generation (nlg) tasks effendi's son is eccentric, always behaving opposed to what effendi has ordered him to do. familiar to his son's temper, effendi usually communicates using irony. one day, the father and son were blocked by a river after purchasing flour from a mill. and while they were crossing the river, one bag on the donkey's back lost its weight and leaned. effendi told his son with irony:''my boy! drop the sack into the river!'' the son heard the words and thought:''i have been opposed to my father for so many years. for this only time, i have to obey him.'' therefore, he followed effendi's words and indeed pushed the sack into the river. ''my boy! what are you doing?''effendi shouted in anger.'' ... furthermore, we release longlm, a chinese long text pretraining language model. longlm is a transformer-based model with an encoderdecoder architecture. longlm has three different versions ranging from 60 million to 1 billion parameters. we pretrain longlm on 120g chinese novels with two generative tasks, including text infilling @xcite and conditional continuation @xcite . the pretraining data do not include other types of texts (e.g., news, wiki-texts) since we mainly focus on commonsense and discourse relations within general long texts instead of factual and technical knowledge. to the best of our knowledge, longlm is the first pretraining model of the same size scale that focuses on modeling long-form stories. extensive experiments on lot show that longlm outperforms strong baselines substantially on both the understanding and generation tasks. however, we also observe that longlm is still far behind human performance, which requires better semantic representations of events and deeper modeling of the commonsense and discourse relations between them. we summarize the main contributions of this paper as follows: i. we propose a new story-centric benchmark lot for evaluating chinese long text understanding and generation. lot consists of four tasks for testing the fundamental abilities to model long texts. we also present new datasets for these tasks. ii. we release a new chinese pretraining model named longlm. experiment results demonstrate the strong performance of longlm on lot, but there still exists considerable room for improvement. 1 2 related work nlp benchmarks recently, there have been a lot of multi-task benchmarks proposed to drive the progress of generalizable models. the benchmarks usually aggregate multiple model-agnostic tasks under a unified framework, enabling researchers to fairly compare different models. senteval @xcite gathered multiple classification tasks involving either one or two sentences as inputs to evaluate sentence representations. discoeval @xcite extended these tasks to the discourse level regarding inter-sentence relations. glue @xcite included more diverse tasks such as natural language inference @xcite . @xcite proposed superglue as a more challenging counterpart of glue by introducing multi-sentence tasks. but the additional tasks are only limited to the formats of coreference resolution and question answering. in addition to these english benchmarks, many benchmarks were proposed to evaluate nlu for other languages, such as clue @xcite for chinese. moreover, glge @xcite and gem @xcite were proposed for evaluating nlg models across diversified generation tasks such as text summarization and personalizing dialogue. however, there is no benchmark designed specifically for long text modeling, especially chinese. additionally, the above benchmarks were originally designed to cover as diverse task formats as possible. in contrast, we design the lot tasks with the guidance of necessary abilities for long text modeling as suggested by @xcite , making it easier to figure out where models are failing, and how to improve them. long text datasets previous studies in the field of long text modeling have frequently focused on the rocstories @xcite and writingprompts @xcite datasets. rocstories contains 100k artificial five-sentence stories, while writingprompts consists of 300k pairs of prompts and stories with hundreds of words. recent works collected stories with thousands of words to model longer-range dependencies, such as wikitext-103 @xcite , roleplayerguild @xcite , pg-19 @xcite , storium @xcite , and long-range arena @xcite . however, these datasets are written in english. lot will drive the development of chinese language models. moreover, lot does not include datasets of extra-long texts, like pg-19, for the following two reasons: (1) extra-long texts are far beyond the scope of current machine learning models because the discourse-level linguistic phenomena are entangled and complicated in these texts. therefore, extra-long texts usually serve for computing perplexity of language models @xcite but hardly provide fine-grained guidance for improving model designs. (2) lot aims not to spur research on building fuller connections across tokens within an extra-long sequence, but to drive the progress of machines in the aforementioned fundamental abilities for long text modeling. ## lot benchmark we design lot as an aggregation of two understanding tasks including cloze test (clozet) and sentence position prediction (senpos), and two generation tasks including plot completion (plotcom) and outline-conditioned generation (outgen). we show the task descriptions and data statistics in tables 2 and 3 , respectively. we use the jieba tokenizer foot_1 for word tokenization. we design lot based on the following principles: (1) task diversity: the tasks vary in task formats, types and lengths of inputs and outputs, and focused abilities, making lot a comprehensive framework for evaluating the generalization of models. (2) task difficulty: the tasks take hundreds of words as inputs or outputs, and do not involve domain-specific knowledge about science, films, and so forth. therefore, they are beyond the scope of current state-of-the-art models, but are solvable by most chinese native speakers. (3) task formulation: the tasks have been well formulated in prior studies and agreed to be challenging but meaningful. we introduce new chinese datasets for these tasks, which are constructed to focus more specifically on testing tasks abilities inputs outputs metrics clozet commonsense reasoning a text with a sentence removed (the position specified); two candidate sentences. choosing the correct sentence from two candidates. accuracy senpos inter-sentence relationship a text with a sentence removed (the position unspecified); the removed sentence. choosing the correct position for the removed sentence. accuracy plotcom commonsense reasoning; inter-sentence relationship a text with a sentence removed (the position specified). generating a sentence to complete the text. bleu; dist outgen discourse structure; coherence; controllability a title, an outline as an out-of-order set of phrases about characters and events. generating a coherent text adhering to the title and outline. bleu; dist; cover; order as story generation from titles, which is difficult to automatically evaluate @xcite because the tasks suffer from the notorious one-to-many issue: there are many plausible outputs for the same input @xcite . we constructed datasets for lot through automatic and manual annotation. first, we crawled human-written stories from public web pages as the data source. these stories are under licenses that allow use and redistribution for research purposes. then, we hired a commercial team to create the lot examples. the team is led by a professional screenwriter and has taken on hundreds of nlp annotation projects. all annotators are native chinese speakers and well-trained for the annotation tasks. we show the full list of the source web pages and the annotation details in the appendix. ## experiments in this section, we tested longlm and existing models on lot with automatic and manual evaluation. furthermore, we conducted extensive experiments to investigate the potential biases of the clozet and senpos datasets (section 5.5), and measure the overlap between training and testing data (section 5.6). ## conclusions we present lot, a story-centric benchmark for chinese long text understanding and generation. lot includes two story understanding tasks and two story generation tasks, which comprehensively investigate the abilities of commonsense reasoning, controllable generation, and modeling inter-sentence relations and the global discourse structures. we provide standard datasets for the four tasks, which are constructed based on human-written stories processed by automatic and manual annotation. furthermore, we release a new chinese long text pretraining model longlm, which outperforms strong baseline models substantially on both the understanding and generation tasks in lot. the lot benchmark and the pretraining model will encourage further research on chinese long text modeling.