A World CLASSE Student Summary Corpus (2024)

Abstract: This paper introduces the Common Lit Augmented Student Summary Evaluation (CLASSE) corpus. The corpus comprises 11,213 summaries written over six prompts by students in grades 3-12 while using the CommonLit website. Each summary was scored by expert human raters on analytic features related to main points, details, organization, voice, paraphrasing, and language beyond the source text. The human scores were aggregated into two component scores related to content and wording. The final corpus was the focus of a Kaggle competition hosted in late 2022 and completed in 2023 in which over 2,000 teams participated. The paper includes a baseline scoring model for the corpus based on a Large Language Model (Longformer model). The paper also provides an overview of the winning models from the Kaggle competition.

## introduction many educational applications are interested in assessing student-generated knowledge to assess learning and development. in terms of assessing student comprehension of text, generation effects @xcite that result from students writing about what they have read have been shown to substantially improve learning @xcite . a number of educational applications have taken advantage of generation effects to enhance students' reading comprehension skills, including summary street @xcite , the interactive strategy training for active reading and thinking (istart) tool @xcite , the commonlit online reading program (commonlit.org), and the intelligent textbooks for enhanced lifelong learning (itell) framework @xcite . the most common approach to assessing students' reading comprehension through text generation is likely through text summarization. text summarization is a valuable tool to build and assess student knowledge @xcite because the process of summarization helps students build and consolidate their knowledge about reading materials @xcite . text summarization has also been shown to lead to stronger learning gains than other forms of comprehension assessment, including constructed responses @xcite , long-form essays @xcite , and traditional assessments like multiple-choice and fill-in-the-blank questions @xcite . while effective, many teachers hesitate to integrate summary assessments of reading in the classroom because manually grading summaries is resource-intensive @xcite . however, student text summarization can also be assessed automatically through the use of natural language processing (nlp) techniques such as semantic similarity metrics @xcite @xcite replication, and generalizability of summarization models. additionally, many of the corpora used in previous research have included summaries written by crowdsourced workers and not students @xcite @xcite the goal of this study is to introduce the common lit augmented student summary evaluation (classe) corpus. the corpus comprises 11,213 summaries written over six prompts by students in grades 3-12. all summaries were written on the commonlit website. each summary was scored by expert human raters on analytic features related to summarization content and wording. the study also introduces a baseline nlp summary scoring model for the corpus as well as the winning models developed in a large-scale data science competition hosted for the corpus. ## the classe corpus while research ase has gained traction and shown improvements over the last 20 years, the work is somewhat fragmented. a major reason for this is that researchers do not have a large-scale opensource summarization corpus to develop, test, and validate ase models. other reasons include the use of different nlp approaches to model summarization quality, the sampling of different populations of writers, and the use of different scoring metrics. the common lit augmented student summary evaluation (classe) corpus is meant to help address this fragmentation by providing researchers with a gold-standard corpus of opensource summaries written by students. the corpus is freely available in the following repository: https://github.com/scrosseye/classe . ## baseline prediction model for classe corpus we developed a simple baseline model for the classe by finetuning a longformer model @xcite to predict the content and wording scores, given the original text and the summary. the baseline model is not meant to extend the technical boundaries of summary classification models but rather provide a simple metric from which to measure scoring gains. deberta model (v3 large), but critically, they augmented the training set by creating 1000 new prompts with associated sources using generative ai. for each prompt, they also created 21 summaries and pseudo-labeled those summaries. other common approaches used to improve the models included using a head mask for only the student summaries instead of a normal attention mask, using generative ai models to generate varieties of the existing prompts, hyperparameter searches, extending the inference max length, and using all of the input (summary, prompt, source, and title) in the training models. ## discussion and conclusion this paper has introduced the classe corpus, the scoring metrics for the corpus, and a baseline model for summary scoring based on a deberta transformer-based encoder. the paper also introduced the winning summarization models from the kaggle competition held in support of the classe corpus. the classe comprises 11,213 summaries written over six prompts by students in grades 3-12 while using the commonlit website. each summary was scored by expert human raters on analytic features related to summarization content and wording. reliability metrics for the human scoring indicated substantial reliability in all items except paraphrasing/wording, which reported moderate reliability. paraphrasing is the restatement of a passage such that the propositional meaning is similar, but the words and structures differ. recognizing when words differ between passages is relatively easy, but recognizing the alteration of clauses is a difficult task @xcite , which may explain the moderate reliability reported by human raters. the analytic scores were aggregated into components using a principal component analysis (pca) to better represent the underlying structure of the human ratings. the pca reported two components related to content and wording. content included features related to main ideas, details for those ideas, the organization of those ideas, and the objectivity of how those ideas were presented. the content component provides an overall assessment of how the ideas in the source text are distilled into a coherent and objective framework in the student summaries. wording includes features related to paraphrasing and the use of language beyond the source. this component was concerned with the manner in which the summary presented the ideas from the source text, specifically, did the summary use original wording (paraphrasing) and whether this wording was lexically and syntactically complex. the baseline model introduced in this paper used a longformer model that used both the summary and the source text as input for model predictions. the longformer performed well on the training data but reported drops in the validation and test data. this is the result of the longformer model learning the patterns of successful summarization specific to the four prompts in the training set but not learning how to extend scoring beyond those prompts to the two unique prompts in the validation and test sets. the results of the subsequent kaggle competition showed a number of innovations that helped competitors produce winning models, many of which addressed the limitations of the baseline model. the winning model used a single transformer encoder (deberta v3 large), but, importantly, they augmented their training data to include a much larger number of prompts and summaries written on those prompts. extending the number of prompts and summaries allowed the model to generalize better to the unique prompts found in the validation and test set. other innovations in summary scoring that resulted from the kaggle competition included pseudo-labeling of ai generated summaries for content and wording scores, the use of head masks, and extending the inference max length.