PMAES : Prompt-mapping Contrastive Learning for Cross-prompt Automated Essay Scoring (2023)

Abstract: Current cross-prompt automated essay scoring (AES) is a challenging task due to the large discrepancies between different prompts, such as different genres and expressions. The main goal of current cross-prompt AES systems is to learn enough shared features between the source and target prompts to grade well on the target prompt. However, because the features are captured based on the original prompt representation, they may be limited by being extracted directly between essays. In fact, when the representations of two prompts are more similar, we can gain more shared features between them. Based on this motivation, in this paper, we propose a learning strategy called “prompt-mapping” to learn about more consistent representations of source and target prompts. In this way, we can obtain more shared features between the two prompts and use them to better represent the essays for the target prompt. Experimental results on the ASAP++ dataset demonstrate the effectiveness of our method. We also design experiments in different settings to show that our method can be applied in different scenarios. Our code is available at https://github.com/gdufsnlp/PMAES .

## introduction automated essay scoring (aes) aims to evaluate the quality of essays automatically. compared with human grading process, a robust aes system can not only reduce the work of teachers, but also improve the consistency of grading @xcite and make it broadly available to language learners. aes has been studied for many years. early studies focus more on handcrafted features, such as lexical features @xcite @xcite . with the rise of deep learning, many studies based on neural networks for prompt-specific settings have been proposed and achieved better results @xcite @xcite . these studies follow the same setting, that is, both rated training essays and unrated test essays belong to the same prompt. another type of work is cross-prompt aes. in this setting, labeled training essays are from source prompts and unlabeled test essays are from a different target prompt. existing studies mainly focus on obtaining sufficient shared features between source and target prompts to grade the target prompt essays effectively. some of them obtain shared features by extracting handcrafted features @xcite @xcite while others learn shared features by optimizing additional training objectives, such as the multi-task learning @xcite , two-stage strategy @xcite and self-supervised learning task @xcite . although these methods can effectively capture shared features between different prompts, we argue that these features are captured based on the original representations of the essays from source and target prompts. it may be limited by directly extracting the shared features among them. intuitively, when the representations of the essays from the source and target prompts are more consistent, they can share more knowledge between them. to this end, we propose a prompt representation learning framework for cross-prompt aes (pmaes) in which we design a prompt-mapping contrastive learning strategy to effectively learn about more consistent representations of source and target prompts. to do this, we design a mapping operation to project each essay from the source prompt to the target prompt and get its mapping representation specific to the target prompt. for each essay on the source prompt (let's say r s ), we first determine how similar it is to all the essays in the target prompt by their original representations (e.g., by taking the dot product with the inverse matrix of the representations of all the essays in the target prompt) as the weights of the r s to each target essay. then, we employ a learnable parameter matrix (specifically, a prompt-mapping matrix) to acquire the weighted representation of the source prompt essay projected on the target prompt to express the mapping representation of the source essay r s (let's say rs ). these source essay representations and source mapping representations are treated as the source-to-target mapping pairs (r s , rs ). by decreasing the distance between the essays in these mapping pairs, we may gradually reduce the discrepancy between the source and target prompts and finally make the representations of the two prompt essays more consistent. it is worth noting that the above description is about mapping from source to target. naturally, we also perform target-to-source prompt mapping operations to further learn a more consistent representations of the two prompts, which will be described in section 3.4. as demonstrated in figure 1 , given the original essay representations of a source and a target prompt (which we marked in green and red, respectively), there are very few shared features between them under the original representations (which we marked in yellow). when we train the model using our proposed prompt-mapping approach, the representations of the two prompts may become more similar, which enables more shared features across the two prompts. we show them in figure 1(b) and figure 1(c ). as the shared features increase, we can get more accurate representations of target prompt essays and grade them more accurately. to summarize, the main contributions of our work are as follows: ## our approach the whole architecture of our approach is shown in figure 2 . it contains three components: shared encoder, scorer and prompt-mapping contrastive learning. the shared encoder provides a shared representation for the other two components, the scorer is used to predict the score, and the promptmapping contrastive learning is used to maximize the consistency of source and target prompts. ## conclusions in this paper, we propose a new method for cross-prompt aes that aims to capture more shared features between the source and target prompts. specifically, we design prompt-mapping contrastive learning to decrease the distance between the mapping pairs from source-to-target and target-to-source simultaneously and finally make the representations of the two prompts more consistent. experimental results demonstrate that our approach achieves the state-of-the-art on both singleoverall scoring task and multi-attribute scoring task. we further design experiments for three sourcetarget settings, which proves that our approach can be adapted to multiple scenarios.