A Comparative Analysis of the Effectiveness of Rare Tokens on Creative Expression using ram BERT (2023)

Abstract: Until now, few studies have been explored on Automated Creative Essay Scoring (ACES), in which a pre-trained model automatically labels an essay as a creative or a non-creative. Since the creativity evaluation of essays is very subjective, each evaluator often has his or her own criteria for creativity. For this reason, quantifying creativity in essays is very challenging. In this work, as one of preliminary studies in developing a novel model for ACES, we deeply investigate the correlation between creative essays and expressiveness. Specifically, we explore how rare tokens affect the evaluation of creativity for essays. For such a journey, we present five distinct methods to extract rare tokens, and conduct a comparative study on the correlation between rare tokens and creative essay evaluation results using BERT. Our experimental results showed clear correlation between rare tokens and creative essays. In all test sets, accuracies of our rare token masking-based BERT (ramBERT) model were improved over the existing BERT model up to 14%.

## introduction in the era of the fourth industrial revolution, new knowledge creation based on existing knowledge is becoming more important than any other field. with the advent of artificial intelligence (ai) technology, which is the core of this industrial revolution, ai takes charge of simple repetitive tasks and allow humans to focus more on creative activities. so that, new innovations can be achieved through collaboration between humans and ai. this knowledge creation is based on human creative thinking @xcite one of the representative creative thinking activities is writing. creative texts are new and at the same time communicable to readers within social and cultural contexts. creative writing is the act of writing creative texts. specifically, it is defined as a writing activity in which writers express their new and original ideas so that they can be communicated appropriately and effectively in social and cultural contexts. in this sense, through creative writing, middle and high school students can develop their ability for creativity that is essential for the fourth industrial revolution. creativity is a human mental activity, known as a complex characteristic that is difficult to explain. in addition, it is not limited to one academic field, but the nature of creativity varies slightly depending on the disciplines in question, such as linguistics, psychology, and literature. academically, there is no clear definition of creativity yet. looking at the discussion so far, it has been defined differently depending on which part of creativity the researcher has focused on. for example, @xcite defined creativity as "the process of becoming sensitive to problems, flaws, gaps in knowledge, missing elements, and incongruities", @xcite as "generating new and useful ideas," and @xcite as "the interaction between processes and capacities that produce novel and useful outputs within a specific social context." to date, the most widely accepted definition of creativity by many researchers is "an individual's ability to create something new and appropriate." as shown in the table, the creativity in outcome is divided into creativities in expression and content. because clearly evaluating the creativity in content is implausible with current technology, we focus now on assessing the creativity in expression while leaving the creativity in content as a topic for our future research. the creativity in expression has two common creativity evaluation indices. one is originality and the other is appropriateness. technically, the originality is the ability to produce new and unique expressions that are different from existing ones, and the appropriateness must include all the requirements of 'good writing', such as having to strictly follow the grammatical rules and, in any case, having to meet the requirements appropriate for the given context. in order to quantify the appropriateness index, various automated essay scoring (aes) models that take essays as input are actively developed in the field of natural language processing in recent years. we will introduce the state-of-the-art aes models in detail in section 2. please note that studying the appropriateness index is out of the scope of this work. in this paper, we focus only on the originality index in the creativity in expression. throughout this paper, the expression includes the tokens of subword, word, n-gram, and span(phrase or sentence). furthermore, to clearly quantify the originality index, as we already mentioned, we first consider it to be new and unique tokens within the given corpus, but the meaning of "new and unique" is ambiguous. in our second thought, we define it as rare tokens in the corpus. the goal of our research is to deeply investigate the correlation between rare tokens and creativity assessment. in the corpus, rare tokens can be extracted through various methods such as byte pair encoding (bpe), inverse document frequency (idf), clustering in latent semantic spaces, span-bert, and existing rare word dictionaries including stanford rare word @xcite , cambridge rare word @xcite , contextual rare word @xcite , and definitional nonce @xcite . in our framework, bidirectional encoder representations from transformers (bert) is pre-trained with masked language model (mlm). unlike the existing bert model, rare words rather than random words are masked and predicted in our model. then, in the fine-tuning step, the pretrained bert model learns with the training set of essays. our contributions are the followings: our experimental results show that all accuracies of the pre-trained bert model with rare tokens have been improved up to 14% in assessing creativity in essays compared to the existing bert model. the aes researches have been focused on generating hand-crafted features as an input of classification or regression @xcite @xcite @xcite @xcite . the linguistics features such as style and grammar are used @xcite @xcite . sometimes, we analyze the contents by latent semantic analysis @xcite @xcite @xcite @xcite @xcite , by @xcite and word embedding vectors @xcite , by using specific language features (wong and bong, 2019; @xcite and artificial neural networks @xcite @xcite . we also hybrid the style and content analysis to improve further @xcite @xcite @xcite @xcite . with the advent of deep learning technologies, aes have improved by using the pre-trained models with large data set @xcite @xcite @xcite @xcite @xcite @xcite . the lstm and rnn are naturally choices for aes task and some researchers applied bert for the task. bert based approaches @xcite @xcite shows an inferior performance than lstm @xcite in general. however, cao et al. (2020) and yang et al. (2020) are knowns to show a compatible performance to lstm based systems even with bert. other variations are song et al. (2020) used a transfer learning to overcome the size limitation of training data, wu et al. (2021) applied the r-drop to avoid the overfitting, and wang et al. (2022) used a transfer learning with multi-scale essay representations with bert. on the other hands, aes system has been studied as in a novelty or creativity detection perspective. liang et al. (2021) proposed a model to detect creative essay using a generative adversarial networks on the asap data. doboli et al. (2020) used a cognitive inspired approach to detect novel ideas on short text. chikkamath et al. (2020) applied the machine learning and deep learning approaches with various embedding vectors to find a new technology on patent data. bhattarai et al (2020) proposed a tsetline machine to detect a novel text using conjunctive clauses. nandi and basak (2020) proposed several cnn architectures to detect novel texts. beaty and johnson (2021) proposed an open platform to detect creativity based on semantic distances on word embeddings. amplayo et al. (2019) evaluated the academic research paper novelty detection using time and impact simulations. simpson et al. (2019) proposed bayesian approach to predict humor and metaphor score using gaussian process preference learning. christophe et al. (2021; 2020) proposed a framework to detect a new topic by monitoring the geometrical properties of word embeddings. ghosal et. al. proposed relative document vector based cnn model @xcite and a tap-dlind benchmark data sets @xcite . many researches have been done in detecting creative essays as mentioned in this section. however, as authors aware, this is the first attempt to use the low frequency words to detect creative essays. since the rare words are highly correlated with creative essays, we conjecture that the proposed approach will show a promising improvement. ## methodology the ultimate goal of our study is to understand how strongly a pre-trained encoder model like bert for aces is affected by rare tokens. to do this, given a large-sized set of text documents, we first extract a list of rare tokens using various approaches that we will discuss in detail in section 3.1. then, we will explain in detail our framework for comparative analysis of aces in section 3.2. finally, in section 3.3, we will design main questions for data-driven insights we want to know about through this study. ## experimental set-up for the experiment, we first implemented the five rare token extraction methods. we wrote python script codes to implement f 1 using bertokenizer of hugging face, f 2 and f 3 using scikit-learn 1.2.0, f 4 using spanbert base model (uncased) in py-torch, and f 5 using stanford, cambridge, and contextual rare word dictionaries. we also modified tensorflow code of bert base model (uncased) for implementing the rare token-based masked language model. a total of 4,079,432 documents from wikipedia were collected and text was extracted by removing html tags in each document. such refined text was used as input for bert's pre-training. to train rambert, we used the same default parameters as the bert base model, where we set 32 to batch size, 10 to epoch, 2e-5 to learning rate, and 0.1 to dropout rate. in addition, we used adam optimizer and we set 128 to maximum sequence length, 20 to maximum number of predictions per sequence, and 0.1 to masked language mode probability. to fine-tune rambert and perform the downstream task, we constructed a training set for creative essay assessment. first, we selected 800 essays at random from prompt 1 of the asap dataset (asap, 2022). the topic of the essays is how computers affect people. in the existing asap dataset, the essay score ranges from 2 to 12 points, and the higher the essay score, the better the writing, regardless of creativity. then, each essay was labelled as creative or non-creative by three domain experts who voted to classify each essay as either of creative or non-creative labels. all models were in standalone executed in a high-performance workstation server with intel xeon scalable silver 4414 2.20ghz cpu with 40 cores, 24gb ram, 1tb ssd, and geforce rtx 3080 ti d6 11gb blower with 4,352 cuda cores, 12gb ram, and 7gbps memory clock. ## conclusion in this paper, we proposed a method to detect creative essay writings by using a rambert (i.e. rare token masking-based bert). we used seven different rare token sets and pre-trained a bert after masking with the rare tokens on a large data. our preliminary experimental results show that rare tokens are highly correlated with the creativity essay scores. consequently, the rambert improved the accuracy up to 14% compared to a regular bert which is based on random word masking. the performance improvements are also shown in rogue and blue scores.