Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability (2021)

Abstract: In this paper we implement and compare 7 different data augmentation strategies for the task of automatic scoring of children’s ability to understand others’ thoughts, feelings, and desires (or “mindreading”). We recruit in-domain experts to re-annotate augmented samples and determine to what extent each strategy preserves the original rating. We also carry out multiple experiments to measure how much each augmentation strategy improves the performance of automatic scoring systems. To determine the capabilities of automatic systems to generalize to unseen data, we create UK-MIND-20 - a new corpus of children’s performance on tests of mindreading, consisting of 10,320 question-answer pairs. We obtain a new state-of-the-art performance on the MIND-CA corpus, improving macro-F1-score by 6 points. Results indicate that both the number of training examples and the quality of the augmentation strategies affect the performance of the systems. The task-specific augmentations generally outperform task-agnostic augmentations. Automatic augmentations based on vectors (GloVe, FastText) perform the worst. We find that systems trained on MIND-CA generalize well to UK-MIND-20. We demonstrate that data augmentation strategies also improve the performance on unseen data.

## introduction many state-of-the-art nlp models are limited by the availability of high quality human-annotated training data. the process of gathering and annotating additional data is often expensive and time consuming. it is especially difficult to gather data for tasks within psychology and psycholinguistics, as test administration typically requires highly trained in-domain experts, controlled environments, and large numbers of human participants. data augmentation is a popular technique for artificially enlarging datasets. typically, data augmentation uses one or more predefined strategies to modify existing gold-standard examples while retaining the original label. the objectives of data augmentation are: 1) to increase the size of the dataset; 2) to introduce more variety; 3) to reduce overfitting; and 4) to improve generalizability. data augmentation has been used successfully in computer vision @xcite and has recently become more popular in the field of nlp @xcite @xcite . we use data augmentation to improve the performance of systems for automatic scoring of children's performance on tests of mindreading (i.e., the ability to reason about others' thoughts, feelings and desires) @xcite . automatic scoring of mindreading was recently introduced by @xcite . their corpus, mind-ca contains hand-scored data from more than 1000 children aged 7 to 14. collecting data on children's mindreading performance is complicated, time-consuming, and expensive. it requires in-person testing sessions led by trained researchers and children's open-ended responses must be rated by trained annotators. data augmentation could be very beneficial to improve the performance and consistency of the automated scoring systems. in this paper we aim to measure, in a systematic way, the quality and efficiency of the different augmentation strategies. we evaluate and compare the different strategies intrinsically and extrinsically. for the intrinsic evaluation, we recruit in-domain experts to re-annotate augmented examples and determine the extent to which each strategy preserves the original label. for the extrinsic evaluation, we measure the quantitative improvement (macro-f1, f1-per-question, standard deviation) of automatic systems on the mind-ca corpus. furthermore, we create a new corpus, uk-mind-20, containing 10,320 question-answer pairs in english and we use it to evaluate the performance of automated systems on unseen data. we find that the intrinsic "quality" of the augmentation strategies varies significantly, according to human raters. however, the extrinsic evaluation demonstrates that all strategies improve the performance of the automated systems. we systematically measure the importance of three factors in data augmentation: corpus size, sampling strategy, and augmentation strategy. we find corpus size to be the most important factor. however, the choice of sampling and augmentation strategies also significantly affects the performance of the automated systems. we report a correlation between the "quality" of the augmentation and the performance. with the best configuration we obtain a new state-of-theart on mind-ca, improving macro-f1 score by 6 points and f1-per-question by 10.3 points. we demonstrate that the automated scoring systems can generalize well between mind-ca and uk-mind-20. these findings indicate that the methodology for administering and scoring mindreading is consistent and the automatic solutions can be adopted in practice. the rest of this article is organized as follows. section 2 discusses the related work. section 3 presents the methodologies for data augmentation. section 4 compares the quality of the augmentation strategies. section 5 describes the machine learning experimental setup and evaluation criteria. section 6 analyzes the effect of data augmentation on automated systems. section 7 presents some follow-up experiments and discusses the implications of the findings. section 8 concludes the article and proposes directions for future work. ## related work mindreading (also known as "theory of mind") is the ability to understand others' thoughts, feelings, and desires @xcite . for example, in the final scene of romeo and juliet, romeo holds a mistaken belief that juliet is dead. being able to understand the state of the world ("juliet is alive") and the mistaken belief ("juliet is dead") is important to understand the situation and the motivation of the characters. individual differences in children's mindreading are linked with both social and academic outcomes and children's wellbeing @xcite @xcite . furthermore, difficulties with mindreading are linked with a range of mental health problems and neurodevelopmental conditions @xcite . the task of automatic scoring of mindreading was first proposed by @xcite . they gathered the responses of 1066 children aged 7-14 on two standardized tests of mindreading: the strange story task @xcite and the silent film task @xcite . after digitalizing and manually scoring the responses, they created mind-ca, a corpus of 11,311 questionanswer pairs. they trained and evaluated several automated systems (i.e., svm, bilstm, transformer) and obtained promising initial results. data augmentation is a technique for artificially increasing the size of the dataset. it can also be seen as a type of regularization at the level of the data. data augmentation can be used to increase the number of instances of specific answer types. it can also introduce more variety, and can reduce the imbalance between classes. data augmentation is used to improve the performance of automated systems, to reduce the risk of overfitting, and to enhance the ability of automated systems to generalize to unseen data. it is widely used in computer vision @xcite . the specifics of natural languages make it more difficult to incorporate data augmentation in nlp. a subtle change to the text can often lead to a substantial difference in meaning and a change of the label. the last two years have seen an increase in the popularity of data augmentation in nlp. @xcite present a python library that uses simple augmentation methods for improving text classification. @xcite compare different strategies for augmentation in the context of short-text classification. @xcite compare different data augmentation strategies for the task of named entity recognition. several researchers propose more complex augmentation strategies for nlp. @xcite propose a sequence-to-sequence model for data augmentation. @xcite and @xcite use language models in what they call "contextual augmentation". min et al. (2020) use syntactic augmentation to improve the performance and generalizability on natural language inference. in this paper, we take a different approach towards data augmentation. we implement and compare seven different augmentation strategies. two of the strategies were designed specifically for the task of automatic scoring of mindreading, while the remaining five are task agnostic. we put the emphasis on a systematic evaluation of the augmentation strategies and some key parameters of the augmentation process. we recruit and train in-domain experts to provide intrinsic human evaluation of the data augmentation. we also annotate a new corpus that can measure the performance and improvement on unseen data. ## data augmentation we used 7 different strategies for automatic data augmentation. "dictionary" and "phrase" strategies make use of task-specific resources, created by in-domain experts. the other 5 strategies ("order", "wordnet", "ppdb", "glove", "fasttext") make use of publicly available task-agnostic resources. for a source of the augmentation, we used the mind-ca corpus @xcite . it contains 11,311 question-answer pairs. there are 11 different questions, and an average of 1,028 responses per question. there are three possible labels reflecting the degree to which the response shows context-appropriate mindreading: 0 (fail), 1 (partial score), and 2 (pass). the label distribution for the full corpus is balanced, however the label distribution for the individual questions vary foot_0 . we sought to use data augmentation to create a well-balanced dataset in terms of questions and labels. to achieve this, we created a policy for sampling examples that we used in the augmentation. we split the mind-ca corpus per question and per label, resulting in 33 question-label sub corpora. the average size of each sub-corpora is 343, and the smallest number of instances in a sub corpora is 160. we sampled 125 examples from each question-label sub-corpus, 375 from each question, for a total 4,125 examples. our sampling strategy ensures that each question-label combination is well represented in the augmentation process. in the original mind-ca corpus, nine question-label pairs had less than 125 instances. as a preliminary step in the data augmentation process, our in-domain experts rewrote existing responses to improve the balance of the corpus. we used strategy similar to the one used in @xcite . we ran statistical and machine learning experiments to ensure that the additional examples do not introduce biases. for our experiments we initially chose a conservative number of examples (each augmentation increases the original corpus size by 36 %), to avoid overfitting on the underrepresented question-label pairs. we used a different random state for each augmentation strategy and we ensured that each sample is representative in terms of demographic distribution (age and gender of the participants). in a complementary set of experiments, we applied data augmentation directly without the custom sampling strategy. we also experimented with generating larger number of augmented examples (up to 140% of the original corpus size) via oversampling (see section 7). in the following subsections, we discuss in more details the different augmentation strategies. ## measuring augmentation quality the quality of data augmentation models in nlp research is typically evaluated extrinsically, by measuring the performance of automated systems trained on augmented data. @xcite propose an intrinsic evaluation inspired by the data augmentation research in computer vision. they compare the latent space representations of the original and the augmented sentences and assume that the proximity in latent space indicates that the original labels are conserved. we argue that a direct comparison of the representation of the texts is not sufficient to determine the quality of the augmentation and the extent to which each strategy preserves the original labels. in natural language, unlike in computer vision, a minor difference in the text and the corresponding representation can cause a significant difference in the meaning of the complex expression and ultimately the label or score assigned to that answer. we propose a manual evaluation of the different strategies. for each augmentation strategy, we selected 5 random examples from each questionlabel sub-corpus, adding up to 165 examples per strategy (4% of the full sample). two trained annotators independently rate the augmented pairs for the 7 different augmentation strategies (a total of 1,155 question-answer pairs). to ensure a fair evaluation, the annotators receive a single file with the examples for all augmented strategies shuffled at random. the inter-annotator agreement was 87% with a cohen's kappa of .83. ## evaluating automated systems in our experiments, we used the two best systems reported by @xcite -a bil-stm neural network and a distilbert transformer. these systems obtained good results on the original mind-ca corpus and at the same time were lightweight enough to be implemented in a practical end-to-end application for automatic scoring. we used the same configuration and hyperparameters as reported by @xcite . we modified the existing classes to incorporate and keep track of data augmentation and to implement additional evaluation on uk-mind-20. all of our code and data are available online foot_2 . ## results the distilbert architecture is the best performing system from @xcite . the baseline system, trained on the original data already obtained very good results: .925 f1 and .877 f1per-q on the mind-ca corpus and .889 f1 and .839 f1-per-q on the uk-mind-20 corpus. we demonstrate that systems trained on either of the two datasets can generalize well on the other one (f1 of .89 and f1-per-q of .84). this indicates that the two corpora are compatible and that automated systems can generalize to unseen data. it is evident in the table that all of the augmentation strategies successfully improved the performance of the automated systems across all evaluation criteria. for the mind-ca corpus: f1 improved between 1.7 points (fasttext) and 6 points (all-hq); f1-per-qiestion improved between 4.7 points (fasttext) and 10.3 points (all-hq); stdper-question was reduced by between 1.6 points (wordnet) and 4.8 points (all-hq). for the uk-mind-20 corpus: f1 improved between 0.1 point (fasttext) and 0.9 point (all-hq); f1-per-question improved between 1 point (glove) and 1.9 points (all-hq); std-per-question was reduced between 3.9 points (dictionary) and 4.2 points (ab-hq). based on these results, we can draw two conclusions. first, data augmentation can successfully be used to improve the performance of the systems on the mind-ca corpus. second, data augmentation also improves the performance of the automated systems on the unseen examples from uk-mind-20. while the improvement is not as substantial as seen on mind-ca, the improvement on all three criteria on uk-mind-20 indicates that the systems are not just overfitting to mind-ca. we use the autorank python library @xcite to carry out a statistical analysis on the results and compare the performance gain from each of the augmentation strategies. we use the data from both algorithms and input formats, a total of 480 machine learning models, 40 for each dataset. based on the provided data, autorank determines that the most appropriate statistical test is the friedman-nemeyni test @xcite . the friedman test reports that there is a statistically significant difference between the median values of the populations. that means that some training sets are consistently performing better (or worse) than others. the post-hoc nemenyi test can be used to determine and visualise which training sets are better and which are worse. figure 1 shows the critical difference diagram of the post-hoc nemenyi test for all training sets. each set is plotted with its average ranking across all systems. the difference between systems connected with a line is not statistically significant. the original corpus is the worst performing of all datasets with an average rank of 9. the 7 "basic" training sets are grouped in the middle (rank 6.5 to 8). that is, they are all better than the original corpus, but worse than the combined training sets. there is a significant difference between "all-hq", "all-lq", "ab-hq", and "ab-lq". collectively they are also better than the original training set and the "basic" training sets. figure 2 shows the critical difference diagram of the post-hoc nemenyi test applied only to the 7 "basic" augmentations. after removing the outliers (the original corpus and the collections of multiple augmentation), we can observe a clear, statistically significant distinction between "high quality" augmentations ("dictionary", "phrase", and "order") and "low quality" augmentations ("glove", "fasttext", "wordnet", and "ppdb"). based on the statistical analysis, we can draw two additional conclusions. third, we found that the most important factor affecting the system performance is the number of training examples. we obtain the best results by combining the examples from various different augmentation strategies. fourth, we demonstrated that when the training size is comparable, the high quality augmentations improve the performance more than the low quality ones. the difference is significant and is consistent both in "basic" datasets and in "combined" datasets. vector based augmentations (glove and fasttext) are performing worse than augmentations based on task-specific or task-agnostic knowledge bases. ## discussion and further experiments the intrinsic and extrinsic evaluation presented in section 4 and section 6 answered the main research questions posed in this paper. we demonstrated that data augmentation can improve the performance of automated systems including on novel, unseen data. we found that the data augmentation strategies vary in preserving the original label and in how much they improve the machine learning systems trained on them. we also showed that automated scoring systems can generalize well from mind-ca corpus to uk-mind-20 and the other way around. all these findings are important for further research on mindreading. at the same time, our data augmentation strategies and evaluation methodology can also be extended to other tasks and domains, contributing to the research of data augmentation in nlp in general. in this section we present additional experiments and an analysis of the impact of several different factors in the process of data augmentation foot_4 . corpus size our experiments indicated that the most important factor for improving the system performance is the corpus size. in as expected, the performance of each system increased with corpus size. the ranking of the individual systems remained similar to the one reported with 125 base examples. "high quality" augmentations still performed better than "low quality" ones. the f1, f1-per-q, and std-per-q for the "basic low quality" strategies was approximately the same as the performance for ab-lq. the f1, f1-per-q, and std-per-q for the "basic high quality" strategies was approximately the same as the performance for ab-hq. this new set of experiments confirmed the importance of corpus size. even strategies that human experts perceive as "low quality" are improving the performance of the automated systems. and while the ranking consistently favors the "high quality" augmentations, the absolute difference is relatively small. this is in line with the findings on noisy learning which show that machine learning models can be very noise-tolerant @xcite . we performed one final experiment by combining the all-lq and all-hq data together, but found no increase or decrease of performance compared with using only the all-hq data. ## conclusions and future work we presented a systematic comparison of multiple data augmentation strategies for the task of automatic scoring of children's mindreading ability. we argued that the nature of natural language requires a more in-depth analysis of the quality and performance of the different data augmentation strategies. we recruited in-domain experts and incorporated them in the process of evaluation. we demonstrated that, for some of the augmentation strategies ("glove", "fasttext", "ppdb") there is a substantial portion of the examples (over 20%) where the rating changes or cannot be assigned due to semantically incoherent text. these differences in the datasets cannot be captured trivially via the visualisation techniques that are typically used for intrinsic evaluation. we also found that the difference in augmentation quality corresponds to a difference in the performance of automated systems trained on the data. to the best of our knowledge, this is the first evaluation of data augmentation in nlp that involves both expert evaluation and automatic metrics and the first study that demonstrates the connection between the two. we carried out further experiments measuring the importance of factors such as corpus size and sampling strategy. our findings on the quality and efficiency of data augmentation strategies and on the use of task-specific resources are relevant for researchers in the area of data augmentation, specifically in domains where the quality of the training gold examples is important or where the amount of data is very limited. for the purpose of evaluation, we also created a new corpus: uk-mind-20. it is the second corpus for automatic scoring of mind reading in children. we demonstrated that systems trained on mind-ca generalize well on uk-mind-20. we also showed that data augmentation improves the performance on unseen data. these findings are promising both for the task of scoring children's mindreading and for the use of data augmentation in nlp. to the best of our knowledge, this is the first work where augmentation is evaluated on novel, unseen data for the same task. this work opens several directions of future work. as a direct continuation of this research, we will incorporate the best performing automated systems and data augmentation techniques in the work of developmental psychologists. this will facilitate a large-scale studies on mindreading in children and adolescents. we are also exploring the possibility of using nlp to address other time and labour intensive problems within psychology. open-ended short text responses are widely-used within psychological research and the good results obtained in this paper can be replicated in other similar tasks.