Feeding What You Need by Understanding What You Learned (2022)

Abstract: Machine Reading Comprehension (MRC) reveals the ability to understand a given text passage and answer questions based on it. Existing research works in MRC rely heavily on large-size models and corpus to improve the performance evaluated by metrics such as Exact Match ( EM ) and F_1 . However, such a paradigm lacks sufficient interpretation to model capability and can not efficiently train a model with a large corpus. In this paper, we argue that a deep understanding of model capabilities and data properties can help us feed a model with appropriate training data based on its learning status. Specifically, we design an MRC capability assessment framework that assesses model capabilities in an explainable and multi-dimensional manner. Based on it, we further uncover and disentangle the connections between various data properties and model performance. Finally, to verify the effectiveness of the proposed MRC capability assessment framework, we incorporate it into a curriculum learning pipeline and devise a Capability Boundary Breakthrough Curriculum (CBBC) strategy, which performs a model capability-based training to maximize the data value and improve training efficiency. Extensive experiments demonstrate that our approach significantly improves performance, achieving up to an 11.22% / 8.71% improvement of EM / F_1 on MRC tasks.

## introduction a competency assessment is used to measure someone's capabilities against the requirements of their job @xcite . in other words, it measures how (behaviors) someone does the what (task or skill). by showing what it looks like to be good in a job, a competency assessment can effectively empower and engage people who want to understand and improve their unique skill profile and tell them what action to take to close any gaps so they can own their development. a natural question that arises here is: can we develop competency assessments for machine learning models to help better understand their capabilities and improve their performance on a given task? in this paper, we focus on competency assessments for machine reading comprehension (mrc). mrc is a core task in natural language processing (nlp) that aims to teach machines to understand human languages and answer questions @xcite . recently, pretrained language models (lms) @xcite @xcite have demonstrated superior performance on mrc tasks by pre-training on large amounts of unlabeled corpus and fine-tuning on mrc datasets. the performance is usually evaluated by metrics such as exact match (em ) and f 1 score, lacking interpretability to the capabilities of a model. that is to say, such metrics only tell how good a model performs overall on a specific dataset, but uncovers little about what specific skills a model has gained and the level of each skill. we argue that the value of each data sample varies during the training process of a model, depending on the model's current capabilities. a deep understanding of the model's intrinsic capabilities can help us estimate each data sample's learning value and better manage the training process to improve the training efficiency. take student learning as an example. there is no doubt that a college student can do well in solving primary school level exercises, but such exercises do not help improve his/her ability. on the contrary, a primary school student can not acquire knowledge efficiently from college-level exercises due to the big gap between his/her current knowledge or skills and the require- in this paper, we design a competency assessment framework for mrc model capabilities. specifically, we define four major capability dimensions for understanding text and solving mrc tasks: reading words, reading sentences, understanding words and understanding sentences, which are inspired by the computational models of human text comprehension in psychology @xcite . based on the proposed framework, we can obtain a more appropriate assessment of model capabilities than the regular em or f 1 metrics. furthermore, we analyze a variety of data properties to estimate how good a model has to be to solve a specific data sample and identify the relationships between data properties and model performance. this greatly helps us estimate the learning value of each training sample. based on this analysis, we discover a very common situation: if a sample is scored as a high value in one capability dimension, the other dimensions have the same tendency as well, and vice versa. to alleviate these inevitable correlations, we utilize data whitening to quantify each sample as four capability-specific scores in a decorrelated fashion. finally, to reveal the potential usefulness of our proposed competency assessment framework and evaluate its efficiency, we employ it in a curriculum learning pipeline and design a capability boundary breakthrough curriculum (cbbc) strategy. this strategy gradually enlarges the model capability boundary by picking samples around the boundary and breaking through it. based on the analysis of model capabilities and data properties, we feed the model with training samples that are neither too simple nor too hard for it to solve. extensive experiments on four benchmark datasets demonstrate that our approach significantly improves the performance of existing mrc models, achieving up to an 11.22% / 8.71% improvement of em / f 1 on mrc tasks. these results show the reasonability and effectiveness of our proposed assessment framework and provide a widely applicable measurement for dealing with the relationship between the model capability and data quality. ## competency assessment of mrc capabilities in this section, we first formulate our competency assessment framework of 4-dimensional mrc capabilities. based on this framework, the data properties related to each capability dimension are described as corresponding heuristic metrics. we then uncover the relationship between various data properties and model performance in a decorrelated manner, quantifying each sample as 4-dimensional capability-specific scores with little correlation. ## improve learning efficiency with competency assessment in this section, our universal assessment framework of the model capability is adapted into a specific mrc training scenario to evaluate its usefulness and efficiency. specifically, we embed our proposed assessment framework into a curriculum learning pipeline and make a capability boundary breakthrough curriculum learning (cbbc) strategy. based on the assessment framework, our cbbc can guide a model to learn according to its capability boundary by understanding what the model has learned from data (i.e. capability-specific value v i ) and choosing appropriate samples with comparable learning values from four dimensions. it is worth noting that our competency assessment framework is also applicable to other training pipelines that balance the relationship between the model capabilities and data properties, such as active learning (settles, 2009) and self-training @xcite (appendix e). figure 4 shows an illustration of the pipeline of our cbbc. following the original formulation of curriculum learning @xcite , our cbbc organizes all samples by a sequence of ordered training stages {s} s @xmath0. the classic curriculum learning works @xcite usually consist of two essential components: the performance measurer and the curriculum scheduler. in general, the measurer is used to determine the learning status of a model by evaluating performance, while the scheduler is responsible for deciding when and how to update the curriculum by selecting the input samples. in our work, the measurer and scheduler are implemented by analyzing the multi-dimensional capability levels of the model interpretably and measuring the capability-specific values of the data in a decorrelated way, respectively. that is to say, the only difference between our cbbc and the original curriculum learning design is incorporating mrc capability assessment into the curriculum learning. without significantly increasing the complexity of the pipeline, our proposed assessment framework can generally empower the mrc training pipeline in a plug-and-play manner. performance measurer. recall what we have discussed in section 2.2 that the samples with high v i can be used in indicative measurements to the corresponding model capability c i . in this work, we use samples scored in the top-k of each capabilityspecific value to assess the corresponding model capability. more precisely, we first evaluate the model on the dev set and obtain an average f logits for each capability on the corresponding top-k subset. then partial correlation @xcite (denoted as ρ i ) between dimension v i and f logits is computed to mask the contributions of the other dimensions v\{v i }. after that, each model capability on stage s is quantified as: empirically, we set k in top-k as 32. curriculum scheduler. following the most works @xcite , we schedule the curriculum at a linear pace (every 1,000 training iterations). during each curriculum schedule, we enlarge the training set two times until it includes all the samples. the capability upper bound c s+1 i for s + 1 stage by exponential growth: after that, we use criterion v i (x) < c s+1 i to construct candidate set d s+1 i for the i-th capability on the state s + 1, and use absolute contribution of v i to f logits as sampling ratio (i.e. ρ 1 : ρ 2 : ρ 3 : ρ 4 ) to construct d s+1 . ## experiments datasets. we employ two question styles to evaluate our cbbc: answer span extraction and multiple choice. the former consists of squadv1 @xcite , squadv2 @xcite and hotpotqa @xcite , while the latter adopts race @xcite . for each dataset, we train and evaluate the model on official training and dev split, respectively. implementation details. the source code and hyperparameters are included in the supplementary material. we use bert-base @xcite as our backbone model, which is initialized by pre-trained parameters from cased bert. adamw @xcite optimizer with weight decay 5e -4 and epsilon 8 is used to finetune the model with max sequence length 384, document stride 128. the learning rate warms up over the first 10% steps and then decays linearly to 0 for all experiments with training batch size 16 and maximum iteration 40, 000. baseline models. in addition to the bertbase model, we also consider the following ten baselines. the first two baselines are trained through a pre-defined curriculum learning strategy, which sorts the samples, then feeds them to the model stage-by-stage. "b+cl+v (m 2 )" sorts the samples by four capability-specific scores in an easy-to-difficult order. "b+anticl+v (m 3 )" does like "m 2 ", but in a reverse difficult-to-easy order. the following five baselines are trained using our cbbc strategy to maximize the data value in each dimension, respectively. "b+c+v 1 (m 4 )", "b+c+v 2 (m 5 )", "b+c+v 3 (m 6 )" and "b+c+v 4 (m 7 )" use the corresponding v 1 , v 2 , v 3 and v 4 respectively to perform the competency test and filter samples. "b+c+v corr (m 8 )" is trained using four correlated scores through cbbc. the following three baselines are devised by embedding other instance scoring methods into our cbbc pipeline. "b+c+datasetmap (m 9 )", "b+c+forgetting (m 10 )" and "b+c+predictability (m 11 )" substitute the capability-specific scores with the confidence score of true answer span @xcite , number of "forgotten" events @xcite and predictability score @xcite , respectively. the last two baselines (denoted as m 12 and m 13 ) are start-of-the-art curriculum learning pipelines consisting of drca @xcite and cbcl @xcite . finally, our full model is trained using four decorrelated scores through cbbc instead. the critical difference between the full model and m 8 is the decorrelation operation. ## conclusion we design a competency assessment framework for mrc capabilities, which describes model skills in an explainable and multi-dimensional manner. by leveraging the framework, we further uncover and disentangle the connections between various data properties and model performance on a specific task, as well as propose a capability boundary breakthrough curriculum (cbbc) strategy to maxi-mize the data value and improve training efficiency. the experiments performed on four benchmark datasets verified that our approach can significantly improve the performance of existing mrc models. our work shows a deep understanding of model capabilities and data properties helps monitor the model skills during training and improves learning efficiency. our framework and learning strategy are also generally applicable to other nlp tasks.