Ling- CL : Understanding NLP Models through Linguistic Curricula (2023)

Abstract: We employ a characterization of linguistic complexity from psycholinguistic and language acquisition research to develop data-driven curricula to understand the underlying linguistic knowledge that models learn to address NLP tasks. The novelty of our approach is in the development of linguistic curricula derived from data, existing knowledge about linguistic complexity, and model behavior during training. Through the evaluation of several benchmark NLP datasets, our curriculum learning approaches identify sets of linguistic metrics (indices) that inform the challenges and reasoning required to address each task. Our work will inform future research in all NLP areas, allowing linguistic complexity to be considered early in the research and development process. In addition, our work prompts an examination of gold standards and fair evaluation in NLP.

## introduction linguists devised effective approaches to determine the linguistic complexity of text data @xcite @xcite . there is a spectrum of linguistic complexity indices for english, ranging from lexical diversity @xcite to word sophistication @xcite ) to higher-level metrics such as readability, coherence, and information entropy @xcite . these indices have not been fully leveraged in nlp. we investigate the explicit incorporation of linguistic complexity of text data into the training process of nlp models, aiming to uncover the linguistic knowledge that models learn to address nlp tasks. figure 1 shows data distribution and accuracy trend of roberta-large @xcite against the linguistic complexity index "verb variation" (ratio of distinct verbs). this analysis is conducted on anli @xcite validation data. samples with greater verb variation are more complex and also harder for the model to classify. such linguistic indices can inform difficulty estimation and linguistic curriculum development for nlp tasks. for individual bins separately. the accuracy trend indicates that verb variation can describe the difficulty of anli samples to the model. in addition, the data distribution illustrates potential linguistic disparity in anli; see @xcite to reveal the linguistic knowledge nlp models learn during their training, we will employ known linguistic complexity indices to build multiview linguistic curricula for nlp tasks. a curriculum is a training paradigm that schedules data samples in a meaningful order for iterative training, e.g., by starting with easier samples and gradually introducing more difficult ones @xcite . effective curricula improve learning in humans @xcite and machines @xcite @xcite . curriculum learning has been found effective in many nlp tasks @xcite @xcite @xcite @xcite @xcite @xcite . a multiview curriculum is a curriculum able to integrate multiple difficulty scores simultaneously and leverage their collective value @xcite . we assume there exists a subset of linguistic complexity indices that are most influential to learning an nlp task by a particular model. to identify these indices for each model and nlp task, we derive a weight factor ρ @xmath0, 1] for each linguistic index that quantifies how well the index estimates the true difficulty of data samples to the model, determined by model instantaneous loss against validation data. by learning these weight factors, we obtain precise estimations that shed light on the core linguistic complexity indices that each model needs at different stages of its training to learn an nlp task. in addition, these estimates can be readily used for linguistic curriculum development, e.g., by training models with linguistically easy samples (with respect to the model) and gradually introducing linguistically challenging samples. to achieve the above goals, we should address two gaps in the existing literature: first, existing curricula are often limited to a single criterion of difficulty and are not applicable to multiview settings. this is while difficulty is a condition that can be realized from multiple perspectives, can vary across a continuum for different models, and can dynamically change as the model improves. second, existing approaches quantify the difficulty of data based on instantaneous training loss. however, training loss provides noisy estimates of sample difficulty due to data memorization @xcite in neural models. we will address both issues as part of this research. the contributions of this paper are: we evaluate our approach on several nlp tasks that require significant linguistic knowledge and reasoning to be addressed. experimental results show that our approach can uncover latent linguistic knowledge that is most important for addressing nlp tasks. in addition, our approach obtains consistent performance gain over competing models. source code and data is available at https://github.com/clu-uml/ling-cl . algorithm 1 correlation method require: dtrain, d val , model θ, optimizer g, loss function f , curriculum c 1: step ← 0 2: ρ ← random initialization 3: while @xmath1: training_batch ← samplebatch(step, dtrain) 5: loss ← computeloss (training_batch, θ) 6: ling ← getlinguisticfeatures(training_batch) 7: difficulty ← calculatedifficulty(ling, ρ) 8: confidence ← determineconfidence(step, difficulty) 9: weighted_loss ← loss ⊗ confidence 10: θ ← updatemodel (weighted_loss, θ) 11: if step % @xmath2: l ← computeloss (d val , θ) 13: ling ← getlinguisticfeatures(d val ) 14: for ρi ∈ ρ do 15: ρi ← pearsonr(l, ling[:, i]) 16: end for 17: end if 18: step ← step + 1 19: end while 2 multiview linguistic curricula we present a framework for multiview curriculum learning using linguistic complexity indices. our framework estimates the importance of various linguistic complexity indices, aggregates the resulting importance scores to determine the difficulty of samples for learning nlp tasks, and develops novel curricula for training models using complexity indices. the list of all indices used is in appendix a. ## conclusion and future work we propose a new approach to linguistic curriculum learning. our approach estimates the importance of multiple linguistic indices and aggregates them, provides effective difficulty estimates through correlation and optimization methods, and introduces novel curricula for using difficulty estimates, to uncover the underlying linguistic knowledge that nlp models learn during training. furthermore, we present a method for a more accurate and fair evaluation of computational models for nlp tasks according to linguistic indices. furthermore, the estimated importance factors present insights about each dataset and nlp task, the linguistic challenges contained within each task, and the factors that most contribute to model performance on the task. further analysis of such learning dynamics for each nlp task will shed light on the linguistic capabilities of computational models at different stages of their training. our framework and the corresponding tools serve as a guide for assessing linguistic complexity for various nlp tasks and uncover the learning dynamics of the corresponding nlp models during training. while we conducted our analysis on seven tasks and extracted insights on the key indices for each task, nlp researchers have the flexibility to either build on our results or apply our approach to other nlp tasks to extract relevant insights. promising areas for future work include investigations on deriving optimal linguistic curriculum tailored for each nlp task; examining and enhancing linguistic capabilities of different computational models, particularly with respect to linguistically complex inputs; and developing challenge datasets that carry a fair distribution of linguistically complex examples for various nlp tasks. in addition, future work could study why specific indices are important, how they connect to the linguistic challenges of each task, and how different linguistic indices jointly contribute to learning a target task. we expect other aggregation functions, such as log-average, exponential-average, and probabilistic selection of maximum, to be effective approaches for difficulty estimation based on validation loss. finally, other variations of the proposed gaussian curriculum could be investigated for model improvement.