Characterizing Text Datasets with Psycholinguistic Features (2024)

Abstract: Fine-tuning pretrained language models on task-specific data is a common practice in Natural Language Processing (NLP) applications. However, the number of pretrained models available to choose from can be very large, and it remains unclear how to select the optimal model without spending considerable amounts of computational resources, especially for the text domain. To address this problem, we introduce PsyMatrix, a novel framework designed to efficiently characterize text datasets. PsyMatrix evaluates multiple dimensions of text and discourse, producing interpretable, low-dimensional embeddings. Our framework has been tested using a meta-dataset repository that includes the performance of 24 pretrained large language models fine-tuned across 146 classification datasets. Using the proposed embeddings, we successfully developed a meta-learning system capable of recommending the most effective pretrained models (optimal and near-optimal) for fine-tuning on new datasets.

## introduction pre-training a language model on a large, diverse, and unlabeled corpus, then fine-tuning it with task-specific data has proven to be highly effective for enhancing performance of natural language processing (nlp) applications @xcite . since then, numerous pretrained large language models (llm) have been released to the public, each pretrained on different corpora with varying sizes, and using different architectures. although the prospect of a universal pretrained model (ptm) that excels across all nlp tasks is attractive, evidence (still) indicates that no single model performs optimally in every scenario @xcite @xcite . this poses a challenge for machine learning practitioners, who must select a ptm to fine-tune for a task-specific dataset. performing an exhaustive search over all possible candidates can be very time and resource consuming. in practice, this is not realistic. the performance of ptms often fluctuates depending on some characteristics of the target dataset @xcite , usually referred to as meta-features @xcite . for example, a model that performs exceptionally well on carefully written news articles may encounter difficulties with the brevity and slang commonly found in social media posts @xcite @xcite . therefore, the challenge is to decide, for a particular dataset, which ptm is expected to perform the best after fine-tuning. one low-cost approach could be to search for public benchmarks and check which ptm performed the best for similar datasets. however, defining such dataset similarity in an objective manner, specially for text datasets, is very challenging. to address this, we propose a novel framework named psymatrix, which characterizes text datasets by analyzing multiple aspects of text and discourse. these range from simple part-of-speech (pos) statistics to more complex and deep psycholinguistic features @xcite . our framework generate dataset embeddings that are both interpretable and low-dimensional, providing a deeper understanding of the intricacies inherent in different datasets. figure 1 illustrates the core components of the framework. the code of the framework is open-source and has been made publicly available 1 . to sum up, our contributions are as follows: the paper is organized as follows: section 2 reviews existing methods for characterizing datasets in general and for the problem of selecting a pretrained model for fine-tuning on task-specific data. section 3 provides a detailed description of the framework, including the psycholinguistic dimensions and complexity measures used. section 4 presents and analyzes the findings from fine-tuning models on 146 datasets and the predictive performance of the provided embeddings, presenting practical applications of psymatrix in guiding model selection and its broader implications in machine learning. finally, section 5 summarizes the findings, contributions, and potential future work. ## related work the process of selecting an optimal ptm for a specific dataset typically involves trial and error. practitioners usually start by choosing a few potential models based on prior knowledge and experience @xcite @xcite @xcite . these models are then fine-tuned on the target dataset and have their performance compared, allowing for an assessment of how well each model adapts to the unique characteristics of the data. however, the result of this approach can be limited by the assumptions made during initial model pre-selection, which may not always be valid for the dataset in question, which might lead to suboptimal outcomes. this mismatch underscores the need for a refined selection strategy that takes into account the unique features of the dataset, promoting a more effective model selection process. ## the psymatrix framework the psymatrix framework, as depicted in figure 1 , is designed to synthesize text datasets into a manageable form (i.e., dataset embeddings) using a series of feature extractors and dimensionality reduction techniques. these tools combined, not only capture the essence of the documents, but also facilitate a deeper understanding of underlying patterns and relationships. this way, the psymatrix framework aims to provide dataset embeddings that can be employed as performance predictors for different downstream tasks. this section details the components of the psymatrix framework, illustrating how each module contributes to the overall goal. ## experiments and results this section experimentally investigates the practical application of the psymatrix framework to predict the final performance y ij of a model i on a dataset j using a surrogate function η i (ϕ(ψ(d j ))). the performance metric employed is the negative logarithm of the cross-entropy on the testing sets. the objective is to test the ability of psymatrix in predicting the most suitable model for any target dataset. to do this, we evaluated the performance of various pretrained language models across a broad spectrum of text classification tasks. specifically, we used 90% of the data for training, and the remaining 10% was employed for validation. to further enhance the reliability of our results, we employed 10-fold cross-validation, ensuring a more robust assessment of psymatrix's generalization ability across various datasets and reducing the impact of data variability. ## conclusion in this paper, we introduced psymatrix, a novel framework for characterizing text-based datasets through psycholinguistic dimensions, topic distributions, and complexity measures. by providing interpretable and low-dimensional dataset embeddings, psymatrix aids in understanding the latent dataset features and accurately predicting the performance of pretrained language models finetuned for classification tasks. our extensive validation across 146 text classification datasets demonstrated the robustness and effectiveness of psymatrix in guiding model selection. in future work we will explore the application of psymatrix to other domains beyond text classification and expand psymatrix to support multiple languages.