Context Minimization for Resource-Constrained Text Classification: Optimizing Performance-Efficiency Trade-offs through Linguistic Features (2025)

Abstract: Pretrained language models have transformed text classification, yet their computational demands often render them impractical for resource-constrained settings. We propose a linguistically-grounded framework for context minimization that leverages theme-rheme structure to preserve critical classification signals while reducing input complexity. Our approach integrates positional, syntactic, semantic, and statistical features, guided by functional linguistics, to identify optimal low-context configurations. We present a methodical iterative feature exploration protocol across 6 benchmarks, including our novel CMLA11 dataset. Results demonstrate substantial efficiency gains: 69-75% reduction in GPU memory, 81-87% decrease in training time, and 82-88% faster inference. Despite these resource savings, our configurations maintain near-parity with full-length inputs, with F1 (macro) reductions averaging just 1.39-3.10%. Statistical significance testing confirms minimal practical impact, with some configurations outperforming the baseline. SHAP analysis reveals specific feature subsets contribute most significantly across datasets, and these recurring configurations offer transferable insights, reducing the need for exhaustive feature exploration. Our method also yields remarkable data compression (72.57% average reduction, reaching 92.63% for longer documents). Ablation studies confirm synergistic feature contributions, establishing our context minimization as an effective solution for resource-efficient text classification with minimal performance trade-offs.

## introduction pretrained language models have achieved remarkable results across various downstream natural language understanding (nlu) tasks such as text classification. however, attaining high accuracy often requires training these models on large-scale datasets, which demands significant computational resources and entails considerable training and inference times @xcite . as modern plms continue to grow in size, fine-tuning them with extensive datasets and long contexts becomes impractical for many regular computing environments. the parameter sizes of prominent nlu models, such as bert @xcite , roberta @xcite , xlm-r @xcite , xlnet @xcite , and electra @xcite , range from millions to billions, depending on the variant. as training datasets expand, computational power, storage, and time requirements increase exponentially in the pursuit of higher accuracy @xcite . fine-tuning these models for downstream tasks often improves accuracy but also amplifies resource demands. similarly, generative large language models (llms), such as the largest variants of llama @xcite and @xcite , are several gigabytes in size, making them infeasible for fine-tuning on everyday computers, unusable in many real-world scenarios, and resulting in a large carbon footprint @xcite . driven by the challenges of high computational demands, large datasets, and extended training times, we explored methods to reduce context while maintaining competitive accuracy. our initial experiments revealed that the first sentence often strongly predicts the class. fine-tuning models using only the first sentence achieved competitive performance with significantly lower computational costs, motivating further exploration of key linguistic and statistical features. our experiments include a combination of three positional elements: first sentence (ϕ 1 ), second sentence (ϕ 2 ), and last sentence (ϕ n ); four syntactic components: nouns (n), verbs (v), adverbs (a v ), and adjectives (a d ); two semantic attributes: named entities (n e ) and proper nouns (p n ); and two statistical measures: tf-idf scores (t f ) @xcite and rake keywords (r k ) @xcite . each feature uniquely contributes to text representation, enabling the reduction of contextual requirements while maintaining task performance. for certain combinations, we selected subsets in four different amounts @xcite ) from each article to ensure focused and efficient representation. our extensive experiments on 7 nlu models and 5 popular text classification benchmark datasets, agnews @xcite , enron @xcite , imdb @xcite , bbc @xcite , and 20 news-groups @xcite , as well as our custom dataset, cmla11 (clean mixed long articles -11 categories), confirm our hypothesis: models can be fine-tuned with minimal context, requiring fewer computational resources, enabling faster training and inference speeds, while still achieving comparable accuracy. our contributions are as follows: ## related works while no prior work directly addresses the specific problem investigated in this paper, several studies offer relevant insights that inform our approach. recent research has focused on optimizing language model performance and efficiency across various dimensions. regarding context utilization, liu et al. ## methodology finding appropriate context reduction methods for accurate classification was crucial to our work. the first sentence often captures significant information in various classification tasks (news, sentiment, topic, email), as shown in appendix a ## experiments and results in this section, we first describe our datasets and experimental setup, followed by the results of our experiments and an analysis of their implications. ## conclusion this paper presents a systematic approach to context minimization for efficient text classification through strategic combinations of linguistic features. our evaluation across 6 datasets and 7 nlu models demonstrates that reduced-context configurations maintain competitive performance while enhancing efficiency. the method significantly reduces dataset sizes while preserving accuracy, making it valuable for resource-constrained environments. future work should explore applying this approach to tasks such as natural language inference, question answering, and text generation to enable more efficient language model deployment.