Do BERT -Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLM s? (2025)

Abstract: The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing “LLM-centric” trend by systematically comparing three category methods, *i.e.,* BERT-like models fine-tuning, LLM internal state utilization, and LLM zero-shot inference across six challenging datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Subsequently, we conducted experiments on a broader range of text classification tasks to demonstrate the generalizability of our findings. We further investigated how the relative performance of different models varies under different levels of data availability. Finally, based on these findings, we propose **TaMAS**, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs. Code is available at [https://github.com/jyzhang2002/TaMAS-TextClass](https://github.com/jyzhang2002/TaMAS-TextClass).

## introduction and related work with the widespread application of large language models (llms) across diverse domains @xcite @xcite , the task paradigm of text classification is undergoing a transformation. the academic community currently exhibits a pronounced "llm-centric" trend @xcite , i.e., an increasing number of studies focus on enhancing the classification performance of llms through techniques such as prompt engineering @xcite , internal state extraction @xcite @xcite , or parameter-efficient finetuning @xcite . however, this trend overlooks a critical issue: traditional bert-like models @xcite may still hold unique advantages in certain key scenarios. notably, even sota llms achieve only marginal and costly performance gains on challenging tasks like implicit hate speech detection involving near homophones or various emoji substitutions @xcite . unlike previous studies that focused mainly on llm applications in single-type text classification @xcite or evaluated text classification tasks without sufficient interpretability analysis or categorization @xcite , this work identifies an overlooked research gap and raises a key question: under the llmdominated paradigm, have we prematurely overlooked the potential of bert-like models? to address this question, we performed a comprehensive comparative evaluation in the field of text classification. for the first time, we systematically examined the performance boundaries of three computationally low-cost mainstream methods in the llm era, i.e., bert-like models fine-tuning, llm internal state utilization, and llm zero-shot inference, across six challenging classification tasks. ## comprehensive test across six typical datasets in this section, we conduct comprehensive experiments across six typical datasets and three major categories of text classification methods. ## analysis & discussions based on experimental results, this section classifies six datasets into three categories and analyzes the results using model hidden states for pca visualization and layer-wise probing. performance rankings differ by dataset type: for three implicit hate speech datasets, bert-like methods are superior, followed by those using llm internal states, with direct llm querying performing poorest. malicious code and legal text detection show bert-like and llm internal state methods performing competitively and better than direct querying. for hallucination detection, llm internal state methods surpass direct querying, both exceeding bert-like model performances. motivated by the observed performance differ-ences, we performed pca and probing computation, with corresponding visualizations presented in figure 2 and 3, and then performed an in-depth analysis of the characteristics of these dataset types to better understand the sources of variation. details of these figure are demonstrated in §e. pattern-preserving perturbated texts implicit hate speech datasets feature substantial covert language, from basic euphemisms (toxicloakcn-base) to more sophisticated emoji and homophone substitutions (toxicloakcnemoji, toxicloakcn-homo). this linguistic obfuscation increases semantic opacity and requires specific contextual knowledge, which is often unavailable to noncommunity members, including llms. from a modeling perspective, directly querying llms via prompting strategies for classification purposes yields suboptimal results. this limitation primarily stems from the fact that such models have had limited exposure to these highly concealed linguistic patterns during their pre-training phase. moreover, even when employing methods based on internal representations, i.e., such as probing techniques, the classification performance remains constrained. at the same time, the presence of covert language, which, under the conventional pre-training objective of next-token prediction, introduces a significant amount of redundant or misleading information into the internal representations of llms. as figure 2 shows, llm hidden layer representations of perturbed and implicit hate speech are poorly separated and intermingled. notably, although the three categories of subtle hate speech are semantically challenging to distinguish, close observation reveals a degree of coherence and regularity in the use of covert expressions. for example, there exist systematic patterns in the deployment of emojis and structural consistencies in homophonic substitutions. these linguistic phenomena fundamentally rely on contextual understanding, an area in which bidirectional attentionbased models like bert demonstrate particular strength. as depicted in figure 2 , the cls token embeddings demonstrate a high degree of separability between harmful and harmless instances. consequently, such models exhibit superior performance on this type of task. rule-based clean texts in malicious code and legal text datasets, both bert-like models and methods using llm internal states perform competitively. while challenging for non-experts, expert analysis shows the discriminative patterns are coherent, regular, and rule-based. crucially, unlike heavily obfuscated data from previous tasks, the current data is largely clean, free from perturbations or complex euphemisms. this data is wellrepresented in llm pre-training, forming highquality representations. as figure 2 shows, pca on bert and llm hidden representations reveals relatively good separability, supporting the strong performance of methods like saplma and mm-probe, comparable to bert-like models. directly querying llms shows weaker performance than bert-based or llm representation-based methods, likely due to misalignment between llms and human decision-making @xcite . ## generalization across diverse text classification tasks to further demonstrate the generality of our findings, we conducted additional experiments across three new text classification tasks: natural language inference (nli), german hate speech detection (extending the language coverage to english, chinese, and german), and multi-label emotion classification involving six categories. specifically, we selected gluerte @xcite , germanhate-speech @xcite , and emotion @xcite . the experimental results in ## cost performance analysis we have previously highlighted the advantages of blms and llms in various classification tasks. in the following analysis, we investigate whether the relative strengths of different models shift under varying data availability, particularly examining whether llms offer superior cost-effectiveness in scenarios of extreme data scarcity. to this end, we systematically reduce the training data to 50%, 10%, and as low as 1% of the original dataset size. our findings in however, a notable shift emerges under conditions of extreme data limitation. specifically, when the training set is reduced to 10% (corresponding to sample sizes ranging from 132 to 443 across six datasets) or less, approaches leveraging the intrinsic representations of llms consistently achieve competitive performance across all six classification tasks. we believe this advantage stems from the rich, general-purpose knowledge encoded dur-ing pre-training. in contrast, bert-like models exhibit a more pronounced degradation in performance under data-constrained conditions. this suggests that llms demonstrate greater robustness and adaptability in low-data regimes, potentially offering a more effective solution when labeled data is severely limited. ## tamas based on our findings, we propose tamas, a finegrained strategy shown in figure 1 which guides the selection of bert-like models or llms based on the characteristics of the texts. first, we summarize the proposed pattern-preserving perturbated texts and rule-based clean texts as general text classification, as there is no distinction in model selection between them. when the data volume is moderate to abundant, bert-like models demonstrate superior parameter efficiency and performance for general text classification. in contrast, there are two critical scenarios where conventional bert-like models exhibit limitations: knowledge-intensive classification: when the task requires substantial domain-specific prerequisite knowledge that cannot be adequately covered by existing training datasets. this typically occurs where the label determination depends on implicit knowledge beyond surface-level textual patterns. semantic-deep classification: cases where accurate categorization demands profound semantic understanding that cannot be reliably inferred from lexical features alone. when the data volume is very limited, the situation changes significantly. llms demonstrate stronger robustness and adaptability across all classification tasks, among which methods utilizing the intrinsic representations of llms show more competitive performance than other methods. ## conclusion our study challenges the prevailing "llm-centric" trend in text classification by demonstrating that bert-like models often outperform llms while being computationally efficient. through extensive experiments, we identify three dataset types and propose tamas, a fine-grained strategy guiding optimal model choice based on task requirements. this work advocates for a rational, task-driven approach over blind adherence to llms, ensuring efficiency without sacrificing performance.