LLME mbed: Rethinking Lightweight LLM ’s Genuine Function in Text Classification (2024)

Abstract: With the booming of Large Language Models (LLMs), prompt-learning has become a promising method mainly researched in various research areas. Recently, many attempts based on prompt-learning have been made to improve the performance of text classification. However, most of these methods are based on heuristic Chain-of-Thought (CoT), and tend to be more complex but less efficient. In this paper, we rethink the LLM-based text classification methodology, propose a simple and effective transfer learning strategy, namely LLMEmbed, to address this classical but challenging task. To illustrate, we first study how to properly extract and fuse the text embeddings via various lightweight LLMs at different network depths to improve their robustness and discrimination, then adapt such embeddings to train the classifier. We perform extensive experiments on publicly available datasets, and the results show that LLMEmbed achieves strong performance while enjoys low training overhead using lightweight LLM backbones compared to recent methods based on larger LLMs, *i.e.* GPT-3, and sophisticated prompt-based strategies. Our LLMEmbed achieves adequate accuracy on publicly available benchmarks without any fine-tuning while merely use 4% model parameters, 1.8% electricity consumption and 1.5% runtime compared to its counterparts. Code is available at: https://github.com/ChunLiu-cs/LLMEmbed-ACL2024.

## introduction recently large language models (llms) have shown remarkable abilities on various nlp applications, such as gpt @xcite , palm @xcite and llama @xcite , offering services to users through dialogue. since llms refer to large-scale pre-trained language models (plms) that undergo extensive training on massive textual corpora to understand the complexity and relationships within language, llms exhibit amazing emergent abilities in comprehension and reasoning @xcite . this phenomenon further promotes research of prompt learning for llms @xcite , such as chain-of-thought (cot) @xcite and tree-of-thoughts (tot) @xcite . prompt-learning @xcite is closely connected to the training and inference of llms. instead of adapting the pre-trained language models (plms) to address downstream tasks, promptlearning directly adapts llms to cloze-style prediction, autoregressive modeling, or sequence to sequence generation, leading to promising performances on various tasks @xcite . its major advantage of is that, given a suite of appropriate prompts, llms can be used to solve a great number of tasks @xcite with no necessity of training from the scratch. however, such a paradigm is deeply involved with prompt engineering to find the most appropriate prompts to improve the overall performance. based on the sophisticated prompts, llms are guided to generate results and have achieved counterpart, or even stronger performance comparable to supervised baselines in various downstream nlp tasks such as natural language inference @xcite , question answering @xcite , information extraction @xcite , named entity recognition @xcite and relation extraction @xcite . nevertheless, as a generative model, llms still underperform discriminative models in text classification. recently sun et al. introduced the so-called clue and reasoning prompting (carp) @xcite to guide gpt-3 through text classification and got the sota performances on 5 widely-used text classification benchmarks. ## methodology as mentioned in the section 1, improving the performance of locally deployed lightweight llms is urgently important, considering specific scenario customization of llms @xcite , user data privacy security @xcite , and limited computational resource @xcite . however, the emergent abilities of lightweight llms fall behind online llms due to their far smaller parameter scale @xcite , which subsequently limits the effectiveness of lightweight llms in prompt-based text classification methods. therefore, we present the so-called llmembed, which directly employ the lightweight llms to extract text embeddings, and improve the overall performance of text classification. in this paper, we extract and fuse embeddings extracted by multiple backbones at different depths to improve the robustness and generalization. let us take f (•|m, d m ) to denote the embedding extraction where m is the model and d m is the network depth. g(•|θ g ) refers to the classifier head. @xmath0, y i } n @xmath1, y i is the label and n is the total number of datapoints. d is divided into d train and d test for training and evaluation. during training, we first randomly sample a batch of datapoints {x i , y i } n b @xmath2, then feed them into f (•|m, d m ) to extract the semantic embeddings {ϕ following we fuse all available embeddings via the operator v(•) to get the final semantic embedding ψ i for the i-th datapoint. avg the fusion operators investigated in this paper are listed in co-occurence pooling. given the embedding set {ϕ m,dm i } n b @xmath3, we first use linear mappings to project them into the same latent space and align them to the same length 1 × k, thus making ϕ m,dm @xmath4, d. then they are stacked to formulate the φi ∈ r h×k where @xmath5. following, we calculate their cooccurrence statistics ψ @xmath6. overall, our llmembed pipeline can be regarded as a typical transfer learning framework, which focuses on difference fusion strategies on llm-based embeddings. as the embedding extraction process is fast, our model can be trained and adapted to the target domains with very high efficiency. ## conclusion in this paper, we propose a concise and effective llm-based paradigm, namely llmembed, to address the text classification task. this novel llmembed paradigm achieves the state-of-theart performance when solving text classification with lightweight llms, even comparable to the performance of llms (e.g. gpt-3) with sophisticated prompt-based strategies. the llmembed paradigm is flexible and scalable in which the text embeddings obtained from different lightweight llms can be fused to improve the overall performance. moreover, llmembed is far more efficient and budget saving than the prompt-based paradigms. lastly, we expect that our novel llmembed paradigm can be beneficial for using generative llms to handle other down-stream discriminative tasks.