Prompt Tuned Embedding Classification for Industry Sector Allocation (2024)

Abstract: We introduce Prompt Tuned Embedding Classification (PTEC) for classifying companies within an investment firm’s proprietary industry taxonomy, supporting their thematic investment strategy. PTEC assigns companies to the sectors they primarily operate in, conceptualizing this process as a multi-label text classification task. Prompt Tuning, usually deployed as a text-to-text (T2T) classification approach, ensures low computational cost while maintaining high task performance. However, T2T classification has limitations on multi-label tasks due to the generation of non-existing labels, permutation invariance of the label sequence, and a lack of confidence scores. PTEC addresses these limitations by utilizing a classification head in place of the Large Language Models (LLMs) language head. PTEC surpasses both baselines and human performance while lowering computational demands. This indicates the continuing need to adapt state-of-the-art methods to domain-specific tasks, even in the era of LLMs with strong generalization abilities.

## introduction investors leveraging thematic investment strategies concentrate their efforts on specific industry sectors, such as "circular economy." this strategy involves compiling a comprehensive list of companies within these sectors by analyzing unstructured natural language data on platforms such as @xcite and @xcite . for instance, investors might utilize the description and associated keywords of a company like "vinted" to identify the industries it operates in. in this context, machine learning can be instrumental by framing this as a multi-label text classification challenge: given a natural language description of a company x, the goal is to categorize it into one or more industries from a predefined industry sector taxonomy @xmath0, y 2 , . . . , y n }. while there exist various machine learning solutions for multi-label text classification, this industrial application encompasses some challenges: the necessity to infer industries for over 10m companies, coupled with the likelihood of the inference data being out-of-distribution compared to the annotated dataset in terms of language use and descriptiveness. traditional text classification approaches demand large amounts of annotated training data and often struggle to generalize effectively to novel data @xcite . large language models (llms) exhibit superior generalization capabilities to unseen data and can be fine-tuned on smaller annotated datasets @xcite . however, fine-tuning llms may lead to the undesirable phenomenon of "catastrophic forgetting" of pretraining knowledge @xcite , and is computationally demanding. these challenges can be mitigated through parameter-efficient fine-tuning (peft, @xcite techniques such as prompt tuning (pt). pt minimizes the number of parameters that need fine-tuning by focusing on a soft prompt appended to the tokenized and embedded input text, thus reducing computational costs and preserving the pretrained knowledge of the llm, as the main body of the llm's parameters remains unaltered @xcite @xcite . hence, pt emerges as a viable solution for computational efficiency and knowledge retention in llm applications. this research evaluates the scalability, efficiency, and performance of pt in a real-world industry classification scenario, benchmarked against common baseline methods. however, pt as a textto-text (t2t) classification approach encounters limitations on multi-label tasks as discussed in subsection 2.4. we enhance pt by (1) integrating constrained decoding using trie search @xcite and ( 2 ) replacing the language model head with a specialized classification head. our key contributions include: the paper first outlines existing text classification methodologies and their limitations. we then introduce constrained trie search decoding and ptec as potential solutions to these limitations. subsequently, we describe our experimental setup and compare the efficiency and performance of current and proposed methods. our codebase and the hatespeech dataset can be accessed at https://github.com/eqtpartners/ptec . ## conclusion this study benchmarks computational cost and multi-label text classification performance of pt as a parameter-efficient alternative to fine-tuning all llm parameters. to address the limitations of a t2t approach on multi-label classification problems, pt is extended with trie search as a constrained decoding strategy, and with embedding classification as an alternative to t2t classification. results indicate that trie search can significantly improve the performance of n -shot prompting. pt can outperform popular text classification approaches on both our domain-specific industry-sector classification task, and the publicly released hatespeech classification benchmark. both performance and efficiency can be further improved by combining pt with embedding classification. the proposed solution, ptec, outperforms baselines and human professionals and can be deployed at scale to accelerate and objectify industry sector allocation.