Prompt-Tuned Muti-Task Taxonomic Transformer ( PTMTT axo F ormer) (2024)

Abstract: Hierarchical Text Classification (HTC) is a subclass of multi-label classification, it is challenging because the hierarchy typically has a large number of diverse topics. Existing methods for HTC fall within two categories, local methods (a classifier for each level, node, or parent) or global methods (a single classifier for everything). Local methods are computationally expensive, whereas global methods often require complex explicit injection of the hierarchy, verbalizers, and/or prompt engineering. In this work, we propose Prompt Tuned Multi Task Taxonomic Transformer, a single classifier that uses a multi-task objective to predict one or more topics. The approach is capable of understanding the hierarchy during training without explicit injection, complex heads, verbalizers, or prompt engineering. PTMTTaxoFormer is a novel model architecture and training paradigm using differentiable prompts and labels that are learnt through backpropagation. PTMTTaxoFormer achieves state of the art results on several HTC benchmarks that span a range of topics consistently. Compared to most other HTC models, it has a simpler yet effective architecture, making it more production-friendly in terms of latency requirements (a factor of 2-5 lower latency). It is also robust and label-efficient, outperforming other models with 15%-50% less training data.

## introduction hierarchical text classification (htc) is a specialized form of text classification that involves categorizing text documents into a hierarchical structure of labels or categories. in htc, the labels are organized in a tree-like structure or a directed acyclic graph (dag) @xcite ; @xcite ), where parent-child relationships exist between categories at different levels. htc allows for more nuanced and detailed classification of text compared to flat classification systems. htc is used in various applications where organizing and categorizing large amounts of textual information is necessary @xcite . common use cases include document organization, content management, e-commerce product categorization, knowledge management and scientific literature classification @xcite . these applications benefit from the hierarchy, which allows for more precise and granular categorization, particularly in domains with complex, multi-level category structures. htc models aim to capture both the content of the text and the relationships between different levels of categories, making them effective for organizing and retrieving information in hierarchical systems. htc approaches are generally divided into two groups -local and global approaches. local approaches tend to partially or fully ignore the hierarchical structure in the training paradigm, which lead to information loss @xcite ; @xcite ). local models also have massive trainable parameters as they consist of multiple models. global methods overcome the limitations above by taking into consideration the hierarchy and using the hierarchy information either explicitly or implicitly @xcite ). we refer the readers to @xcite for an in-depth review of these approaches. prompt-based learning methods are a subset of the global methods. recent advancements in prompt-based learning @xcite , @xcite , @xcite , @xcite ) have shown that prompt-based approaches produce better results than vanilla finetuning of the encoded text representation. various prompt-based learning techniques have been explored, including in-context learning @xcite ), prompt based few shot learning @xcite ), multi-prompt learning @xcite , @xcite ) and prompt based training @xcite ). one of the main drawbacks to these methods is that the performance is sensitive to the prompts chosen. to mitigate this issue, we adopted differentiable prompts that are learnt through backpropagation during training, similar to @xcite . another key aspect in the aforementioned approach is the treatment of labels in the hierarchy, where most earlier approaches leveraged verbalizers chosen ad-hoc @xcite . however, in a complex hierarchy, especially for a highly-specialized domain, labels could have nuanced semantic differences that are difficult to capture by the verbalizer. in the meantime, it is not viable to come up with and maintain the mapping. to this end, we represent each label in the hierarchy with a new, differentiable token and these tokens are also learned through backpropagation during training. our method uses text input appended with prompt tokens and [m ask] tokens (one for each level of hierarchy). the model is then trained to unmask the [m ask] tokens to output the correct label tokens (again, one / more for each level of hierarchy). the prompt tokens, the label tokens and the model parameters are jointly optimized during the training process. we describe the process in further detail in subsection 2.1. the objective function we use to optimize the above parameters is a combination of weighted asymmetric loss, l1 loss and mlm loss, as described in subsection 2.2. we summarize our contributions as follows: this is critical for many industrial applications, as getting training data is expensive. ## methodology the problem of hierarchical multi-label prediction consists of two components: hierarchical prediction and multi-label prediction. this section describes our approach in solving both components. ## discussion and future work this paper presents ptmttaxoformer -a simple framework that can be extended to any hierarchical text prediction task using any pretraining language model available. our method does not require any complex heads or modules for hierarchy injections, verbalizers or engineering of prompts. while retaining simplicity, our framework also attains state-of-the-art performance. we attribute the improvement in performance to the fact that the classification task is made to resemble the pretraining task (mlm in case of bert) and it utilizes the ability of the model to perform pretraining tasks well. during training, attention between the label tokens allows the model to learn the hierarchy implicitly -the bi-directional self attention proves to be a powerful architecture to implicitly learn hierarchies. as a part of the future work, we'd like to extend our approach to use a decoder or encoderdecoder-based approach as we expect next token prediction problem to also be a good proxy for explicit hierarchy injection. the prompts used in our method are the same for all samples. a path worth exploring would be to have a small language model to predict the prompt tokens making the prompts customized to each sample.