Efficient Strategies for Hierarchical Text Classification: External Knowledge and Auxiliary Tasks (2020)

Abstract: In hierarchical text classification, we perform a sequence of inference steps to predict the category of a document from top to bottom of a given class taxonomy. Most of the studies have focused on developing novels neural network architectures to deal with the hierarchical structure, but we prefer to look for efficient ways to strengthen a baseline model. We first define the task as a sequence-to-sequence problem. Afterwards, we propose an auxiliary synthetic task of bottom-up-classification. Then, from external dictionaries, we retrieve textual definitions for the classes of all the hierarchy’s layers, and map them into the word vector space. We use the class-definition embeddings as an additional input to condition the prediction of the next layer and in an adapted beam search. Whereas the modified search did not provide large gains, the combination of the auxiliary task and the additional input of class-definitions significantly enhance the classification accuracy. With our efficient approaches, we outperform previous studies, using a drastically reduced number of parameters, in two well-known English datasets.

## introduction hierarchical text classification (htc) aims to categorise a textual description within a set of labels that are organized in a structured class hierarchy @xcite . the task is perceived as a more challenging problem than flat text classification, since we need to consider the relationships of the nodes from different levels in the class taxonomy @xcite . both flat text classification and htc have been tackled using traditional machine learning classifiers @xcite or deep neural networks @xcite . nevertheless, the majority of the latest approaches consider models with a large number of parameters that require extended training time. in the flat-classification scenario, some studies have addressed the problem of efficiency by proposing methods that do not focus on the model architecture, but in external ways of improving the results @xcite . however, the listed strategies are still underdeveloped for htc, and the most recent and effective methods are still computationally expensive @xcite . the described context opens our research question: how can we improve htc at a lower computational cost? therefore, our focus and main contributions are: ## experimental setup datasets. we test our model and proposed strategies in two well-known hierarchical text classification datasets previously used in the evaluation of state-of-the-art methods for english: web of science (wos; @xcite and dbpedia @xcite . the former includes parent classes of scientific areas such as biochemistry or psychology, whereas the latter considers more general topics like sports season, event or work. general information for both datasets is presented in model, hyper-parameters and training. we use the allennlp framework @xcite to implement our methods. our baseline consists of the model specified in §2.1. for all experiments, we use 300 units in the hidden layer, 300 for embedding size, and a batch size of 100. during training time, we employ adam optimiser (kingma and ba, 2014) with default parameters (β 1 = 0.9, β 2 = 0.98, ε = 10 -9 ). we also use a learning rate of 0.001, that is divided by ten after four consecutive epochs without improvements in the validation split. furthermore, we apply a dropout of 0.3 in the bidirectional gru encoderdecoder, clip the gradient with 0.5, and train the model for 30 epochs. for evaluation, we select the best model in the validation set of the 30 epochs concerning the accuracy metric. settings for the proposed strategies. ## results and discussion we also notice that the accuracy increment is relatively higher for all experiments on the wos corpus than on dbpedia. a primary reason might be the number of documents in each dataset, as dbpedia contains almost seven times the number of documents of wos. if we have a large number of training samples, the architecture is capable of learning how to discriminate correctly between classes only with the original training data. however, in less-resourced scenarios, our proposed approaches with external knowledge integration could achieve a high positive impact. as our strategies are orthogonal and focus on different parts of the model architecture, we proceed to combine them and assess their joint performance. in the case of wos, we observe that every combination of strategies improves the single counterparts, and the best accuracy is achieved by the merge of the auxiliary task and pnc, but with an original beam search of size five. concerning dbpedia, most of the results are very close to each other, given the high accuracy provided since the seq2seq baseline. however, we note the relevance of combining the pnc strategy with the original or modified beam search to increase the performance. finally, we compare our strategies to the best htc models reported in previous studies @xcite . we then observe that the results of our methods are outstanding in terms of accuracy and number of parameters. moreover, the training time of each model takes around one hour (for the 30 epochs), and the proposed auxiliary task do not add any significant delay. ## related work most of the studies for flat text classification primarily focus on proposing a variety of novel neural architectures @xcite . other approaches involve a transfer learning step to take advantage of unlabelled data. @xcite used the encoder unit of a neural machine translation model to provide context for other natural language processing models, while @xcite pre-trained a language model on a general-domain monolingual corpus and then fine-tuned it for text classification tasks. in htc, there are local or global strategies @xcite . the former exploits local information per layer of the taxonomy, whereas the latter addresses the task with a single model for all the classes and levels. neural models show excellent performance for both approaches @xcite . furthermore, other studies focus on using transfer learning for introducing dependencies between parent and child categories @xcite and deep reinforcement learning to consider hierarchy information during inference @xcite . the incorporation of external information in neural models has offered potential in different tasks, such as in flat text classification. by using categorical metadata of the target classes @xcite and linguistic features at word-level @xcite , previous studies have notably improved flat-text classification at a moderate com-putational cost. besides, @xcite outperform several state-of-the-art classification baselines by employing multitask learning. to our knowledge, the latter strategies are not explicitly exploited for htc. for this reason, our study focuses on the exploration and evaluation of methods that enable hierarchical classifiers to achieve an overall accuracy improvement with the least increasing complexity as possible. ## conclusion we presented a bag of tricks to efficiently improve hierarchical text classification by adding an auxiliary task of reverse hierarchy prediction and integrating external knowledge (vectorized textual definitions of classes in a parent node conditioning scheme and in the beam search). our proposed methods established new state-of-the-art results with class hierarchies on the wos and dbpedia datasets in english. finally, we also open a path to study integration of knowledge into the decoding phase, which can benefit other tasks such as neural machine translation.