BART - TL : Weakly-Supervised Topic Label Generation (2021)

Abstract: We propose a novel solution for assigning labels to topic models by using multiple weak labelers. The method leverages generative transformers to learn accurate representations of the most important topic terms and candidate labels. This is achieved by fine-tuning pre-trained BART models on a large number of potential labels generated by state of the art non-neural models for topic labeling, enriched with different techniques. The proposed BART-TL model is able to generate valuable and novel labels in a weakly-supervised manner and can be improved by adding other weak labelers or distant supervision on similar tasks.

## introduction as topic modeling has been used for unsupervised exploration of large text corpora, several topic labeling approaches have been proposed. these range from heuristic-based methods @xcite that focus on the underlying topic distributions to newer methods that use word embeddings @xcite . supervised topic labeling methods @xcite typically use annotator data with the quality of the labels to train a more accurate ranker than the unsupervised counterpart. deep learning approaches, which gained quick popularity in nlp, are starting to be used for solving this task as well @xcite . recently, transformer models pre-trained on very large amounts of data achieved impressive results on a lot of downstream nlp tasks using fewer resources than previously necessary. we introduce a method of performing a weakly-supervised fine-tuning on these models pre-trained on english data in order to obtain human-comprehensible and meaningful topic labels. we also provide a quality evaluation of the model-generated labels, in addition to an analysis of the contribution gained from using this approach that we ultimately refer to as bart-tl, inspired by the name of the original transformer architecture. ## related work topic modeling is a popular unsupervised method for exploring large corpora of documents. topics are represented as distributions over words, while documents as mixtures of topics. historically, these methods used dimensionality reduction techniques @xcite , then migrated to probabilistic-based methods @xcite , with latent dirichlet allocation @xcite gaining popularity. lda makes use of variational inference to obtain the distribution matrices. further developments include hierarchical @xcite and online @xcite versions of lda. while the resulting distributions of topic models are useful for computational purposes, such as measuring the similarity of two documents, these may prove difficult to interpret by humans. topic labeling aims to solve this issue by computing labels for each topic. historically, this was achieved by establishing a pool of labels and ranking them using certain scoring functions. first attempts were fully unsupervised, extracting labels from the original corpus @xcite . later approaches started using external corpora, such as wikipedia, as candidates for labels and trained supervised rankers @xcite , as well as employed word embeddings @xcite such as word2vec @xcite and doc2vec @xcite for computing the similarity between a topic and a candidate label. huge progress was made in the nlp field with the introduction of attention models @xcite and, later on, transformers @xcite , which are deep neural networks that use an encoder-decoder architecture. a multitude of transformer-based models @xcite @xcite emerged that achieved state of the art performance on a large number of nlp tasks through transfer learning. these models are pre-trained on large amounts of data in order to encompass general knowledge of the language to be later finetuned on downstream tasks. this allows for better results on small datasets, where deep learning was not a viable option beforehand. however, research on using deep learning methods for topic labeling is scarce. a very recent study proposes an rnn-based encoder-decoder architecture @xcite trained with distant supervision using wikipedia page titles and employing bertscore @xcite for evaluation. ## method our method utilizes a pre-trained bart @xcite transformer model, with a denoising autoencoder architecture, as we adopt a sequenceto-sequence approach for the task of topic labeling. ## results we gather annotations in the form of surveys with 7 questions, one per topic, on the quality of topic labels on a scale from 0 to 3. the annotators have varying backgrounds, including computer science, medicine, law, and economics. for each of the 5 subjects in the corpus, we select 6 coherent topics for evaluation. the labels are taken from the unsupervised and supervised versions of the original netl method, along with bart-tl-ngram, and bart-tl-all. for each method, only top-10 labels are considered for evaluation. an extra stopword label is introduced as a distractor, removing answers from annotators with over 25% of these @xmath0. a topic is presented using its top-10 terms, along with 2 relevant short paragraphs, to offer additional context when the topic is unclear. each survey has balanced topics based on the 5 subjects and each question contains 9 balanced labels based on the models. we gathered a total of 35 survey responses and filtered out the labels that had only a single annotation. this annotation was performed pro-bono and we estimate that the average time per annotated survey was 10 minutes. there is no bias in the annotations for certain models, as the average standard deviation for rating of individual labels is between 0.42 and 0.44 for all of them. the results of this study are presented in ## conclusion we introduced the bart-tl model that builds upon previous topic labeling solutions by adopting a generative deep learning strategy. large pre-trained transformer models are fine-tuned in a weaklysupervised manner using unsupervised labelers to obtain meaningful labels. while current results have varying quality compared netl, bart-tl is able to generate novel labels of similar quality. although bart-tl experiments have been carried out for english, our generative methodology can be applied to any language if a pre-trained bart model is available.