T ext2 T opic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilities (2023)

Abstract: Multi-label text classification is a critical task in the industry. It helps to extract structured information from large amount of textual data. We propose Text to Topic (Text2Topic), which achieves high multi-label classification performance by employing a Bi-Encoder Transformer architecture that utilizes concatenation, subtraction, and multiplication of embeddings on both text and topic. Text2Topic also supports zero-shot predictions, produces domain-specific text embeddings, and enables production-scale batch-inference with high throughput. The final model achieves accurate and comprehensive results compared to state-of-the-art baselines, including large language models (LLMs). In this study, a total of 239 topics are defined, and around 1.6 million text-topic pairs annotations (in which 200K are positive) are collected on approximately 120K texts from 3 main data sources on Booking.com. The data is collected with optimized smart sampling and partial labeling. The final Text2Topic model is deployed on a real-world stream processing platform, and it outperforms other models with 92.9% micro mAP, as well as a 75.8% macro mAP score. We summarize the modeling choices which are extensively tested through ablation studies, and share detailed in-production decision-making steps.

## introduction in the digital age, large-scale online travel platforms (otps) face the challenge of effectively extracting valuable insights from massive volumes of textual data. such an otp can get hundreds of millions of customer reviews in one year, so structured insights are crucial for comprehending customer behavior and making data-driven decisions in order to improve the overall travel experience. one application example is to find the top facilities for each hotel, by extracting information from the positive reviews, which can lead to better accommodation recommendations. similarly, understanding travel destination themes, such as romantic getaways, city trips, or family trips can enhance destination recommendations. in this study, we research use cases from booking.com and define in total 239 valuable topics. each topic is set with a topic name and a topic description, to better match the natural customer language and for optimal model training results. the main data source is user-generated content on booking.com, including customer reviews and forum posts from hotel owners and travelers. developing an architecture that ensures high accuracy, scalability for a large number of topics, low cost and low latency on real-world inference is of utmost importance. sentence-bert (reimers and gurevych, 2019) extends bert @xcite for sentence-level embeddings, achieving impressive performance on tasks like sentence similarity and semantic retrieval. multilingual universal sentence encoder for semantic retrieval (muse) @xcite , a multilingual extension of the universal sentence encoder @xcite , enables cross-lingual semantic retrieval and provides multiple open-source models. though there are also other state-of-the-art approaches, the two methods above are prevalent in real industry applications, due to the computational efficiency, high and robust in-domain performance by fine-tuning, zero-shot ability, and strengths in scalability. our proposed text2topic framework adopts a fine-tuning approach upon pre-trained language models. specifically, we employ the bi-encoder transformer @xcite architecture proposed by sentence-bert, which allows separate injection of the text and topic information, as section 2 shows. this architecture not only enables the model to have zero-shot capabilities (handle new topics for inference) but also exhibits text embedding abilities. by leveraging the strengths of the pre-trained language model and incorporating topic-specific information, the model effectively addresses the challenge of topic detection. the paper's contributions are summarized as follows: ## architectures in this study, we research 3 main architectures. for each text-topic (with topic description), we know one binary ground truth for this pair, and perform: the bi-encoder architecture has 3 benefits over cross-encoder: 1) low inference time-complexity: we pre-calculate and cache all topic embeddings, only embed each text once and repeat the text vector to score on all topics. given n as the number of texts and t as the number of topics, to get n × t predictions, the bi-encoder needs o(n + t ) encoding operations, while it is o(n • t ) for the cross-encoder. 2) in-house embedding: bi-encoder enables us to have the text part embeddings, that can be used as features for other tasks. 3) for the same base model, the bi-encoder allows longer text input since text and topic are embedded separately. all the above architectures can easily extend to include new topics for training, and also have zero-shot possibility when the unseen topic is well-defined with a description. for all architectures, we experiment with three pooling strategies (reimers and gurevych, 2019): using the output of the [cls]; computing the mean or max of output vectors on all tokens (mean-pooling, max-pooling). ## in-house dataset construction with hundreds of topics, the annotation becomes challenging: how to define, merge, and distinguish topics; how to decide annotation volume and candidate texts per topic and reduce cost. this section shows how we tackle them by smart model-based sampling and partial labeling. ## experimental setup all experiments are performed on a computation instance equipped with 1 nvidia tesla t4 tensor core gpu, 4 vcpu, and 16gb ram. the experiments have the following general settings: bertbase-multilingual-cased model as the pre-trained base model; fine-tuning all layers; mixed-precision training; batch size of 12 text-topic pairs, with gradient accumulation steps as 8; weight decay (on all weights that are not gains or biases) with coefficient 0.01; adamw optimizer @xcite ; initial learning rate 1e-5 with a linear scheduler; allowing maximum 6 epochs with early stopping patience as 3 steps, and warm-up steps as 10k. we apply stratified sampling (on topic frequency) on the texts, and get training/validation/test sets, with a ratio of 70/15/15 respectively. ## real-world applications this section describes 3 main real-world use cases which employ text2topic. besides, the text2topic training system is also re-used effectively for training other data sources like search queries. it is important to note that use cases may require varying threshold settings. we use the f-beta score @xcite to determine the optimal threshold setting on the topic's probability score for a given use case. for recall-oriented scenarios, where minimizing false negatives is critical, we set @xmath0. conversely, for precision-oriented cases, where reducing false positives is a priority, we set @xmath1. for each topic, to find the best threshold, we systematically vary its value by computing the threshold that yields the highest f-beta score, using the chosen beta value. property recommendation: reviews contain rich information that encapsulates the users' preferences towards different properties. text2topic turns them into structured features, which enhance the in-house property recommendation models' performance. with classification scores on reviews, we perform property-level score aggregations, to extract a variety of insightful attributes such as a property's relevancy for different themes (e.g., beach, spa/wellness). these attributes are integrated into the recommendation models to increase relevant inventory (e.g., number of beach properties is increased by 4% by leveraging text2topic); and to create novel and nuanced categories of recommendations (such as castle-type hotels, romantic getaways, etc.). furthermore, the model provides a natural mechanism to serve explainable recommendations by linking them to relevant reviews. detect property type: with text2topic predictions on reviews, we are able to detect hidden properties categorizations, by analyzing relevant topic (guest house, farm stay, resort, chalet etc.) frequencies. for example, an apartment property that is described as a guest house, could be surfaced to users that are searching for a guest house. we detect over a million extra properties supply (774k more apartments, 25k more villas and 60k more cabins/chalets). fintech: text2topic training pipeline enables us to train a new model on fintech data and topics, such as payments and questions about invoices and commissions. the model auto-classifies incoming messages from customers and correctly re-routes them to the right self-service solution, which increases the auto-reply success rate by 9% and reduces manual handling time. ## conclusion in this paper, we present text2topic, a flexible multi-label text classification system that is de-ployed at booking.com, with high performance and supports multiple applications. we summarize lessons learnt from the end-to-end production journey, including practical annotation approaches, modeling choices, and production decisions, which are valuable references for the industry domain. we also compare the performance with llm like gpt-3.5, and text2topic is a more feasible choice from multiple aspects. for future work, we can explore if parameter efficient fine-tuning techniques (e.g., lora @xcite , p-tuning @xcite ) on open-source llms could bring better performance, and how to better balance the model specialization and generalization power for zeroshot. a appendix this plot includes the popular topics which have more than 50 positive annotations each in the test set. we can see that when inspecting topic-level performance, the text2topic bi-encoder concat has stronger zero-shot ability than muse. muse is better at some topics, which we find are mainly facility specific topics (bbq, towel, vending machine, stairs etc.).