A Million Tweets Are Worth a Few Points: Tuning Transformers for Customer Service Tasks (2021)

Abstract: In online domain-specific customer service applications, many companies struggle to deploy advanced NLP models successfully, due to the limited availability of and noise in their datasets. While prior research demonstrated the potential of migrating large open-domain pretrained models for domain-specific tasks, the appropriate (pre)training strategies have not yet been rigorously evaluated in such social media customer service settings, especially under multilingual conditions. We address this gap by collecting a multilingual social media corpus containing customer service conversations (865k tweets), comparing various pipelines of pretraining and finetuning approaches, applying them on 5 different end tasks. We show that pretraining a generic multilingual transformer model on our in-domain dataset, before finetuning on specific end tasks, consistently boosts performance, especially in non-English settings.

## introduction online platforms and social media are increasingly important as communication channels in various companies' customer relationship management (crm). to ensure effective, qualitative and timely customer service, natural language processing (nlp) can assist by providing insights to optimize customer interactions, but also in real-time tasks: (i) detect emotions @xcite , (ii) categorize or prioritize customer tickets @xcite , (iii) aid in virtual assistants through natural language understanding and/or generation @xcite , etc. despite this nlp progress for crm, often small and medium-sized companies (smes) struggle with applying such recent technology due to the limited size, noise and imbalance in their datasets. general solutions to such challenges are transfer learning strategies @xcite : feature extraction uses frozen model parameters after pretraining on an external (larger) training corpus, while finetuning continues training on the smaller in-domain corpus. in the large body of work adopting such strategies (e.g., pan and yang 2009), little effort has been put into addressing specific crm use cases that need to rely on social media data that is noisy, possibly multilingual, and domain-specific for a given company. in this paper, we analyze the possibilities and limitations of transfer learning for a number of crm tasks, following up on the findings of @xcite who demonstrate gains from progressive finetuning on in-domain and taskspecific monolingual data. specifically, our contributions are that we (1) collect a multilingual corpus of 275k twitter conversations, comprising 865k tweets between customers and companies in 4 languages (en, fr, de, nl), covering distinct sectors (telecom, public transport, airline) (section 4.1); ## related work a wide range of nlp research has been devoted to customer services. @xcite employed data mining techniques to extract features from a customer service database for decision support and machine fault diagnosis. @xcite extracted a set of sentiment and syntactic features from tweets for customer problem identification tasks. @xcite introduced the customer obsession ticket assistant for ticket resolution, using feature engineering techniques and encoder-decoder models. highly popular pretrained language models, such as bert @xcite , have also been explored for different customer service tasks: hardalov et al. (2019 ) considered re-ranking candidate answers in chatbots, while deng et al. (2020) proposed bert-based topic prediction for incoming customer requests. although the performance gains obtained by pretraining language models are well-established, they need further exploration in terms of multilinguality. some studies @xcite @xcite have investigated the transferability of multilingual models on different tasks, but they do not consider the effect of progressive pretraining on a smaller and less diverse multilingual corpus, as we will do. ## experimental setup we focus our experiments on text classification problems that are commonly dealt with by customer service teams. first, we describe our twitter conversation corpus used for in-domain finetuning (section 4.1), then we introduce the target tasks and corresponding datasets (section 4.2). for most target tasks, we hold out 10% of the data for testing, while the remaining part is used for training. we then utilize 10-fold cross-validation on the training data to select optimal hyper-parameters for each end task. when the dataset comes with a predefined train-test split, we keep that. for the pretrained transformer models (roberta, xlm, bertweet), we use the publicly available 'base' versions. ## results and discussion we now present our findings for two finetuning scenarios: transfer across languages and across tasks. section 5.1 investigates the effect of unsupervised multilingual pretraining. section 5.2 then explores how to further improve by finetuning the pretrained language models on similar tasks. ## conclusion we investigated multilingual and across-task transfer learning for customer support tasks, based on transformer-based language models. we confirmed prior insights that finetuning the models on lowresource end tasks is important. additional pretraining on a moderately sized in-domain corpus, however, provides a complementary increase in effectiveness, especially in the non-english setting and starting from a generic multilingual language model. we provide a newly collected multilingual in-domain corpus for customer service tasks and derive the aforementioned findings from experiments using it on five different tasks.