Three Approaches to Client Email Topic Classification (2023)

Abstract: This paper describes a use case that was implemented and is currently running in production at the Nova Ljubljanska Banka, that involves classifying incoming client emails in the Slovenian language according to their topics and priorities. Since the proposed approach relies only on the Named Entity Recogniser (NER) of personal names as a language-dependent resource (for the purpose of anonymisation), that is the only prerequisite for applying the approach to any other language.

## introduction together with nova ljubljanska banka's (nlb) centre of excellence, belgrade it company nlb digit has a mission to incorporate smart, data-driven it solutions to various aspects of everyday work in different bank's business sectors. one such case is the classification of client emails sent to the bank's contact centre (kontakt centar in slovenian, dubbed kc) with respect to their topic (e.g. accounts/loans/cards, etc.) and priority @xcite . in this paper we present the whole procedure, from having only the plain outlook files to the models assigning topics and priorities to emails in real time. section 2 mentions some of the previously published scientific articles that inspired this research. we propose and discuss three different approaches for the modelling of emails in section 3. afterwards, we explain the process of preparing the dataset in section 4: selection of optimal classification schemes and manual annotation, followed by certain cleaning steps and anonymisation of personal information. having the prepared dataset, we trained and exhaustively evaluated different nlp models for the case of topic classification. we explain the training process in more detail in section 5, where we also show evaluation results first on a validation and then on a separate test set. finally, we close with some final remarks, ideas for further improvement and conclusions in section 6. ## related work the problem of email classification has been an active area of research for several decades, with numerous studies focusing on developing effective algorithms to accurately categorise incoming emails based on their content. researchers still experiment with different techniques and approaches in order to improve the existing spam and phishing email classifiers. @xcite achieved the 98.06% accuracy using the binary support vector machine (svm) classifier for the case of spam, whilst @xcite developed the optimal spam classifier using rotation forest algorithm, achieving the accuracy of 94.2%. @xcite experimented with feature engineering and rnn/cnn architectures, concluding that rnn provides the highest 94.9% accuracy. the binary svm classifier also proved to be the best phishing email detector for @xcite , with 87.85% accuracy. svm proved to be the optimal classifier in all our experiments, which we clarify in the coming sections. ## methodology in this section we will describe three different approaches that we hypothesised to be appropriate for the multi-class email topic classification: the bert "all-in-one" approach refers to training each of the models on all samples at once by fine-tuning an off-the-shelf bert language model for the slovenian language. having the same general pre-processing pipeline for all cases of email classification (described in subsection 4), we propose to fine-tune the sloberta (ulčar and robnik-šikonja, 2021) model for the slovenian language on the whole dataset. the first step is to ensure that only slovenian emails are present in the dataset, using an off-the-shelf language detection tool. after performing a pre-defined text processing procedure, two different models are to be trained separately: topicsloberta (for the topic classification) and priorsloberta (for the priority classification). the same would hold for any other bert model. there are certain drawbacks with this method. first, the performance of the final model depends on the underlying bert model. if the bert model itself is trained on data that comes from a domain that very much differs from the lexica of client emails, one cannot expect too much from the classification outcome. fine-tuning of bert models demands all class labels to be well covered in the means of number of representative samples, and to be well balanced, which represents one potential issue with client emails. additionally, despite there not being any official lower boundary in the means of number of instances on which a bert model should be fine-tuned, it is well known that for the case of deep neural network models holds the "more the merrier" rule, which could also be one of the performance limitations in our case. finally, from the technical point of view, finetuning of this model demands strong computing resources, which could also represent one of the reasons against using this approach. ## dataset it was first necessary to define the annotation schema and the data cleaning process. after taking into account the internal schema kc used over time when archiving finished email correspondences with clients into a database, and agreeing that the priority should directly follow from the topic, we settled on the 10 topics shown in before using textual data for any machinelearning-related task, certain pre-processing steps should be performed. what has turned out to be a very beneficial in practice is to reduce the vocabulary of the text collection as much as possible, as long it does not affect the semantics of the content. let us observe an email given in only segments that were not struck out (subject and middle of the body) contain the client's query, while the rest is either some generic content (generated by the nlb's mail server or by the user's mobile email application). similarly, personal information such as addresses, full names, mobile phone numbers, pin, and account numbers represent sensitive information, yet do not in any way influence classification predictions. on the email body concatenated to its subject, pre-processing procedure consists of the following stages: 1) removal of generic content; 2) tokenisation/the first anonymisation (masking personal names)/ lemmatisation; 3) the second anonymisation (masking email addresses, urls) and final clean. after step 1 there is no more generic content (e.g., sent from my iphone). in the 2 nd step, whenever possible, words are replaced by their lemmas. simultaneously, personal names are replaced with a predefined token "janez", and only words comprising of alphanumeric characters are kept. this was done using the conll-u format outputted by the classla python library, foot_1 . for every sentence segmented from the input text provided to the classla's processor, a verticalised list of tokens is given. in each row, there are 10 columns. for our needs, we used the third column that contains lemma of the original token, and the last column that contains information about a recognised named entity, if that was the case. during this second step of the cleaning procedure, we kept only the non-punctuation tokens, simultaneously replacing every token with its lemma. special case is when a token is recognized as a personal named entity, what we treat by replacing the original name with the common slovenian name "janez." finally, after the 3 rd step, all sequences of numbers were masked with the word "num", and using the scrubadub python library foot_2 remaining sensitive information was also masked by corresponding tokens predefined by the library's configuration. the reduced vocabulary of the final, processed dataset of emails was 13,943. ## model training and evaluation in this section we will describe the conducted experiments on the dataset of prepared emails, using the approaches proposed in section 3. ## conclusions and future work one way to further enhance the model would be to log the topic labels predicted by the model and see how many assigned topics were corrected by the person who received the email. this way the dataset would naturally grow, and we would get the feedback about number of cases the kc accepted the model's predictions, and in situations when that was not the case, what were the common mistakes and the reasons behind them. with the enlarged dataset, it would be possible not only to improve existing waterfall-1 model, but also to give another try to the other two approaches, since their bottlenecks in practice were lack of training samples.