Multilingual Email Zoning (2021)

Abstract: The segmentation of emails into functional zones (also dubbed email zoning) is a relevant preprocessing step for most NLP tasks that deal with emails. However, despite the multilingual character of emails and their applications, previous literature regarding email zoning corpora and systems was developed essentially for English. In this paper, we analyse the existing email zoning corpora and propose a new multilingual benchmark composed of 625 emails in Portuguese, Spanish and French. Moreover, we introduce OKAPI, the first multilingual email segmentation model based on a language agnostic sentence encoder. Besides generalizing well for unseen languages, our model is competitive with current English benchmarks, and reached new state-of-the-art performances for domain adaptation tasks in English.

## introduction worldwide, email is a predominant means of social and business communication. its importance has attracted studies in areas of machine learning (ml) and natural language processing (nlp), impacting a wide range of applications, from spam filtering @xcite to network analysis @xcite . the email body is commonly perceived as unstructured textual data with multiple possible formats. however, it is possible to discern a level of formal organization in the way most emails are formed. different functional parts can be identified such as greetings, signatures, quoted content, legal disclaimers, etc. the segmentation of email text into zones, also known as email zoning @xcite , has since become a prevalent preprocessing task for a diversity of downstream applications, such as author profiling @xcite , figure 1 : okapi is composed of two building blocks: 1) a multilingual sentence encoder (xlm-roberta) to derive sentence embeddings; and 2) a segmentation module that uses a bilstm with a crf on top to classify each sentence into an email zone. since email communication is a worldwide phenomenon, all previous applications are in fact highly multilingual. despite this, email zoning literature remains english-centric and without a standardize zone taxonomy. to mitigate those problems, we make the following research contributions: to the best of our knowledge, okapi is the first end-to-end multilingual system exploring pretrained transformer models @xcite to perform email zoning. besides having multilingual capabilities, okapi is competitive with existing approaches for english email zoning, and attained state-of-the-art performance in domain adaptation tasks for english email zoning. the rest of the paper is organized as follows: section 2 presents an overview of the related literature. section 3 provides a comprehensive review of existing email zoning corpora, and introduces cleverly zoning corpus, our new multilingual email zoning corpus. section 4 describes the okapi model architecture. section 5 reports and discusses the results achieved. finally, section 6 concludes the paper. ## literature review chen et al. (1999) were one of the pioneers in the topic of email segmentation. looking at linguist and geometrical patterns, their work focuses on the identification of email signature. similarly, @xcite developed jangada, a supervised learning system that classifies each line using a conditional random field (crf) @xcite and a sequence-aware perceptron @xcite , that identifies signature blocks and quoted text from previous emails. @xcite proposed an email data cleansing system based on a support vector machine (svm) @xcite model that aimed at filtering the nontextual noisy content from emails independently of downstream text mining applications, based on hand-coded features. @xcite were the first to introduce a general segmentation schema for email text. segmentation of emails is a crucial part on their work, which aims at identifying the author's basic demographic and psychometric traits. in that work, the authors compared a range of ml algorithms together with feature selection to classify email segments into five functional parts, attaining improvements in the end task of auto profiling. later, @xcite formally defined the functional parts as email zones, describing the different segments inside email messages based on graphic, orthographic, and lexical features. @xcite also proposed zebra, an email zoning system based on a svm. in a posterior work towards detecting emails containing requests for action, @xcite used zebra to "zone" emails, considering only the zones that had relevant patterns to increase the accuracy of their request detection task. as email zoning surpassed its original purpose of signature identification and text cleansing into a more general task, @xcite extended its utility to thread reconstruction. inspired by zebra @xcite , the authors proposed quagga @xcite , a neural system with a convolutional neural network (cnn) @xcite ) to produce sentence representations followed by a recurrent neural network (rnn) @xcite . quagga was trained and evaluated on english emails from both the enron @xcite corpus and the public mail archives of the apache software foundation (asf) foot_0 , outperforming jangada and ze-bra. until very recently, email zoning resorted to small samples of mailing lists or newsgroup corpus and was limited to the english language. bevendorff et al. (2020) were the first to crawl email at scale, extracting 153 million emails from the gmane email-to-newsgroup gateway foot_1 in different languages such as english, spanish, french and portuguese foot_2 . the authors annotated email zones for a subset of gmane english emails and, due to the idiosyncratic characteristics of the corpora, they developed a more fine-grained zone classification schema with 15 zones. moreover, @xcite introduced an email zoning system, named chipmunk, that combines a bidirectional gated recurrent unit (bigru) @xcite ## email zoning corpora several corpora and zoning schemes have been proposed in the literature under different contexts. this section provides an overview of the existing corpora, hoping to make it easier to develop and compare new email zoning methods in the future. bevendorff et al. (2020) introduced the gmane corpus for email zoning foot_9 . even though the corpus is composed of 31 languages, the annotated emails are mostly in english, and their test set only contains a residual number of non-english emails (38 emails covering 13 different languages), which is insufficient for a consistent multilingual evaluation. due to the richness of the gmane conversations on technical topics, @xcite developed a more fine grained classification schema, considering the segmentation of blocks of code, log data and technical data. whilst also preserving most of the common zones introduced in previous works, they ended up with a total of 15 zones: closing, inline headers, log data, mua signature, paragraph, patch, personal signature, quotation, quotation marker, raw code, salutation, section heading, tabular, technical, visual separator. following the same zone taxonomy they also released a set of 300 english emails from the enron database dump. in both enron and gmane emails, the majority of the email segments belong to the paragraph and quotation zones. this being said, gmane has much more lines of quotation than paragraph, while enron is the other way around. overall, email zoning corpora show a great variability of zone taxonomies and most works have introduced new zones to face the nature of each email source or downstream task. the enron database dump has been the most used source to retrieve emails to build new corpus. on the other hand, the recent gmane raw dump of emails is multilingual and it contains various functional zones, which opens the door to new challenges in email zoning and multilingual methodologies. this section presents cleverly zoning corpus, the first multilingual email zoning corpus. to create the corpus, we searched the gmane raw corpus @xcite for portuguese (pt), spanish (es) and french (fr) emails. then, following the classification schema proposed by @xcite , we produced a total of 625 annotated emails. the annotation was carried out by two annotators. the first annotator was a native portuguese speaker and the second annotator a native spanish speaker, both with academical background in french and fluent in the third language. each email was annotated by both annotators using the tagtog foot_10 annotation tool. ## okapi architecture we propose okapi, an email segmentation model composed of two building blocks: a multilingual sentence encoder and a segmentation module. figure 1 shows the okapi architecture. ## results and discussion in this section, we analyse both multilingual and monolingual capabilities of okapi, considering various zoning corpora and annotation schemas. we evaluate the multilingual capabilities of okapi in a zero-shot fashion. for that, we trained the model with the gmane english corpus released by @xcite , and tested it with the cleverly multilingual corpus that we annotated for portuguese, spanish and french. ## conclusion to overcome the english-centric email zoning literature we propose okapi. besides having multilingual capabilities, the proposed model is competitive with existing approaches for english email zoning, and attained state-of-the-art performance in domain adaptation tasks of english email zoning. futhermore, to evaluate our model and to foster future research into multilingual email zoning, we release cleverly zoning corpus -a corpus with 625 emails annotated in portuguese, spanish and french.