Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis (2020)

Abstract: This paper introduces the Webis Gmane Email Corpus 2019, the largest publicly available and fully preprocessed email corpus to date. We crawled more than 153 million emails from 14,699 mailing lists and segmented them into semantically consistent components using a new neural segmentation model. With 96% accuracy on 15 classes of email segments, our model achieves state-of-the-art performance while being more efficient to train than previous ones. All data, code, and trained models are made freely available alongside the paper.

## introduction email is perhaps the most reliable and ubiquitous means of digital communication. notwithstanding the mainstream adoption of social media for private communication as of about 2010, email prevails unrivaled for workplace communication and beyond. compared to social media, however, emails have attracted much less research attention in the fields of computational linguistics, natural language processing, and information retrieval. key reasons for the neglect can be found in the presumed difficulty of obtaining emails at scale, the lack of open technologies to parse them, and that, despite their importance, they are hardly considered en vogue. although mailing lists as a rich and accessible source for emails have been tapped before, this has never been done at scale. our contributions in this respect are (1) the webis gmane email crawl 2019, a crawl of more than 153 million emails from a wide range of mailing lists, (2) the chipmunk email segmenter, a newly developed end-to-end neural model, and (3) the complete preprocessing of the crawled emails using our model to construct the largest corpus of "ready-to-use" emails to date. our corpus encompasses more than 20 years worth of discussions on a diverse set of topics, including important political and societal issues. we believe that providing the research community with access to clean and preprocessed communication data from emails will foster open research in several areas, such as the analysis of dialogs and discourse, stylometry, language evolution, argument mining, as well as information retrieval, and the synthesis of conversations and argumentation. ## related work for research purposes, the three primary sources of email data are public mailing lists and newsgroups, volunteered or leaked private email datasets, and email databases at companies and service providers. the westburylab usenet corpus @xcite was crawled between 2005 and 2011. more widely employed has been the "20 newsgroups" corpus @xcite . the w3c corpus compiles the public w3c mailing lists @xcite , @xcite examined 8 years of patch submissions to the linux kernel mailing list, and @xcite inspected the process of standardization across ietf bodies via its mailing lists. the cspace corpus consists of 15,000 student dialogs volunteered for research during a management course at cmu @xcite . all of the above have been extensively analyzed @xcite @xcite , yet the most widely studied corpus remains the leaked enron corpus @xcite , built as part of the u.s. ferc's investigation into the enron corporation. it has been subject to studies on speech act and dialog analysis @xcite , named entities @xcite , and word usage patterns @xcite , among many others. another recently leaked dataset comprises the clinton emails that surfaced during the 2016 u.s. presidential election @xcite . regarding email data at companies and service providers, not many researchers are able to disclose their datasets @xcite . regardless of their source, emails are usually unstructured and difficult to process even for human readers @xcite . thus, many approaches have been proposed for cleansing newsgroup and email data. as one of the earliest, de @xcite developed a specialized method for detecting and removing signatures based on typical text indicators. @xcite developed a high-accuracy model for detecting blocks of noncontent in emails using a mixture of svm models and hard-coded rules. an unsupervised approach was employed by @xcite , who applied a noisy channel model for filtering out noncontent. similarly, @xcite used spell checking techniques for uncovering technical artifacts like source code, disentangling them from the main content. a more general approach, befittingly named zebra, was published by @xcite , who split messages into a series of structural and semantic "zones", such as author text and signature. finally, @xcite developed quagga, the first neural end-to-end model inspired by lampert et al.'s zebra, which showed very substantial performance improvements. most machine learning-based approaches rely on classifying lines of text, either by detecting the start and the end of structural blocks with specialized models, or by assessing each line individually via its surrounding context. with the increase in machine-generated emails, recent studies have shifted their focus away from dialogs and towards parsing and categorizing @xcite or threading notifications @xcite , as well as automated template induction @xcite @xcite . ## the webis gmane email corpus 2019 our dataset was crawled from gmane, 2 a popular email-to-newsgroup gateway, which allows users to subscribe to mailing lists via the nntp newsgroup protocol that formed the basis for the usenet. while gmane's web portal has been offline for years and was recently replaced by a minimal website under a new domain name, the newsgroup portal is still alive and messages from active mailing lists arrive every day. unlike a mailing list server, a newsgroup server keeps an archive of messages, allowing a user to download the history of a newsgroup even if they did not participate in it from 2 https://news.gmane.io or rather: nntp://news.gmane.io the beginning. traditional newsgroup servers often have a limited retention period, though fortunately, gmane archived all messages since its launch in 2002. about a million messages date back even further to the year 2000 and a small number even to the early 90's. the latest message in our corpus is from mid-may 2019, which is when we stopped crawling. considering this enormous time span and the uncertain future of gmane, we see archiving these messages as both a great research opportunity and an attempt at preserving our digital heritage. following the style of the usenet, gmane groups are ordered in a hierarchy of subjects under the common gmane root. this hierarchy makes it easy to categorize mailing lists into topical domains giving a rough overview of what is being talked about. the majority of groups is of a generally technical nature (e.g., in gmane.comp or gmane.linux), a large number of other categories exists, most notably culture, politics, science, education, music, games, and recreation. below these main categories, a plethora of individual subjects are found. a cursory topic modeling study reveals not only software development discussions, but also debates about environmental issues, climate change, gender equality, mobility, health, business, international conflicts, general political concerns, philosophy, religious beliefs, and many more. ## the chipmunk email segmenter cleansing email plaintexts is laborious and first requires splitting them into different functional and semantic segments (also sometimes called zones). our first attempt at this was a re-implementation of the classic approach by tang et al. despite our best efforts, its handcrafted feature set, and the need to train two individual svms for each type of content block caused generalizability and scalability issues on our much larger and more diverse dataset. also, a context window of three lines was not nearly enough to reliably identify all types of content blocks, and making the window larger did not yield satisfying results due to the simplicity and the lack of shared weights among the individual models. we also needed a much more fine-grained segmentation, which not even the more recent neural approach by repke and krestel could deliver without substantial changes, so it was decided to develop a new email segmenter. we identified 15 common segments recurring in emails: (1) paragraphs (main content), (2) salutations, (3) closings, (4) quotations, (5) quotation markers (quotation author and date), (6) inline email headers, (7) personal signatures, (8) automated mua signatures (i.e., mail user agent, but also mailing list details or advertising), ( 9 ) source code, (10) source code diffs, (11) log data, and (12) technical noise (e.g., inline attachments or pgp signatures), (13) semi-structured tabular data, (14) ornaments (e.g., separator lines), and (15) structural section headings (e.g., in a call for papers). we annotated segments in a stratified sample of 3,033 emails from a range of different groups, totaling 170,309 line annotations. annotated segments are mostly unambiguous so that a single annotator can produce consistent and high-quality annotations in multiple correction passes. although the sample is technically multilingual, most emails are in english. of the 3,033 emails, we set aside 300 for model validation and extracted another sample of 1.5 million emails and concatenated them to a single file of 80 million lines (2.8 gib). here we replaced all email addresses with the token @email@, all urls with @url@, mapped numbers to the digit 0, replaced all hexadecimal values with @hash@, runs of four or more indenting spaces with @indent@, split words on special characters (mainly for tokenizing quotations and source code), and normalized unicode characters to nfkc. we used this processed dump to train a fasttext embedding @xcite with a default vector dimension of 100. ## ethical considerations the distribution of email data raises ethical concerns, such as possible violations of privacy and legal requirements, which we addressed to the best of our ability. all emails in our corpus are from public mailing lists and by policy, gmane only accepts such lists whose users are comfortable with their emails being publicly readable. at the time of writing, the original messages in our corpus are openly available to anyone through the nntp interface and other mailing list archives. nevertheless, we took measures to avoid abuse of the readily parsed and compiled form of the data, one being the aforementioned anonymization of email addresses to inhibit trivial mass harvesting. furthermore, we enforce a strict release policy in compliance with the gdpr academic exemptions. access to the data is granted solely to researchers and academic institutions and we prohibit further distribution for non-academic purposes.