E mil. R ule Z ! – An exploratory pilot study of handling a real-life longitudinal email archive (2023)

Abstract: An entire generation that predominantly used email for official communication throughout their lives is about to leave behind a significant amount of preservable digital heritage. Memory institutions in the USA (e.g. Internet Archive, Stanford University Library) recognised this endeavor of preservation early on, therefore, available solutions are focused on English language public archives, neglecting the problem of different languages with different encodings in a single archive and the heterogeneity of standards that have changed considerably since their first form in the 1970s. Since online services enable the convenient creation of email archives in MBOX format it is important to evaluate how existing tools handle non-homogeneous longitudinal archives containing diverse states of email standards, as opposed to often archived monolingual public mailing lists, and how such data can be made ready for research. We use distant reading methods on a real-life archive, the legacy of a deceased individual containing 11,245 emails from 2010 to 2023 in multiple languages and encodings, and demonstrate how existing available tools can be surpassed. Our goal is to enhance data homogeneity to make it accessible for researchers in a queryable database format. We utilise rule-based methods and GPT-3.5 to extract the cleanest form of our data.

## introduction we live in a time when many people's email correspondence is preserved as a digital legacy. as these people have mostly used email for official communication throughout their lives (because of their habits) it is possible to look back over the majority of their electronic written communication @xcite . however, such digital email legacy raises a number of moral and legal questions. for example, data protection and privacy legislation, such as the general data protection regulation (gdpr) (european union, 2016), demands a challenging compliance process and encourages institutions not to implement long-term preservation for their own safety, which may become a relevant issue in the future. furthermore, in most cases, there is not necessarily a complete separation between private and corporate emails @xcite . while from the technical perspective, email has undergone many changes since its inception and has become heterogeneous in terms of standards and implementation @xcite , which makes longitudinal archives difficult to process, analyse and aggregate. while different tools can be used to create email archives in different formats @xcite in our case it is assumed that the mail archive is already available as an mbox file (e.g. google takeout or similar services). we got legal authorisation to use (without publishing partially or fully) a real-life email archive of a deceased public figure's correspondence (hungarian, romanian, english) for our pilot study to uncover and solve possible technical difficulties. our aim was to create a methodology that could successfully process a real-life mbox file that contains a longitudinal correspondence and produce an output that could be searched, visualized, and analyzed by researchers interested in the author's official communication. ## evaluation of the available tools several tools exist for processing mbox archives and all of them are built by adopting alternative ap-proaches with specific perspectives in mind. since these approaches are very diverse, each tool has its own strengths and weaknesses which should not be ignored when pursuing our goal @xcite . in this section, we introduce a selection of the tools we evaluated (divided into two classes) before we decided to write our own. for each test we used two mbox files. one is an artificially created demo mbox file (willison, 2022b) containing only two emails. it lacks misspelled emails, notifications, circulars, returned error messages from mailer-daemons (i.e. mail delivery systems), etc. the other is the aforementioned real-life mbox file that we want to process covering more than one decade and therefore containing many of the errors mentioned above. all examined tools more or less handle the important metadata headers (from, to, cc, bcc, subject, date, etc.) but most of them fail when it comes to decoding the textual data that particular email bodies contain. we can classify these tools into two main classes: a) those often abandoned and only half done, and b) the complex monolith solutions of institutions that are hard to use without appropriate expertise. we will use this classification in the rest of this chapter. the first group of programs @xcite @xcite @xcite promises to process mbox files and insert the output into various data formats (e.g. csv, json, sql) for further processing. for the artificial data set, this group of tools worked well but with our non-artificial test data, they failed to produce usable output (i.e. contained raw undecoded string fragments e.g. in quoted-printable or base64 form) if any. we examined the root cause of the errors and it turned out that these programs apply false assumptions and are beyond repair. in general, if one wants to process a very complex data set, their use is not recommended without technical expertise, and we decided it was better to start with a clean slate. for people without technical skills, viewing an mbox file in mozilla thunderbird or mbox viewer is a great opportunity to interact with the data (i.e. read it), as these tools are halfway between the two mentioned groups: they can handle non-artificial data and do not require expertise (neither technical nor archiving) to operate. however, we found that the export functionality of mbox viewer is half broken: it can produce a csv file for our non-artificial data set, but we could not properly load it with ms excel or libreoffice calc probably because the garbled delimiters, limiting further deeper analysis. we assumed that it did not escape characters with syntactical meaning in tabular format, breaking the data structure. in the second group, we tested two wellestablished solutions that promise more than extracting mbox to common data formats. mailbagit (university at albany, 2023b) can import various kinds of email archives and convert them to mailbag format (university at albany, 2023a) while exporting the data to other formats (txt, warc, pdf, etc.). for our non-artificial data set it yielded a lot of error messages, but could produce a good csv file with the mandatory metadata and individual files for the payload of each message. when necessary it uses automatic character encoding detection but detects absurd encodings (e.g. turkish code page), and there is no way to correct such mistakes manually due to its complexity. in the long run, the bugs may be fixed in some of the aforementioned programs by their maintainers. however, as they did not fit our primary goals, we decided to implement our own lightweight solution for this specific archive which can be easily extended with little technical skills if an error occurs while processing other archives. ## our method to accommodate storing a sequence of multiple emails, the format of raw mbox data contains encoded fragments, i.e. one-byte long ascii characters with no syntactical meaning chunked into 80byte long lines, which allows easy handling. non-ascii byte sequences are encoded by a binaryto-text encoding scheme such as quoted-printable or base64 which when decoded returns the original values (i.e. string or binary) of the individual emails. one email record is composed of multiple case-insensitive key-value-style (standard and nonstandard) headers and the payload consisting of recursively embedded payload parts (e.g. html, text, or binary attachment). to get the character representation of non-ascii, byte-represented text segments, bytes need to be decoded with the supplied character encoding. in some cases, no or wrong character encoding was specified e.g. binary data erroneously has character encoding. we used the built-in email module of python which contains utility functions for most of the aforementioned steps. proceeding carefully from the headers to the payload we realised they posed different challenges, therefore in the following, we discuss them separately. ## using gpt for curation the remaining problems turned out to be the result of non-standard behaviour, which was easier to solve with a solution more capable of handling semi-structured data, therefore we chose to use gpt-3.5 @xcite (to facilitate affordable reproducibility) adopting a few-shot methodology (see @xcite . upon replying to, or forwarding an email the old text is separated by the main metadata of the email (in string form formatted in a non-standard, language-specific way) from the new text which is usually written at the top of the email body, however, in some cases at the discretion of the user, the reply is inlined (inserted between the lines of the old email). this process often results in concatenated email-body texts requiring separation. to solve these issues, we identified the following tasks: a) decoding the use of non-standard string form metadata inside the email body, and b) separating concatenated emailbody texts. furthermore, we also used gpt-3.5 to fix the remaining edge-case encoding errors that were caused by "lossy" decoding (e.g. replacement characters, omitting faulty byte sequences, incorrectly decoded characters). gpt-3.5 performed best on metadata extraction, where it was able to extract from and to addresses, dates, and subject strings even with sim-ple prompts. handling concatenated email-body texts, however, proved to be a more difficult task, therefore, for testing their separation accuracy, we chose 100 examples with varying numbers of previous email text recursively included in the payload. 77 were successfully separated, failing mostly on inputs with more complex text structures that did not contain proper helper annotation (e.g. ">" used for indenting previous email text lines) (see figure 3 . in appendix a.1.). we found that many of the erroneously encoded emails that were left had the same problem: an automatic mechanism (antivirus) had pasted a footnote to the payload, but with wrong encoding, which caused the decoding of the whole message to fail. although gpt-3.5 successfully handled these cases, we also implemented a rule-based method of splitting the text and applying another encoding to the footnote part, fully eliminating this type of encoding problem. only a few complex encoding errors remained that neither gpt-3.5 nor a rulebased method could solve, as they were products of several layers of incorrect processing, and the resulting character combinations were indecipherable even for humans. our experiments show that a rule-based workflow could potentially be expanded by using large language models, if tasks are well compartmentalised and split into separate problem areas. ## visualising the resulting data with the data cleaned and normalised to the limit, as a final step to facilitate access to authorised digital humanists who prefer visual representations of data, we loaded it into an off-the-shelf application suitable for n-gram based data exploration (n-gram trend viewer @xcite ). one example of exploring the metadata-rich text-based corporausing only metadata that is safe from compromising gdpr -is the frequency of different email providers that the owner of the account interacted with over time (see figure 1 .). naturally, those who have legal access to the data can make more in-depth queries that the system supports. ## evaluation and conclusion our experiments with the automatic character encoding recognizer systematically resulted in turkish code pages, which can be safely ruled out from the set of possibly used languages and code pages, therefore, we opted to manually observe each oc-figure 1 : usage count of selected anonymized email domains (from, to, cc, and bcc) over time. there was a drop in traffic around 2017 when the subject passed away, however, strangely enough after a period of inactivity the account started interacting with several domains again, most likely due to someone gaining access. our method and epadd both found that many email addresses had different names associated with them, which enables these names to be added as an alias along with a canonical name to a semantic database (e.g. the "also known as" field in wikidata @xcite ) for later use. epadd used english word lists with little success to recognise named entities in the text. the lists can be changed but due to the monolithic nature of the program, the clearly not state-of-the-art method cannot. we conclude our pilot project a success foot_1 , as we recovered most of the errors and created an intuitive webui for the mbox data to help researchers explore the email archive. to open up more possibilities in the future, the conversion of emails to standard tei xmls (derose, 1999) is an option worth exploring as it could additionally handle the complex philological aspects of inline replies. trying other mbox files is also desirable to make our tool more robust and handle more non-standard headers since it was built with extensibility and customisability in mind.