Twenty-two Historical Encyclopedias Encoded in TEI : a New Resource for the Digital Humanities (2020)

Abstract: This paper accompanies the corpus publication of EncycNet, a novel XML/TEI annotated corpus of 22 historical German encyclopedias from the early 18th to early 20th century. We describe the creation and annotation of the corpus, including the rationale for its development, suggested methodology for TEI annotation, possible use cases and future work. While many well-developed annotation standards for lexical resources exist, none can adequately model the encyclopedias at hand, and we therefore suggest how the TEI Lex-0 standard may be modified with additional guidelines for the annotation of historical encyclopedias. As the digitization and annotation of historical encyclopedias are settling on TEI as the de facto standard, our methodology may inform similar projects.

## introduction encycnet is a tei-annotated corpus of 22 historical german encyclopedias from the early 18 th to early 20 th century, over 49,300,000 word tokens, with the goal of providing a resource for nlp and to add to the growing amount of historical, lexicographical german texts in consistently annotated xml. initial versions of the texts were provided to us by zeno.org in a proprietary xml schema, and we then developed a tei annotation schema that fits the diverse structures of these encyclopedias and at the same time compromises with existing encoding standards with regard to annotating lexicographic works. we then applied the annotation schema to our corpus using xslt (extensible stylesheet language transformations, a language for transforming xml documents into other xml documents). the tei methodology described in detail below is intended to connect our corpus to existing annotated, lexicographic corpora (while still accurately representing the original entry structures), as well as the semantic web, and also provide similar projects working with historic encyclopedias, in a variety of languages, with a reference point. in this paper, we first comment on the value of encyclopedia texts for humanities research, summarize related work, and provide an overview of the individual encyclopedias included in this corpus. we then present our choices for tei annotation and other transformation objectives that intend to homogenize the data and improve shareability and incorporation into related projects. specific encoding choices and corresponding examples from the corpus will be given when necessary. the encyclopedia corpus is openly accessible under a cc-by license on zenodo 12 while the xslt files are available on github. 3 following corpus publication, we will begin the creation of encycnet as a knowledge graph, which we hope will aid research in diachronic linguistic change in the german language, enriching existing language models with explicit semantic knowledge, and humanistic examination of changes in societal attitudes over time, with centuries of aligned encyclopedia texts as evidence. ## data the original texts of the twenty-two german encyclopedias in our corpus were initially made available to us by zeno as part of a larger, literary corpus for an earlier data transformation / preparation project. the focus of this previous project was a wide variety of literary text forms, meaning that the transformation process was designed to fit a wider variety of ambiguous xml markup (i.e. not specifically tailored for the encyclopedias which made up only part of their larger corpus). in encycnet, we are now able to attend to the specific structures and markup of encyclopedias texts while re-evaluating previous transformation decisions, most importantly choosing a tei @xmath0. for a more detailed view of the content and the size of the corpus, see the encycnet corpus has also been annotated with semantic web data, namely the automatic alignment of encyclopedia headwords with dbpedia @xcite , which contains millions of structured and classified entities from wikipedia, and germanet, a lexical-semantic net for the german language @xcite , together with its aligned wiktionary definitions @xcite . dbpedia spotlight @xcite provides a disambiguation algorithm, while the lesk algorithm @xcite was used for germanet. 10 in both cases, the encyclopedia glosses were used as context input for disambiguating the encyclopedia headword in semantic web classification. the original encyclopedias encode one encyclopedia entry in @xmath1, while the child @xmath2. the usage of these elements is, along with typography elements such @xmath3, fairly consistent and unambiguous. however, the internal structure of a gloss can be vastly differently annotated. most notably, <@xmath4, a line from a verse, or a list item. <@xmath5, but these classes do not align with their structural function, sometimes not even within the same encyclopedia. the same problem also applies to references. apart from the issue that references are not explicitly linked, the @xmath6, <plink>, and other typographical elements can indicate a footnote, gloss reference or footnote "anchor"-and these elements are also used interchangeably. in section 4, these two main issues (basic structure and references) in the transformation 10 we used dbpedia spotlight because it is specifically tailored to dbpedia. as no such tailored tool exists for germanet, we opted for lesk as the algorithm and its variants still belong to the standards of german word sense disambiguation @xcite . other tools for entity linking such as spacys entity linker or deeptype @xcite could have been used besides the approaches here; however at this early stage of the project our current claims do not benefit from marginal performance improvements. will be discussed in more detail. a markup conversion was therefore necessary to make working with these encyclopedia texts on a quantitative scale possible. for the xsl transformation process, we focused on preserving all data as a general rule. to ensure that no text instances were lost, we implemented a quantitative evaluation, i.e. compared the glosses of all original encyclopedias with their counterpart from their respective transformed encyclopedia with python. we also carried out a qualitative evaluation by transforming the results into simple html and randomly manually comparing parts with their facsimile equivalent. the glosses of the encyclopedias do not exhibit a typically lexicographical structure such as headword, grammatical features (and possibly many other kinds of features), short definition, usage example. we nonetheless chose to adopt the predominant encoding style that exists for lexicographic data (<@xmath7.g. @xmath8, so that researchers working with lexicographic data might be able to integrate these encyclopedias into their workflow more easily. an example of a minimal gloss in our encyclopedia corpus: <?xml @xmath9.0" @xmath10:@xmath11:@xmath12:@xmath13, a. d. lat. 1) im allgemeinen die gelehrsamkeit; 2) insbesondre pflegen einige dieses wort auf @xmath14. </@xmath15, there are also structure-focused elements that can be frequently found within the glosses, namely tables, verse, and lists. within tei lex-0 valid documents, these structures are not allowed. we therefore slightly modified the guidelines, so that glosses may contain these elements, as well. we also decided on another tei modification decision concerning the content @xmath16. in the original encyclopedia files, glosses are composed of multiple paragraphs, realized @xmath17. the paragraphs cannot be directly transformed @xmath18, because one paragraph usually does not align with one definition. as we did not want to lose the paragraph structure by compiling all adjacent paragraphs into @xmath19, we chose to @xmath20, which is not compliant with tei p5. we argue, however, that only allowing phrase-like elements within a definition is too restrictive for encyclopedias in general, as it is possible for a definition to consist of multiple paragraphs. for our encyclopedia structure this means @xmath21. footnotes are an essential aspect of this corpus, but so far, @xmath22, no explicit guidelines for handling footnotes in encyclopedias exist in a standardized form yet. petersen (2010) suggests replacing all footnote references with the corresponding footnote text, however footnotes with no references still need to be addressed then and footnotes with many references pointing to them might unnecessarily inflate the text. we instead chose to include any footnotes at the end of @xmath23, so that in case of two or more definitions, their corresponding footnotes will be grouped with them. single footnotes are again annotated with @xmath24. in these encyclopedias, there exist cases where footnotes are not placed at the end of entries. in the heiligenlexikon, for example, all footnotes are grouped at the end of one letter range. in wander's sprichwörter-lexikon, some entries contain notes in the middle of definitions. in both cases, we moved all footnotes to the end of their respective entries if necessary, thus altering the original structure to preserve a uniform annotation. we thus suggest that: all footnotes, like entries, should carry a unique @xml:id identifier, should only be allowed at the end of the related definition, and should carry the aforementioned @type attribute to distinguish them from other notes. ## conclusion and future work with the publication of encycnet's newly annotated encyclopedia corpus, we hope to provide a novel, digital, historical resource and advance the methodology of tei for encyclopedia texts. we argue that encyclopedias such as these should be treated just like any other lexical resource and thus should be annotated as such, even when the gloss structure they exhibit does not follow typical standards like any other lexical resource. our proposed methodology, a compromise between tei and tei lex-0, can be summarized as follows: allow a paragraph-like structure in definitions, allow less common structures such as lists and tables in definitions, and create explicit rules for footnotes within entries. we strive to include these findings within a future, revised version of the tei and tei lex-0. while our next step in this project is the transformation of the encycnet corpus into a knowledge graph structure, the corpus at this earlier stage may still be improved. currently, many headword features such as synonyms, hyper-or hyponyms are marked with highlight-elements, for example. with tei lex-0, these features could theoretically be annotated as such. automatically distinguishing tagged synonyms from other words marked with the same highlight tag is possible, but not realistic within the scope of an xsl transformation, which is why these features remain "unidentified" for now. a future version of the corpus, where features have been explicitly tagged with the help of appropriate nlp methods and another programming language, is still conceivable. another unresolved issue is the grouping of multiple senses in @xmath25, which can sometimes be the case for homonyms, but also entries listing multiple persons from the same family for example. ideally, these senses should be split into @xmath26, as tei lex-0 proposes. identifying and splitting such glosses is, similarly to identifying headword features within the definition, difficult to realize with xslt alone but feasible with other tools. we hope that encycnet, in both its tei-annotated corpus and planned rdf conforming knowledge graph form, may aid a wide variety of future research. encycnet adds a large new resource for historical german, and diachronic language change as well as language models may use encycnet as a source of additional historical and semantic knowledge. given the unique role of encyclopedias in cultural history, encycnet also offers a wealth of evidence for humanistic research. as shown in the alignment of headwords in