Summarising Historical Text in Modern Languages (2021)

Abstract: We introduce the task of historical text summarisation, where documents in historical forms of a language are summarised in the corresponding modern language. This is a fundamentally important routine to historians and digital humanities researchers but has never been automated. We compile a high-quality gold-standard text summarisation dataset, which consists of historical German and Chinese news from hundreds of years ago summarised in modern German or Chinese. Based on cross-lingual transfer learning techniques, we propose a summarisation model that can be trained even with no cross-lingual (historical to modern) parallel data, and further benchmark it against state-of-the-art algorithms. We report automatic and human evaluations that distinguish the historic to modern language summarisation task from standard cross-lingual summarisation (i.e., modern to modern language), highlight the distinctness and value of our dataset, and demonstrate that our transfer learning approach outperforms standard cross-lingual benchmarks on this task.

## introduction the process of text summarisation is fundamental to research into history, archaeology, and digital humanities @xcite . researchers can better gather and organise information and share knowledge by first identifying the key points in historical documents. however, this can cost a lot of time and effort. on one hand, due to cultural and linguistic variations over time, interpreting historical text can be a challenging and energy-consuming process, even for those with specialist training @xcite . to compound this, historical archives can contain narrative documents on a large scale, adding to the workload of manually locating important elements @xcite . to reduce these burdens, specialised software has been developed recently, such as markus @xcite and docusky @xcite . these toolkits aid users in managing and annotating documents but still lack functionalities to automatically process texts at a semantic level. historical text summarisation can be regarded as a special case of cross-lingual summarisation @xcite @xcite , a long-standing research topic whereby summaries are generated in a target language from documents in different source languages. however, historical text summarisation posits some unique challenges. cross-lingual (i.e., across historical and modern forms of a language) corpora are rather limited @xcite and therefore historical texts cannot be handled by traditional cross-lingual summarisers, which require cross-lingual supervision or at least large summarisation datasets in both languages @xcite . further, language use evolves over time, including vocabulary and word spellings and meanings @xcite , and historical collections can span hundreds of years. writing styles also change over time. for instance, while it is common for today's news stories to present important information in the first few sentences, a pattern exploited by modern news summarisers @xcite , this was not the norm in older times @xcite . in this paper, we address the long-standing need for historical text summarisation through machine summarisation techniques for the first time. we consider the german|de and chinese|zh languages, selected for the following reasons. first, they both have rich textual heritage and accessible (monolingual) training resources for historical and modern language forms. second, they serve as outstanding representatives of two distinct writing systems (de for alphabetic and zh for ideographic languages), and investigating them can lead to generalisable insights for a wide range of other languages. third, we have access to linguistic experts in both languages, for composing high-quality gold-standard modern-language summarises for de and zh news stories published hundreds of years ago, and for evaluating the output of machine summarisers. in order to tackle the challenge of a limited amount of resources available for model training (e.g., we have summarisation training data only for the monolingual task with modern languages, and very limited parallel corpora for modern and historical forms of the languages), we propose a transfer-learning-based approach which can be bootstrapped even without cross-lingual supervision. to our knowledge, our work is the first to consider the task of historical text summarisation. as a result, there are no directly relevant methods to compare against. we instead implement two state-of-the-art baselines for standard cross-lingual summarisation, and conduct extensive automatic and human evaluations to show that our proposed method yields better results. our approach, therefore, provides a strong baseline for future studies on this task to benchmark against. the contributions of our work are three-fold: (1) we propose a hitherto unexplored and challenging task of historical text summarisation; (2) we construct a high-quality summarisation corpus for historical de and zh, with modern de and zh summaries by experts, to kickstart research in this field; and ( 3 ) we propose a model for historical text summarisation that does not require parallel supervision and provides a validated high-performing baseline for future studies. we release our code and data at https://github.com/pzoom522/histsumm . ## related work processing historical text. early nlp studies for historical documents focus on spelling normalisation @xcite , machine translation @xcite , and sequence labelling applications, e.g., part-of-speech tagging @xcite and named entity recognition @xcite . since the rise of neural networks, a broader spectrum of applications such as sentiment analysis @xcite , information retrieval @xcite , and relation extraction @xcite have been developed. we add to this growing literature in two ways. first, much of the work on historical text processing is focused on english|en, and work in other languages is still relatively unexplored @xcite . second, the task of historical text summarisation has never been tackled before, to the best of our knowledge. a lack of non-en annotated historical resources is a key reason for the former, and for the latter, resources do not exist in any language. we hope to spur research on historical text summarisation and in particular for non-en languages through this work. cross-lingual summarisation. the traditional strands of cross-lingual text summarisation systems design pipelines which learn to translate and summarise separately @xcite . however, such paradigms suffer from the error propagation problem, i.e., errors produced by upstream modules may accumulate and degrade the output quality @xcite . in addition, parallel data to train effective translators is not always accessible @xcite . recently, end-to-end methods have been applied to alleviate this issue. the main challenge for this research direction is the lack of direct corpora, leading to attempts such as zero-shot learning @xcite , multi-task learning @xcite , and transfer learning @xcite . although training requirements have been relaxed by these methods, our extreme setup with summarisation data only available for the target language and very limited parallel data, has never been visited before. ## methodology based on the popular cross-lingual transfer learning framework of @xcite , we propose a simple historical text summarisation framework (see fig. 3 ), which can be trained even without supervision (i.e., parallel historical-modern signals). step 1. for both de and zh, we begin with respectively training modern and historical monolingual word embeddings. specially, for de, following the suggestions of @xcite , we selected subword-based embedding algorithms (e.g., fasttext @xcite ) as they yield competitive results. in addition to training word embeddings on the raw text, for historical de we also consider performing text normalisation (norm) to enhance model performance. this orthographic technique aims to convert words from their historical spellings to modern ones, and has been widely adopted as a standard step by nlp applications for historical alphabetic languages @xcite . although training a normalisation model in a fully unsupervised setup is not yet realistic, it can get bootstrapped with a single lexicon table to yield satisfactory performance @xcite . for ideographic languages like zh, word embeddings trained on stroke signals (which is analogous to subword information of alphabetic languages) achieve state-of-the-art performance @xcite , so we utilise them to obtain monolingual vectors. compared with simplified characters (which dominate our training resources), traditional ones typically provide much richer stroke signals and thus benefit stroke-based embeddings @xcite , e.g., traditional '葉' (leaf ) contains semantically related components of '艹' (plant) and '木' (wood), while its simplified version ('叶') does not. therefore, to improve the model performance we also conduct additional experiments on enhanced corpora which are converted to the traditional glyph using corresponding rules (conv) (see § 5.3 for further details). step 2. next, we respectively build two semantic spaces for de and zh, each of which is shared by historical and modern word vectors. this approach, namely cross-lingual word embedding mapping, aligns different embedding spaces using linear projections @xcite . given parallel supervision is very limited in realworld scenarios, we mainly consider two bootstrapping strategies: in a fully unsupervised (uspmap) style and through identical lexicon pairs (idmap). while the former only relies on topological similarities between input vectors, the latter additionally takes advantage of words in the intersected vocabulary as seeds. although their historical and current meanings can differ (cf. § 3.3), in most cases they are similar, providing very weak parallel signals (e.g., 'krieg' (war) and 'frieden' (peace) are common to historical and modern de; '天' (universe) and '人' (human) to historical and modern zh). step 3. in this step, for each of de and zh we use a large monolingual modern-language summarisation dataset to train a basic summariser that only takes modern-language inputs. embedding weights of the encoder are initialised with the modern partition of corresponding cross-lingual word vectors in step 2 and are frozen during the training process, while those of the decoder are randomly initialised and free to update through back-propagation. step 4. upon convergence in the last step, we directly replace the embedding weights of the encoder with the historical vectors in the shared vector space, yielding a new model that can be fed with historical inputs but output modern sentences. this entire process does not require any external parallel supervision. ## conclusion and future work this paper introduced the new task of summarising historical documents in modern languages, a previously unexplored but important application of cross-lingual summarisation that can support historians and digital humanities researchers. to facilitate future research on this topic, we constructed the first summarisation corpus for historical news in de and zh using linguistic experts. we also proposed an elegant transfer learning method that makes effective use of similarities between languages and therefore requires limited or even zero parallel supervision. our automatic and human evaluations demonstrated the strengths of our method over state-of-the-art baselines. this paper is the first study of automated historical text summarisation. in the future, we will improve our models to address the issues highlighted in this study (e.g. zero-shot patterns and language change), add further languages (e.g., english and greek), and increase the size of the dataset in each language. de №11 story die arbeiten im hiesigen arsenal haben schon seit langer zeit nachgelassen, und seitdem die perser so sehr von den russen geschlagen worden sind, hört man überhaupt nichts mehr von kriegsrüstungen in den türkischen provinzen. die pforte hatte nicht geglaubt, daß rußland eine so starke macht nach den ufern des kaspischen meeres abschicken, und daß der krieg mit den persern sobald eine so entscheidende wendung nehmen würde. alle kriegerischen nachrichten, die wir jetzt aus den türkischen provinzen erhalten, erstrecken sich blos auf die bewaffneten räuber-korps, die in der gegend von adrianopel noch immer ihren unfug fortsetzen, der auch wohl nicht eher aufhören wird, bis die pascha's selbst bestraft worden sind, die die räuber beschützen. -im anfange dieses monats erschien eine russische fregatte am eingange des schwarzen meeres, ward durch sturm vor den türkischen forts vorbei in den kanal getrieben, ohne daß die kommandanten dieser forts ihr den geringsten widerstand entgegen stellen konnten, und legte sich, bujukdere gegenüber, vor anker. sobald der kapitän-pascha dies erfuhr, verfügte er, daß jene kommandanten abgesetzt werden sollten, und beschwerte sich bei dem hiesigen russischen minister darüber, daß jenes kriegsschiff sich unterstanden habe, wider alle stipulationen der traktaten in den kanal einzulaufen. nachdem aber der zufall, wodurch dies geschehen ist, näher aufgeklärt war, widerrief der kapitän-pascha die befehle, die gegen die kommandanten der an dem kanal gelegenen forts erlassen wurden. auch ward auf ansuchen des russischen gesandten der gedachten fregatte aller mögliche beistand geleistet, um sich repariren, und nach der krimm, woher sie gekommen war, zurückkehren zu können. -die gesandten, welche die pforte schon seit 2 jahren nach wien und berlin bestimmt hat, sind noch immer hier; dies beweiset, daß alle schwierigkeiten in rücksicht dieser missionen noch nicht gehoben sind der nach paris bestimmte türkische gesandte wird aber, wie es heißt, bald abreisen. -zwei sehr angesehene französische offiziers, die in türkischen dienst getreten waren, sind wieder aus demselben entlassen worden. the porte would not have thought that russia would send such a powerful force to the shores of the caspian sea, and that the war with the persians would at the same time take such a decisive turn. all belligerent news, that we now receive from the durkian provinces, extends only to the armed robber corps, which are in the area of adrianopl still continuing their mischief, which is still unlikely to end until the pashas themselves, who protect the robbers, have been punished. -at the beginning of this month a russian frigate appeared at the entrance to the black sea, was driven by a storm past the durkian forts into the channel, without that the commanders of this fort could oppose it with the slightest resistance, and (the russian frigate) presented itself across from bujukdere at anchor. as soon as the captain pasha found out about this, he decreed that those commanders should be deposed and complained to the local russian minister about that that that grieg ship had dared to enter the canal, against all stipulations of the tracts.but after the coincidence, by which this happened, had been more clearly clarified, the captain-pasha recalled the orders, which would be enacted against the commanders of the fort on the canal. also, at the request of the russian confession, the intended frigate was given all possible assistance in order to repair itself and to be able to return to grimm, whence it had come. expert der kaiser versuchte, durch verhandlungen seine interessen gewährzuleisten. inzwischen boten die türken wider versprechen den rebellen unterstützung (the emperor tried to safeguard his interests through negotiations. meanwhile turks broken the promise and provided support to the rebels.) idmap+norm es befindet sich schon etliche tage her von der crone schweden ein abgeordneter inconitum allhier , aber noch unbewusst in was negotio . am verwichenen montage ist von ihrer käyserl . allda gegen i . (a few days ago there was a member of parliament incognito here from the royal family of sweden, but still unconsciously in some business. on the elapsed monday is from their royalty all against i.) uspmap+norm es befindet sich schon etliche tage her von der crone schweden ein abgeordneter inconitum allhier , aber noch unbewusst in was negotio . am verwichenen montage ist von ihrer käyserl . m . an den abgestanden (a few days ago there was a member of parliament incognito here from the royal family of sweden, but still unconsciously in some business. on the elapsed monday is from their royal m to the stale ...) de №34 story jhre königl. majest. befinden sich noch vnweit thorn/ vnd seynd cosakische deputirte vnter wegs/ jhr factum bey seiner majest. zu justificiren, vnd wegen jhrer treu versicherung zu thun. von den fridens. tractaten zwischen pohlen vnd schweden ist noch wenig zu melden. seithero die pohlen bey marienburg den schweden eine schantz/ der kessel genant/ abgenommen/ ist nichts weiters vorgefallen/ auch hiesiger stadt völcker vor dem haupt noch nichts tentirt, jedoch sagt man daß noch dise woche etwas vorgehen werde/ so bald nur alle battereyen in den 3 quartieren fertig/ vnd die mörser darauff gebracht worden/ vmb solches mit feur zu bezwingen/ weil mit dem schiessen doch nichts zugewinnen/ vnd der sturm vnmöglich zu wagen ist/ daß aber das brau-vnd proviant hauß darin in brand geschossen/ vnd die darinnen befindliche cavallerie also ruinirt werden/ daß sie keinen außfall mehr thun können/ ist gewiß/ deßgleichen hat der obriste zaphlizky mit 2000. mann den elbingern daß viehe weggetriben/ welche darauff mit 500. mann außgefallen/ solches wider zu erobern/ seynd aber mehrentheils nidergemacht/ vnd 6. vornehme officierer neben vielen gemeinen gefangen worden. so ist auch auß churland über memmel sichere zeitung einkommen/ daß herr general duglas nur 2000. mann nach liffland gebracht/ vnd pautzke sich mit accord an die pohlen ergeben habe/ seynd also von den schweden in mittau noch 300. mann übrig/ deren ergebung man nechstens zu verenehmen hoffet/ zumahlen selbige formaliter belägert seynd/ vnnd keinen succurs zuvermuthen. den gefangnen hertzogen von churland haben die schweden wider in liffland nach revel gebracht/ dahero zu seiner erledigung hoffnung gemacht werden will.