H ist RED : A Historical Document-Level Relation Extraction Dataset (2023)

Abstract: Despite the extensive applications of relation extraction (RE) tasks in various domains, little has been explored in the historical context, which contains promising data across hundreds and thousands of years. To promote the historical RE research, we present HistRED constructed from Yeonhaengnok. Yeonhaengnok is a collection of records originally written in Hanja, the classical Chinese writing, which has later been translated into Korean. HistRED provides bilingual annotations such that RE can be performed on Korean and Hanja texts. In addition, HistRED supports various self-contained subtexts with different lengths, from a sentence level to a document level, supporting diverse context settings for researchers to evaluate the robustness of their RE models. To demonstrate the usefulness of our dataset, we propose a bilingual RE model that leverages both Korean and Hanja contexts to predict relations between entities. Our model outperforms monolingual baselines on HistRED, showing that employing multiple language contexts supplements the RE predictions. The dataset is publicly available at: https://huggingface.co/datasets/Soyoung/HistRED under CC BY-NC-ND 4.0 license.

## introduction relation extraction (re) is the task of extracting relational facts from natural language texts. to solve re problems, diverse datasets and machine learning (ml) methods have been developed. earlier work limits the scope of the problem to sentencelevel re, in which the task is to predict a relationship between two entities in a single sentence @xcite @xcite @xcite . however, such a setting is impractical in real-world applications where relations between entities can exist across sentences in large unstructured texts. therefore, document-level re datasets for general and biomedical domains have been introduced (li english* entity: person, location, clothes, datetime relation {sbj_kor: "경영 관사", obj_kor: "돈의문", sbj_han: "京營官舍", obj_han: "敦義門", label: "nearby", evidence_kor: [3, 4], evidence_han: @xcite @xcite @xcite } metadata book_title: "연도기행", book_volume: "연도기행 하", text_chapter: "일록(日錄) ○ 병신년 순치(順治) 13년 (1656, 효종 7) 12월", title: "16일(기축)", writer: "송계", year: 1656, copyright: "한국고전번역원 | 이민수 (역) | 1976" figure 1 : an example from histred. only one relation is shown for readability. the text is translated into english for comprehension (*). relation information includes (i) subject and object entities for korean and hanja (sbj_kor, sbj_han, obj_kor, obj_han, (ii) a relation type (label), (iii) evidence sentence index(es) for each language (evidence_kor, evidence_han). metadata contains additional information, such as which book the text is extracted from. @xcite @xcite @xcite . docred-h @xcite is human-annotated, while docred-d is generated by distant supervision. historical indicates that the dataset contains historical contents, and relation means the dataset is built for the re task. input level is whether the input sequence is a single sentence (sent.) or multiple sentences (doc.). # of doc. represents the number of documents, # of sent. is the number of sentences, and # of tok. is the average number of tokens in a document using the mbert tokenizer. in response, we introduce histred, a documentlevel re dataset annotated on historical documents for promoting future historical re studies. histred contains 5,816 documents extracted from 39 books in the yeonhaengnok corpus (see section 2 for details). as described in our contributions are summarized as follows: ## dataset construction to the best of our knowledge, histred is the first re dataset in the historical domain; thus, there is no consensus regarding the dataset construction process on the historical corpus. in the process of designing our dataset, we collaborated with experts in the linguistics and literature of hanja to arrive at a consensus. this section describes how we collaborated with the domain experts to construct histred without losing annotation quality. ## data analysis in this section, we analyze various aspects of histred to provide a deeper understanding and highlight several characteristics of our historical data. varying sequence length as described in section 2.4, the input text length can be altered via the sequence level (sl). annotation procedure statistics since our dataset construction consists of annotation and cross-checking steps, we summarize the statistics of this procedure. as shown in ## bilingual relation extraction model unlike translation between modern languages, such as translation from english to korean, historical records have been translated hundreds of years after their creation. as a result, the gap between ancient and present makes the translation task from hanja into korean difficult. also, the translated texts can vary across translators; thus, the domain experts read both hanja and korean texts to fully understand the original text. based on this observation, we hypothesize that understanding the bilingual text would help a model extract valuable information and design our bilingual re model. as shown in figure 2 , our model is a joint model of two separate encoders for hanja and korean, along with a cross-attention block from the transformer architecture @xcite . for a document d of length n in hanja and m in korean, we have d @xmath1, where x and y are input tokens of each document. we use the plm encoder to obtain contextualized embeddings: h kor , h han . based on these hidden representations, we adopt the multi-head crossattention block, which consists of a cross-attention layer and residual connection layer @xcite . for instance, when the encoder process the hanja text, we set the query as the hanja token and the key and value to the korean tokens. crossattended representation h ′ is defined as evaluation metric following previous work in re @xcite , precision, recall, and micro-f1 scores are used for evaluating models. hyper-parameters hyper-parameters are set similarly to the bert-base model in @xcite . the size of the embedding and hidden vector dimensions are set to 768, and the dimension of the position-wise feed-forward layers to 3,072. all encoders consist of 12 layers and 12 attention heads for each multi-head attention layer. also, the cross-attention block consists of 8 multi-head attention, and α is set as 0.5 when we get the final logits (l out ). however, when sl is 2, 4, and 8, α is set to 0.6. the batch size for all experiments is set to 8. the learning rate is set to 5e-5 using the adam optimizer @xcite . all models are trained for 200 epochs and computed on a single nvidia tesla v100 gpu. computational details are in appendix b.1. ## analysis in this section, we introduce a real-world usage scenario and analyze our model on histred, describing how our historical dataset can be utilized in detail. ## conclusion in this paper, we present histred, a documentlevel relation extraction dataset of our historical corpus. our study specializes in extracting the knowledge in yeonhaengnok by working closely with domain experts. the novelty of histred can be summarized by two characteristics: it contains a bilingual corpus, especially on historical records, and sl is used to alter the length of input sequences. we also propose a bilingual re model that can fully exploit the bilingual text of histred and demonstrate that our model is an appropriate approach for histred. we anticipate not only will our dataset contribute to the application of ml to historical corpora but also to research in relation extraction.