Building a Manually Annotated H ungarian Coreference Corpus: Workflow and Tools (2022)

Abstract: This paper presents the complete workflow of building a manually annotated Hungarian corpus, KorKor, with particular reference to anaphora and coreference annotation. All linguistic annotation layers were corrected manually. The corpus is freely available in two formats. The paper gives insight into the process of setting up the workflow and the challenges that have arisen.

## introduction the main motivation for building a coreference corpus was the fact that it is always interesting to investigate the behavior of a linguistic phenomenon in real texts. a manually annotated corpus is useful not only for linguists, but also for training and evaluating tools. korkor, a hungarian coreference corpus presented in this paper contains multiple linguistic annotation layers, such as disambiguated pos-tags, lemmata and morphological features (of two morphological tagsets) and dependency relations. all of these ordinary linguistic annotations were corrected manually, as well as the anaphora and coreference annotations. representativeness is an important feature of a corpus if we expect the tools trained on it to work with predictable quality in different genres and domains. however, in the current phase of the research, only two sources of texts were involved, since this phase aimed more at setting up the corpus building workflow and producing the necessary tools. the resource is available under cc-by 4.0 license to enhance accessibility, usability and extensibility. korkor can be found in the following github repository: https://github.com/ vadno/korkor_pilot . apart from the corpus itself, the whole workflow with detailed instructions, the annotation guidelines and the tools prepared in the frame of this project are also available in the github repository to provide help for anyone having the necessary resources (financial resource, human labor, raw material) to continue the project or create a new, similar corpus based on it. ## the workflow the building process was set up as a pipeline, in which as many steps were intended to be automated as possible. human work was used for supervising and -if needed -correcting the annotation. certain processing steps were carried out by the documents tokens (conllup) tokens (xtsv) huwiki 62 16,739 18,262 globv 32 7,760 8,799 total 94 24,499 26,581 although human annotation in multiple cycles is certainly a labour-intensive method, minor faults are easier to fix, than muddled tangles. thus, human annotators corrected the annotations in three phases. the steps of the workflow were the following (tools used are in parentheses -steps where no tools are given were carried out with scripts developed within the project): the annotators reported every problem and quetion arising, therefore the annotation guidelines became finer and more detailed which sped up and made manual work easier. the workflow includes multiple conversion steps between file formats, as the output of a certain step may differ from the expected input format of the following one. each step of the workflow is specified below.