A corpus for Automatic Article Analysis (2022)

Abstract: We describe the structure and creation of the SageWrite corpus. This is a manually annotated corpus created to support automatic language generation and automatic quality assessment of academic articles. The corpus currently contains annotations for 100 excerpts taken from various scientific articles. For each of these excerpts, the corpus contains (i) a draft version of the excerpt (ii) annotations that reflect the stylistic and linguistics merits of the excerpt, such as whether or not the text is clearly structured. The SageWrite corpus is the first corpus for the fine-tuning of text-generation algorithms that specifically addresses academic writing.

## introduction the latest developments in natural language processing (nlp) and natural language generation (nlg) demonstrate a significant gain in performance on many domain-specific nlp tasks, by pretraining on a large corpus of text and fine-tuning using prompt engineering foot_0 in specific task @xcite @xcite . the sagewrite corpus is a manually annotated corpus created as a training dataset for the development of automatic text-generation and quality-assessment tools for academic writing foot_1 . when writing the different sections of an academic paper, authors often start by creating a rough draft or outline of what they want that section to say, which they then proceed to edit -and re-edit-until they are satisfied with it. an author writing the introduction of a linguistics paper may for example start by writing something along the lines of 1, which they will then proceed to edit until it looks something like 2: our primary goal is automate the process that leads from 1 to 2: we want to generate grammatical text starting from a rough draft of what the final text should look like. put differently, what we aim to do is streamline the revision process that leads from 1 to 2. what is required to generate 2 out of 1 stands halfway between natural language generation out of a limited input (qu, 2020) and advanced automatic paraphrasing @xcite . our secondary goal is to develop a classifier that can process scientific articles and automatically assess whether or not they exhibit certain qualities or flaws that we deem relevant to assess scientific publications, such as whether or not information is clearly presented and whether or not the text exhibits a good flow. again, this is in an attempt to streamline the revision process: if the stylistics shortcomings of a paper are flagged automatically, the author(s) of said paper can more readily address them. the sagewrite corpus was created to assist in the training of both of these functionalities. a first version of the corpus (version 0.1), consisting of 100 annotated excerpts, was published online in february 2022 foot_2 . we plan on increasing the size of this dataset as more excerpts get annotated. ## text selection the 100 manually annotated excerpts were extracted from various types of academic articles. to obtain the excerpts, we first created a database containing scientific articles taken from arxiv, pubmed, plus around 70 articles that we randomly selected from various disciplines in the humanities. the articles taken from arxiv were all dated march 2020 onwards. to extract the excerpts, we wrote a python program that automatically extracted excerpts of around 300 words from various points in an article. this was done to ensure that text belonging to various sections of a paper (e.g. introduction, abstract, conclusions) was included. text was always selected from the beginning of a paragraph until the end of a paragraph. the average length of the excerpts was 193 words. the excerpts were annotated by three annotators. as we wanted to work on academic texts, we hired annotators who had ties with academia and experience with academic writing. accordingly, one of our annotators was a ma student, one was a university lecturer and one had a phd degree. all annotators were also native speakers of (american) english. annotations were completed online on a dedicated platform where annotators could automatically log each part of the annotation for a given excerpt. annotators saw rotations consisting of one excerpt from a pubmed article, one from an arxiv article and one from our humanities articles. as we thought it would be interesting to see how different individuals would react to the same text, all annotators saw and hence annotated the same excerpts. ## structure of the corpus for each of the 100 excerpts, the corpus contains (a) three corresponding rough-draft versions of excerpt, each authored by a different annotator and (b) a list of tags that describe the stylistic and linguistic qualities of each excerpt. ## conclusions & limitations in this paper, we have illustrated the structure and creation process behind the sagewrite corpus, a manually annotated corpus created to support automatic language generation and automatic quality assessment of academic articles. version 0.1 of the corpus contains annotations for 100 excerpts taken from various academic articles; each excerpt was annotated by three different annotators, all of whom were native english speakers. for each of these excerpts, the corpus contains (i) a draft version of the excerpt (ii) a selection of tags that reflect the stylistic and linguistics merits of the excerpt. regarding drafts, on average drafts were around 26% shorter than the corresponding original text, although there was definitely variation among different annotators. more specifically, we saw that the ratio between length of the original excerpt and the corresponding draft was 0.77 for annotator 1, 0.81 for annotator 2 and 0.63 for annotator 3. this suggests that one should aim for drafts to be around 26±9.6% shorter than the original text; this value is particularly interesting in the context of automatically generating draft-like text from finished papers by selectively removing specific words and phrases, which is something we are also currently working on. our data also suggests that 23.6% is a good value to aim for when it comes to lexical diversity in the drafts: this is the value obtained by computing the mean value of the lexical diversity indexes for annotators 1, 2 and 3 (22%, 22% and 27% respectively). of interest is the fact that 22% was also the lexical diversity index of the original texts. an