Annotation of lexical bundles with discourse functions in a S panish academic corpus (2023)

Abstract: This paper describes the process of annotation of 996 lexical bundles (LB) assigned to 39 different discourse functions in a Spanish academic corpus. The purpose of the annotation is to obtain a new Spanish gold-standard corpus of 1,800,000 words useful for training and evaluating computational models that are capable of identifying automatically LBs for each context in new corpora, as well as for linguistic analysis about the role of LBs in academic discourse. The annotation process revealed that correspondence between LBs and discourse functions is not biunivocal and that the degree of ambiguity is high, so linguists’ contribution has been essential for improving the automatic assignation of tags.

## introduction lexical bundles (lb) in academic english have been the object of many studies @xcite @xcite . although lbs are strictly defined as recurrent lexical sequences with high frequency and dispersion, their linguistic value comes from the discourse function that they fulfil. it is well known that the mastery of these lbs, such as it should be noted ('to emphasize'), as can be seen ('to resend'), or it is clear that ('to show certainty'), is crucial in academic writing. in english, lexical resources have been proposed (e.g. @xcite in order to offer aid especially to novice writers. however, for academic spanish few resources are available. in light of this, the aim of this paper is to discuss the annotation of a spanish academic corpus with the subset of lbs that have a discursive function, referred here as the umbrella term of formula. to the best of our knowledge, it is the first spanish corpus with this type of annotation. even though there is an extensive research on spanish discourse markers, focused on a lexicographic description @xcite or on its automatic identification and classification @xcite , we do not know any spanish corpus with annotations of academic formulae. our research is related to connectivelex @xcite , although it is based on the tagset of penn discourse treebank 3.0 @xcite . likewise, we must mention da @xcite , the spanish corpus annotated with the discourse relations used in rhetorical structure theory @xcite . the purpose of the annotation described here is to obtain a gold-standard corpus to train and evaluate computational models on the automatic identification and classification of academic formulae in new corpora. if generally multiword units have been especially difficult in nlp, formulae have the extra difficulty that they deal with discourse functions that seem more slippery for language models. although many formulae are compositional, they must be also considered as phraseological units because they work as a whole and cannot be replaced by synonymous expressions that are unnatural; for instance, in english we cannot replace to put it differently with to use some different expressions or to say it in a different way. in our approach (mel'čuk, 2015) multiword expressions (or phrasemes) include compositional and non-compositional phrases. likewise, in the studies developed for academic english such as @xcite , formulae include compositional and non-compositional expressions but all of them are considered formulaic sequences. in what follows, we describe the process of annotation and human validation, where the main challenge has been to select the proper discourse function to ambiguous formulae. ## dataset this section describes the corpus and the formulae list of academic spanish used for the present study. ## annotation procedure the procedure followed for annotating academic formulae is summarized in fig. 1 . the first step involved using the academic formulae list with their discourse functions to identify their occurrences in the corpus through a patternmatching technique. as for the second step, the annotated corpus was split in 15 blocks of ca. 120,000 word tokens each, with the aim of mixing texts from different authors and disciplines. the 15 blocks were uploaded to inception @xcite , a tool that has been used for the manual evaluation of the automatically annotated corpus to validate the results. as illustrated by fig. 2 , once the corpus is uploaded, the main page for the annotator shows the text, the formula underlined and the discourse function's tag. besides the tagged text, the annotator is provided with a panel with access to the 39 discourse functions. here, the annotator can change the discourse function, delete it, as well as associate a new discourse function to a formula that needs to be added. thus, the main task for annotators has been to validate whether discourse functions were correctly tagged by pattern-matching and to revise whether annotated lbs were proper formulae in all contexts, because different situations could have emerged. a more detailed description of each situation is given in section 4. the 15 blocks of texts were distributed among three annotators, in such a way that each annotator had 5 individual annotation blocks, a joint annotation (two annotators who worked on the same block but independently) and, finally, a consensus annotation. the consensus annotation is obtained from applying a curation process to joint annotations. more precisely, the annotator starts a process of "neutralization" of mismatching annotations by changing the discourse function of a formula that was wrongly assigned, by adding a tag in a formula that was not identified, or by removing the formula because it does not behave as such in given contexts. instead of errors, different annotations might be seen as plausible variations among annotators due to different reasons, as pointed out by @xcite . once this exhaustive task has been completed, an annotated corpus of ca. 1,800,000 words was obtained (88% of harta-exp), including 360,000 words of consensus annotations. the product obtained from the curation process is a set of peerreviewed texts that have been used to calculate inter-annotator agreement. ## results and discussion manual examination of the automatically annotated corpus has been time consuming and a demanding task for annotators. it lasted around 180 hours only for the individual annotations, at least 12 hours for each block of 120,000 words. in addition to the validation in inception, we must take into account the previous long and exhaustive linguists' task of identifying formulae and assigning the proper discourse functions. consequently, we can say that linguists' contribution has been essential to identify academic formulae and their functions in corpora as a first step, as well as to improve a part of the automatic annotation (11%) foot_1 , which ensured the high quality of the data in the gold-standard corpus. the time invested led to an average of 414 changes per ca. 3,858 tagged formulae in each block that underwent manual examination. because we wanted to ensure there was coherence among decisions made by annotators, we calculated the agreement for the 3 joint annotations. results have shown high values for the raw agreement (number of agreed items/nº of total items) of the consensus texts ranging between 89% and 92%, so it provided a positive general overview about the annotation process. krippendorff α @xcite was also performed in order to calculate the amount of agreement that was attained above the level expected by chance or arbitrary coding. similarly, values for joint annotations revealed a high level of agreement: α=0.885 for block 1; α=0.898 for block 2, and α=0.925 for block 3. therefore, this agreement was considered as an acceptable reference for annotating the rest of blocks individually. the main findings provided by the annotation process suggest that annotators dealt with four different types of changes: (i) formulae that annotators judged they do not behave as such in given contexts; (ii) ambiguous formulae associated to two discourse functions; (iii) occurrences of nested formulae where only the longest string was identified; and (iv) occurrences of new formulae as different morpho-syntactic forms of existing ones. as we can see in fig. 3 below, the most faced situations by annotators have been discarding lbs (i), that stands for the 50%, followed by changing the discourse function of ambiguous formulae (ii), that represents the 41% of the total amount of changes. conversely, nested bundles (iii) and (iv) addition of new morpho-syntactic forms describes only the 4-5%. regarding the first type of change (i), it is worth emphasizing that some of the occurrences of 12 formulae, such as es más ('in addition'), were discarded because in some specific contexts they were not associated to any discourse function. for instance, es más can be used to 'add information' (1), but in contexts such as (2) it is a lb that is not associated to any discourse function, so it must be removed: as for the second type of change (ii), it turned out that the discourse function chosen for 27 foot_2 ambiguous formulae (two possible functions) was not much more frequent than the other function, so that it involved several changes in annotation. it was especially the case of strings like en relación con ('with regard to'), which depending on its position in the sentence is associated to different functions. thus, en relación con and the like, when used sentence-initially normally serve to 'introduce the topic' of a sentence, whereas in sentence-internal distributions they usually head some 'delimiting' modifier. in this regard, the function 'introduce the topic' was substituted for 'delimiting' 499 times, way above other functions, which were modified 30 times on average during the validation process. the difference of switching times from 'delimiting' to other discourse functions in ambiguous formulae is shown in fig. 4 . in this respect, 'delimiting' frequently alternates with 'introduce the topic' (in-topic) as well as with 'quoting and reporting' (indsource), but hardly ever switches to 'compare' (comp): such type of change is reflected also in the formula de acuerdo con ('according to'), that is used for 'quoting and reporting' (3) or as a 'delimiting' marker (4): takada y lourenço en 2004, las características generales de esta disciplina [...]. 'according to takada and lourenço in 2004, general features of this discipline [...].' (4) '[...] tiene que ver con estrategias y prioridades de actuación de cada biblioteca de acuerdo con su particular circunstancia local.' '[...] it has to do with strategies and priorities of action of each library according to their particular local circumstance.' another example of ambiguity is found within the formula en torno a ('around'), that in some contexts it is used for 'delimiting' (5), but in other contexts to 'mitigate' a quantity (6): concerning the third type of change (iii), annotators dealt with some cases where two formulae were nested but only the longest one was automatically tagged by pattern-matching. for instance, in como podemos observar en la tabla ('as we can see in the table'), we find como podemos observar en (4-gram) and en la tabla (3-gram), so the preposition en ('in') belongs to both formulae. in those cases, annotators selected the formula they considered the most relevant for each context and assigned them its discourse function. finally, the fourth type of change (iv) relates to new formulae that were not identified in the automatic extraction but were of particular interest. new formulae were selected if they met the frequency criterion and were morpho-syntactic variants of already registered ones. for instance, expert writers tend to use the complete and discontinuous formula por una parte, por otra parte ('on the one hand, on the other hand'), but we found instances where the abbreviated and grammatically correct counterpart was used (por otra; lit. 'on the other') and that were not in our initial list. thus, 11 different types of morpho-syntactic variants identified during this phase were added to the initial list of 985, that sums up a total amount of 996 formulae. ## conclusions this paper described the annotation process of a new spanish academic corpus of 1,800,000 words annotated with 996 formulae, that are assigned to 39 different discourse functions. this process is the result of a combination of an automatic annotation and a manual validation. the corpus obtained can be considered a valuable resource because besides of being manually validated, inter-annotator agreement showed high values of coincidence between decisions made by annotators. automatic techniques used to identify specific vocabulary from corpus are a good starting point to provide researchers with preliminary data to work with. the same applies for annotating occurrences of formulae in corpora. however, we found that identification and annotation procedures still needed a human validation in order to obtain a gold-standard corpus as a benchmark. especially in the annotation, ambiguity has demonstrated to be present: many instances with lbs that behaved as a formula in some contexts but not in others were found, as well as different formulae that are associated to two possible discourse functions depending on the context were frequent. further work aims to use the gold-standard corpus obtained from this study to train and evaluate computational models that are capable of identifying automatically adequate lexical bundles in new corpora, as well as for lexicographic and linguistic studies.