Annotating Decomposition in Time: Three Approaches for Again (2023)

Abstract: This submission reports on a three-part series of original methods geared towards producing semantic annotations for the decompositional marker “again”. The three methods are (i) exhaustive expert annotation based on a comprehensive set of guidelines, (ii) extension of expert annotation by predicting presuppositions with a Multinomial Naïve Bayes classifier in the context of a meta-analysis to optimize feature selection and (iii) quality-controlled crowdsourcing with ensuing evaluation and KMeans clustering of annotation vectors.

## introduction the goal of this paper is to present a series of three original methods in the context of, and first handson results for, ascertaining theoretically relevant ambiguities in readings of historical data on decomposition. decompositional adverbs (e.g., again and its relatives in many languages) have attracted attention not only in the context of formal analyses (say, structural vs. lexicalist) since they are insightful, if not uncontroversial, in their own right. they also touch on the representation of events, presuppositions, and more generally the way the structural and the meaning components of particular languages are to be related (cf. @xcite @xcite , among many others). moreover, recent inquiries into diachronic formal semantics have crucially shown that diachronic data can not only receive motivated theoretical analyses but are also able to elucidate synchronic debates that could not be solved otherwise thus far @xcite . however, major practical issues with much needed diachronic data are the costly process of extraction w.r.t. high-quality data, their reliable annotation, stronger validation (than, say, the intuitions of individual researchers), and, when possible, partially automatic amplification/replication. the structure of this paper is as follows: in section 2, we start off with a discussion of the english adverb again and its main readings -as relevant to the discussion at hand. next, we discuss the three methods for producing semantic annotations for again: in section 3, we go into detail regarding the procedure behind exhaustively annotating its various readings with a team of expert annotators based on syntactically parsed diachronic corpora of english (ranging throughout recorded history, from old to modern english; our concrete focus here lies 'only' on the last two to four centuries). the first slice of this semantic annotation, i.e., all 1,901 uses of again in the penn parsed corpus of modern british english (2nd ed., 'ppcmbe2', cf. @xcite , is ready to be shared with the community along with a tool to be merged with users' own instances of the ppcmbe2. the second method discussed in this paper (section 4) seeks to tap into the semantically enriched data and extend the expert annotation: we discuss the performance of a multinomial naïve bayes classifier in predicting the main readings of again in ppcmbe2. we do so in the context of a meta-analysis exploring the best-performing feature combinations based on a set of 16 different features of three different major types (features based on our semantic annotation, structural features drawn from the pre-existing syntactic parsing, and 'naïve' features based on the textual surface). we cover the third and final approach in section 5. it reports on what we call an 'informed crowdsourcing experiment', which we designed to explore crowd aptitude for providing nuanced semantic annotations on diachronic data -natural language data for which our ('informed') crowd workers can have no actual native speaker intuitions whatsoever (as the bearers of truly native intuition are dead). here we report on the performance of kmeans clustering of the crowdsourcing data when compared to our gold standard of expert annotations. we close with a general discussion in section 6. ## again and its readings the natural language phenomenon at the core of all annotation tasks discussed here is the english adverb again and its well-documented ambiguity. consider the example corpus data (1) and ( 2 ): (1) i. [a]ll the plants then must be examined, (token 345) [...] ii. and those which are planted in pots, should in the following year's bloom be again examined (349) (fallowfield-1791-2,34.349, 'gardening calendar') (2) i. he sat really lost in thought for the first few minutes; (token 565) [...] ii. he [mr. knightley] hesitated, (618) iii. got up. (619) [...] iv. and he sat down again; (633) (austen-1815-2,169.633, 'emma') the adverb again in (1) has a repetitive reading ('rep'): an event of the same kind (examining plants) is presupposed. the again in (2) has a restitutive/counterdirectional readings ('res/ct'), i.e., the again here does not presuppose a sitting-down event by mr. knightley but an event in the opposite direction. this presupposition is satisfied in (2-iii) where [he] got up. the result state of the sittingdown event restores a state that held at a time prior to reference time. note, that in (2) we could naturally assume that mr. knightly must have sat down at some point prior to the reference time for (2-iv). in fact, we can infer as much from the context (2-i) but it is never asserted in the prior contexts. thus, in the domain of relevant times (as far as available in the context) we don't find the repetitive presupposition satisfied in the context. while the result state is overtly spelled out in (2-iv), this need not always be the case for res/ct uses, cf. (3) where again -on a decompositional analysis -has access to the result state of its predicate: ## conclusion at the current state of the technical possibilities explored and as far as the natural language phenomenon at hand is concerned, a gold standard cannot be substituted wholesale by either machine learning-based predictions or experimental data. the first upshot is that the gold standard itself must be as solid as possible (we sketched our detailed approach above, and we are open to constantly improving it). at the same time, we think that our two additional case studies are quite telling even if their performance was expectedly lower. the significance of such extensions is obvious when it comes to the annotation of larger amounts of data (be it for decompositional markers or other annotational tasks; of course, for low-frequency phenomena, the use of larger corpora or alternative methods becomes a necessity). the feature-based approach (section 4) then becomes relevant, also for cases in which the syntactic annotation is missing such as the eebo type of corpora in our object-language english. in such a case, some of the syntactic features we have used in our approximations can be translated, e.g., in terms of precedence (an instance of again that precedes its modifying predicate is typically also higher in structure etc.). overall, however, we believe that the human approach, i.e., the type of informed crowdsourcing we have utilized, is the most promising variant of annotational support when one strives to cover more data than one's team can handle or for gaining more certainty empirically. the straightforward advantage is that the relatedness in the languages at hand can be used even if the 'nativeness' of the actual participants is not available. some of our results have indicated that more distant periods in time do not necessarily become worse in the annotational performance. on a conceptual level, there is also initial evidence from independent areas of semantic change (cf. @xcite that speakers adapt astonishingly well in simulated situations of change. finally, even if certain targeted readings are comparatively low performing, one can still place a crowdsourcing approach at the start of an annotation pipeline. by validating crowd annotations with a gold standard for a subset of the data, one can learn which data (i) needs a closer review, (ii) which data needs less attention in a review, and (iii) which data could benefit from a thorough review due to inherent indecisiveness of the crowd.