Where it’s at: Annotating Verb Placement Types in Learner Language (2025)

Abstract: The annotation of learner language is an often ambiguous and challenging task. It is therefore surprising that in Second Language Acquisition research, information on annotation quality is hardly ever published. This is also true for verb placement, a linguistic feature that has re- ceived much attention within SLA. This paper presents an annotation on verb placement in German learner texts at different proficiency levels. We argue that as part of the annotation process target hypotheses should be provided as ancillary annotations that make explicit each annotator’s interpretation of a learner sentence. Our study demonstrates that verb placement can be annotated with high agreement between multiple annotators, for texts at all proficiency levels and across sentences of varying complex- ity. We release our corpus with annotations by four annotators on more than 600 finite clauses sampled across 5 CEFR levels.

## introduction acquiring the different options for verb placement, and more generally constituent order, in finite clauses of german is a well-known challenge for learners and a frequent object of second language acquisition (sla) studies on german @xcite @xcite @xcite @xcite @xcite . one key reason to study verb placement is its theoretical significance for theory building. while learners' interlanguage (il) has been found to be highly variable, it is also known to be systematic @xcite . processability theory (pt) @xcite posits that german verb placement options are acquired in a fixed order 1 https://github.com/dakoda-project/annotating_ verb_placement_with_ths by all learners regardless of other factors, such as learners' age or educational background. this systematicity is attributed to the fact that it depends on the processability of the grammatical structures producing the different orders. these grammatical mechanisms build on each other to the effect that there is no skipping or re-ordering possible among the five major placement options that pt focuses on. unsurprisingly, such strong claims are contested within the field of sla @xcite . a second important reason to verify claims about the acquisition of verb placement empirically is that some common instruments for proficiency testing that are used in educational settings rely on verb placement as a key diagnostic (e.g. mika-d in austria @xcite ): a theory whose application affects educational trajectories in the real world had better be sound. recently, @xcite published specifications for the computational implementation of a system detecting verb placement types as a prerequisite for an automated analysis of learner (l2) language development on a large scale. however, that paper did not show that a key prerequisite for automation holds, namely that verb placement analysis can be performed reliably by human annotators. moreover, as far as we could ascertain, agreement on verb placement analysis also has never been evaluated within sla, where most studies on the topic seem to be based on single coding by one of the authors. while the above specifications suggest that this should be an eminently doable task on proficient native (l1) data, we think it needs to be tested empirically how well human coders agree on verb placement in learner text, which is orthographically, semantically, and/or morpho-syntactically non-canonical. as an illustration, consider example (1). the last two tokens in (1) cannot combine if taken at face value: wir 'we' is a nominative case personal pronoun but treffen 'meeting' is a noun. in this and other similar cases, any labeling of the learner data rests on adopting a particular interpretation of what the learner was trying to say. 2 in the remainder of this paper, we argue for using an annotation protocol where verb placement annotations are performed in conjunction with the annotation of target hypotheses that can explicate the understanding of difficult learner productions such as (1). to that end, we present the design and results of an annotation study on essay data of l2 german learners at different levels of proficiency. we focus on the following research questions. how good is agreement between human annotators on verb placement overall? can we observe differences related to the texts' proficiency levels (given in terms of cefr ratings)? is there an effect of sentence complexity on agreement? ## theoretical background to motivate our study design, we first present the sla theory whose verb placement inventory we use for annotation and then discuss the use of target hypotheses in the analyses of learner data. ## annotating verb placement with ancillary target hypotheses broadly speaking, we can distinguish two types of difficult cases for verb placement analysis: (a) productions whose meaning is understandable but which are not obvious to normalize and (b) productions whose meaning is difficult to understand. our introductory example (1) exemplifies the former situation: while we can understand the semantic import of the learner's utterance (especially in view of the task context of this production), the learner's production is syntactically incoherent and its normalization is not obvious. figure 1 shows several possible ths for the learner production in (1). the different ths themselves have different verb placement annotations and they lead to different conclusions about verb l erstens gibt es viele frage muss man im voraus zu überlegen. firstly gives it many questions must one in advance to consider 'first, there are many issues that one has to think about in advance.' th0 erstens gibt es viele fragen, die muss man sich im voraus überlegen. an example of the second type of difficult case is found in (3). here the verb sagen may or may not be taken to have a complement clause (cf. possible interpretations a-c). depending on how the two finite verbs/clauses relate, we make different assumptions about the type of clause and verb placement we need to assign to the finite form wurde. given cases such as (1) and (3), it seems unavoidable to explicate coders' target hypotheses: simply comparing annotations on the learner layer without reference to ths risks making the annotations appear less valid and reliable than they might be. as a correlate, for instances where multiple ths are plausible, multiple gold standards for verb placement must be entertained. 4 4 while we are concerned directly only with the analysis beyond explicating the understanding attributed to the tokens on the learner layer, ths serve a second function that is important within the language acquisition context: they spell out the structure that was expected in context. for instance, in (4), the learner uses svo (verb-second) in the complement clause. a possible th for this clause would re-order it to final placement of the finite verb (daß immer mehr menschen lieber alleine als in einer großfamilie leben). ## annotation analysis in the final dataset, we have 849 tokens annotated as verbs on the learner layer l. on the target hypothesis layer th, we have 847 instances. figure 3 shows the combinations of sentence type and verb placement found on the learner layer. what we observe are mostly combinations that would be expected for german. for instance, inv(ersion) structures are commonly found in questions and declaratives, while verbfinal (vend) structures are found exclusively in subordinate clause types. however, we can also observe some unexpected combinations involving svo in various types of subordinate clauses. ## conclusion our corpus -the multiply annotated verb placement corpus (mavpc) -is the first dataset for sla studies where verb placement is multiply coded and where target hypotheses are available as ancillary annotation rationales. we have shown that on essay data sampled from two corpora and stratified across cefr levels, high levels of agreement could be achieved for the core annotation categories of finiteness, sentence type, and verb placement. this holds both on the raw learner text and on the ths. the corpus features not only the raw annotations of four annotators but also one or more gold standard labels that reflect contextually plausible interpretations of clausal structure and verb placement. the data can serve as a test set for automatic systems performing verb placement analysis. while the high agreement on the learner layer might suggest that ths are not needed at all, we would caution against that conclusion. the concomitant annotation of ths may improve agreement on the learner layer in a way that might be absent if no ths were constructed. also, our data represents just one written text type and a limited set of l1s. further studies on additional written text types and especially on spoken language are needed. in lernerkorpora des deutschen [data competencies in german as foreign language / german as a second language: exploring language technologybased approaches to the analysis of l2 acquisition levels in learner corpora of german]), grant number 16dkwn035. we would also like to thank jamila bläsing and iulia sucutardean for their support in annotating the data. finally, we thank nils reiter for a fruitful exchange on the details of our agreement evaluation.