Annotation Scheme for E nglish Argument Structure Constructions Treebank (2024)

Abstract: We introduce a detailed annotation scheme for argument structure constructions (ASCs) along with a manually annotated ASC treebank. This treebank encompasses 10,204 sentences from both first (5,936) and second language English datasets (1,948 for written; 2,320 for spoken). We detail the annotation process and evaluate inter-annotation agreement for overall and each ASC category.

## introduction the usage-based constructionist approach posits that language evolves from form-meaning pairings within linguistic structures. as language learners are exposed to a variety of inputs and outputs, they develop constructions of varying complexities @xcite . among these, argument structure constructions (ascs) represent core sentence meanings and are essential communication tools @xcite @xcite @xcite . in cognitive linguistics, ascs have played a central role in shaping cognition and language learning. language learners categorize experience into structured frames. during this process, ascs provide a conceptual framework, especially during the initial period of interaction with their surroundings. for example, with a transitive asc, language learners can articulate a "conceptual archetype" related to situations in which a subject affects an object (e.g., i kick the ball) @xcite . the significance of ascs has been further underscored by research for both first (l1; @xcite @xcite and second language (l2; @xcite @xcite learning. recently, computational linguists have also been interested in investigating how well constructional information is learned and induced by large language models @xcite @xcite @xcite . in this paper, we introduce an asc annotation procedure, its accompanying treebank (22,069 asc tokens), and a supplementary annotation manual (32 pages). our work includes both l1 and l2 english datasets, which serves two purposes: to augment the first version of the asc treebank @xcite , and to enable a comprehensive analysis across different linguistic contexts. the inclusion of l2 data is particularly helpful for researchers in the field of l2 acquisition (e.g., @xcite and for developers of automatic writing evaluation systems used in language proficiency assessments (e.g., @xcite . previous findings have suggested that the accuracy of partof-speech and dependency tagging models on l2 datasets improves when these models improves on l2 datasets when l1 training data is supplemented with even relatively small amounts of l2 data (e.g., @xcite . this dataset is expected to be instrumental in the evaluation and enhancement of tools for linguistic feature analysis, particularly for asc-related statistical measures, such as the association strength between verbs and ascs @xcite . additionally, the dataset would offer opportunities for investigating how large language models process ascs within both l1 and l2 contexts. ## related work research into asc annotation has been limited. some pioneering efforts have focused on specific asc types, such as a caused-motion asc @xcite or ascs with light verbs (e.g., take, make) @xcite . recently, @xcite developed a publicly available silver-annotated treebank of ascs based on previous related projects such as propbank @xcite , framenet @xcite , verbnet @xcite and universal propbank (up) for multilingual semantic role labeling @xcite . using a semi-automatic approach, they extracted largegrained semantic role frames from propbank (e.g., arg0-verbsense-arg1) then refined them (e.g., agent-verbsense-theme) using propbank mapping protocols with framenet and verbnet databases. they then manually assigned each semantic role frame to an asc (e.g., agent-verbsense-@xmath0. after resolving ambiguities and conducting spot-checks for errors, they categorized 94.1% of the ascs (@xmath1,437) foot_0 in the english web treebank (ewt) @xcite . the reliability of the semiautomatic annotation was preliminarily evaluated using a small sample of manually annotated sentences (@xmath2. the results suggested that the semi-automatic approach was reasonable, but far from perfect (simple agreement @xmath3.1%, @xmath4.884). while this approach presents an initial effort in building the silver-standard asc treebank, the accuracy of their semi-automatic approach is still unclear. beyond annotations, researchers have explored the relationship between ascs and language learning or understanding (recently processed by large language models) in various ways. these include psycholinguistic experiments @xcite , manual extractions based on researchers' judgments @xcite , automatic extractions that leverage the syntactic forms of the ascs (using automatic dependency annotations) @xcite . furthermore, some studies delved into broader linguistic constructions, identifying them through unsupervised approach that leverage statistical association measures @xcite . ## annotation scheme in the development of the annotation scheme, we first address two important levels of linguistic annotation @xcite . the first level is selecting the annotation unit, and the second level is choosing the asc categories to be employed during the annotation process. ## datasets we utilized l1 and l2 english datasets that were publicly and freely available and are part of universal dependency (ud) projects. both datasets use 4 for example, when a sentence a car hit the man is transformed into the man was hit by a car, the arguments and their semantic roles remain consistent (car-agent; man-theme), but only the focus within the statement is shifted. this allows for ascs from active voice categories, which have agent and theme arguments (i.e., transitive simple, ditransitive, causemotion, transitive resultative), may have respective passive counterparts and could potentially be grouped into separate passive categories. however, in our current categorization, we have grouped them all under the passive tag. we acknowledge that this is a methodological decision and that exploring distinct passive categories for different asc types could be a valuable direction for future research. the conll-u format, which presents sentences vertically with columns detailing morphological and syntactic attributes. we annotated asc tags into the 10 th misc column. due to limited annotation capacity, we randomly selected a subset of sentences from l1 and l2 written datasets. ## annotation the annotation project spanned about 8 months, from april 2023 to november 2023. ## conclusion in this study, we introduced a manually annotated asc treebank, complemented by a detailed annotation procedure. access to the asc treebank and the accompanying annotation manual is available here: https://osf.io/v75qu/?view_ @xmath5.