Leveraging pre-trained language models for linguistic analysis: A case of argument structure constructions (2024)

Abstract: This study evaluates the effectiveness of pre-trained language models in identifying argument structure constructions, important for modeling both first and second language learning. We examine three methodologies: (1) supervised training with RoBERTa using a gold-standard ASC treebank, including by-tag accuracy evaluation for sentences from both native and non-native English speakers, (2) prompt-guided annotation with GPT-4, and (3) generating training data through prompts with GPT-4, followed by RoBERTa training. Our findings indicate that RoBERTa trained on gold-standard data shows the best performance. While data generated through GPT-4 enhances training, it does not exceed the benchmarks set by gold-standard data.

## introduction argument structure constructions (ascs) are lexicogrammatical patterns at the clausal level. they consist of an argument structure and a main verb, with each argument contributing to the clause's meaning @xcite . the characteristics of asc use, such as frequency and/or the strength of association between a verb and its argument structure, have been actively explored in previous studies on first language (l1) and second language (l2) learning and assessment @xcite @xcite . to effectively model human language learning/development using asc features, ascs must be reliably identified in target texts. recent studies have shifted from manual (e.g., @xcite to automatic asc analyses (e.g., @xcite @xcite . however, building automated asc annotation systems has proven challenging due to syntactic ambiguity. for example, while syntactic analyses would represent the clauses (1) she ran [to @xcite the mountains] and (2) she ran [in the mountains] identically based on the form (i.e., subject-verbprepositional phrase structures), they imply different meanings and represent distinct asc types. in case @xcite , the prepositional phrase [to the mountains] is an argument that completes the meaning by specifying the goal of the movement. in contrast, in case (2), the phrase [in the mountains] modifies the location of the event, as illustrated in figure 1 . one potential reason for this mismatch is that human language often employs a pre-built form-meaning schema at the clausal level @xcite , which can be challenging to capture from a bottom-up perspective. a top-down approach, directly assigning asc types based on their clausal contexts, is therefore likely more effective than a bottom-up approach. recent advancements in pre-trained language models (plms) may offer a promising solution to these challenges, given their effectiveness in storing sentence-level contextual knowledge, as well as part-of-speech and syntactic knowledge within their word embeddings @xcite . the follow-up empirical question is whether these models can reliably capture specific types of ascs, both with and without top-down annotations provided by trained human annotators focusing on the linguistic characteristics of clausal forms. to address this, the current study explores the use of plms for identifying ascs, evaluating three methodologies: @xcite supervised training with a encoder model (roberta) using a gold-standard asc treebank, (2) promptguided annotation of unlabeled data with a decoder mode (gpt-4), and (3) prompt-guided generation of training data with gpt4, followed by training with roberta. ## methodology 4.1 datasets in this study, we utilize two treebanks, namely the silver (v1) and gold (v2) versions of the asc treebank. the first silver version @xcite includes 26,437 asc tokens that were semiautomatically annotated 5 . the second gold version @xcite includes 22,069 manually annotated asc tokens 6 . the sentences in this treebank were sampled from the english web treebank (etw) @xcite , l2-written (esl-wr) @xcite , and l2-spoken (esl-sp) @xcite treebanks, which are all part of the universal dependencies project @xcite . given the relatively small representation of l2-written (esl-wr) and spoken (esl-sp) data, training, development, and test sets were resampled with a 34/33/33 distribution. the l1 (ewt) sentences retained their original sections and were roughly distributed at 80/10/10. ## conclusions this study highlights the potential of integrating plms into linguistic analysis frameworks, particularly for examining the characteristics of ascs in the context of modeling l1 and l2 learning/development. roberta, when trained on goldstandard datasets, demonstrated the best performance in asc annotations, underscoring the importance of comprehensive, high-quality annotated data. additionally, the use of gpt-4 for promptguided annotation and data generation offered some insights into the effectiveness of synthetic data in model training. while these methods did not surpass the f1 scores of the baseline model trained solely on gold-standard annotations, they proved effective in identifying and processing certain types of ascs. future directions this study serves as a promising foundation for automated annotation systems in both l1 and l2 language contexts. however, it did not directly assess the effectiveness of asc annotation in automatic writing evaluation or feedback systems, which represent critical avenues for future research and applications of nlp for building educational applications.