I ndo CL : Benchmarking I ndonesian Language Development Assessment (2024)

Abstract: Recently, the field of language acquisition (LA) has significantly benefited from natural language processing technologies. A crucial task in LA involves tracking the evolution of language learners’ competence, namely language development assessment (LDA). However, the majority of LDA research focuses on high-resource languages, with limited attention directed toward low-resource languages. Moreover, existing methodologies primarily depend on linguistic rules and language characteristics, with a limited exploration of exploiting pre-trained language models (PLMs) for LDA. In this paper, we construct the IndoCL corpus (Indonesian Corpus of L2 Learners), which comprises compositions written by undergraduate students majoring in Indonesian language. Moreover, we propose a model for LDA tasks, which automatically extracts language-independent features, relieving laborious computation and reliance on specific language. The proposed model uses sequential information attention and similarity representation learning to capture the differences and common information from the first-written and second-written essays, respectively. It has demonstrated remarkable performance on both our self-constructed corpus and publicly available corpora. Our work could serve as a novel benchmark for Indonesian LDA tasks. We also explore the feasibility of using existing large-scale language models (LLMs) for LDA tasks. The results show significant potential for improving LLM performance in LDA tasks.

## introduction the advancement of artificial intelligence has significantly enhanced the evolution of educational technology and the utilization of computer-assisted learning @xcite ). 1 our code and corpus can be obtained from https://github.com/gklmip/indocl . recent interest has surged in applying natural language processing (nlp) and machine learning (ml) to evaluate language development in both first (l1) and second (l2) language acquisition. the goal is to analyze learners' linguistic attributes and the progression of their language skills across various modalities and stages @xcite . language development assessment (lda) @xcite is a critical task in the field of language acquisition that focuses on evaluating the progress of language learners over time. as shown in figure 1 , a student wrote two paragraphs at two different time periods, denoted as text a and text b. if the model determines that text b was written after text a, it is considered that text b is better written than text a, indicating an improvement in the student's writing skills. conversely, if the model believes that the writing order is that the student wrote text b before text a, then text b is considered inferior to text a, indicating a decline in the student's writing skills. for lda, the majority of research has been focused on high-resource language. @xcite and @xcite conducted investigations into the development of writing skills among german-speaking students from elementary to secondary school levels. @xcite proposed a method to track the evolution of written language competence in l2 spanish and l1 italian learners. to the best of our knowledge, few existing lda studies focus on low-resource languages, such as indonesian, and there is a lack of available lda corpora for these languages. early research in lda has explored enhancing model performance through text feature extraction @xcite . due to the complexity and tediousness of the feature extraction process, some research @xcite adopts the fully data-driven approach for lda tasks. this type of approach tracks language development by automatically extracting linguis-figure 1 : illustration of the language development assessment task that focuses on evaluating the progress of language learners over time. the red curve indicates the order of text predicted by the model. tic features. recent advances in plms demonstrate strong capabilities in generating fluent text and providing useful representations of textual patterns. nevertheless, a limited amount of research has concentrated on employing plms for lda tasks. @xcite introduced a novel sequential information attention mechanism to analyze the interaction between essay pairs. extremita @xcite employed two plms trained for undergoing joint fine-tuning using prompting techniques. however, an important issue to be addressed in this regard is how to effectively mine language-independent text features, and extract the differences and common information between the first-written and second-written essays. to address these gaps, we construct the indocl corpus (indonesian corpus of l2 learners), which is a longitudinal corpus of essays written by l2 indonesian students in the second and third year of the undergraduate period. furthermore, we propose a novel model for lda tasks, which effectively extracts the differences and common information between first-written essay and second-written essay through sequential information attention and similarity representation learning respectively, gaining substantial improvement over existing models on our self-constructed corpus. simultaneously, our proposed model exhibits remarkable merit in alleviating the laborious computation and preventing reliance on specific language. experimental results on publicly available corpora reveal that our model achieves promising performance. the main contributions of the paper are: ## related work language acquisition (la) is the field of study concerned with understanding how individuals acquire first or second language competencies. in recent years, with the rapid development of natural language processing (nlp) technologies, the field of la has significantly benefited @xcite @xcite . previous studies on lda tasks can be broadly divided into two distinct groups: one group focuses on constructing a lda model that is based on linguistic features while the other focuses on constructing a lda model that is based on neural networks. to address the difficulty of identifying unique linguistic features, linguisticfeatures-based methods extract a broad range of linguistic-level features for training supervised classification systems on authentic learner data from multiple languages @xcite @xcite @xcite @xcite . recent studies on neural network-based language modeling (e @xcite @xcite have shown that certain neural architectures can understand syntactic details from text without being directly programmed to do so. some studies employ neural-networks-based methods to track and model students' writing skills @xcite @xcite ## indonesian corpus of l2 learners to ensure that our trained model evaluates language proficiency based on the content of the text, without reliance on domain-specific information, we have constructed a general domain dataset for indonesian. the corpus has a diverse range of essay types, including narrative, descriptive, argumentative, and expository essays, which encompass a diverse range of essay topics, including but not limited to food, environment, and lifestyle. these topics are selected to cover a wide range of everyday and relatable topics that are likely to elicit meaningful and context-rich responses from the students. ## model our proposed framework is shown in figure 2 . initially, in the text representation module, we concatenate the first-written essay and second-written essay in a text pair to form a text sequence and encode it employing the plm. subsequently, the sequential information attention mechanism is leveraged to capture the information interaction in the text sequence, so as to obtain the enhanced global representation of the first-written essay and second-written essay respectively. in the similarity representation learning module, the information similarity between the enhanced firstwritten text representation and the second-written text representation is calculated via exploiting a learnable similarity vector. finally, the language development assessment module combines the enhanced global first-written text representation, the enhanced global second-written text representation, the raw text representation, and the vector-based similarity representation to conduct a language development assessment. it is worth mentioning that our framework focuses on capturing both the differences and common information between two essays for language development assessment without the need to extract specific linguistic features, demonstrating extensive applicability. each module is comprehensively presented in detail below. ## conslusion in this paper, we introduce the icl corpus, which, to the best of our knowledge, is the first corpus tailored for indonesian lda tasks. moreover, we present a model capable of automating the extrac-tion of language-independent features for lda tasks, which yields improved performance on both our self-constructed corpus and publicly available corpora. our work could serve as a novel benchmark for indonesian lda tasks. in addition, our research reveals that the performance of llms on lda tasks still has considerable room for improvement. in the future, we will further expand our corpus to encompass additional languages. meanwhile, we will explore innovative ways to enhance the capabilities of llms on lda tasks.