Designing the LECOR Learner Corpus for R omanian (2023)

Abstract: This article presents a work-in-progress project, which aims to build and utilize a corpus of Romanian texts written or spoken by non-native students of different nationalities, who learn Romanian as a foreign language in the one-year, intensive academic program organized by the University of Bucharest. This corpus, called LECOR – Learner Corpus for Romanian – is made up of pairs of texts: a version of the student and a corrected one of the teacher. Each version is automatically annotated with lemma and POS-tag, and the two versions are then compared, and the differences are marked as errors at this stage. The corpus also contains metadata file sets about students and their samples. In this article, the conceptual framework for building and utilization of the corpus is presented, including the acquisition and organization phases of the primary material, the annotation process, and the first attempts to adapt the NoSketch Engine query interface to the project’s objectives. The article concludes by outlining the next steps in the development of the corpus aimed at quantitative accumulation and the development of the error correction process and the complex error annotation.

## introduction the lecor corpus is developed through the project "learner corpus of romanian (lecor). collection, annotation and applications", funded by the romanian government as part of the subprogram dedicated to research projects to stimulate independent young teams (te). the host institution of the project is the university of bucharest. in this section we present the core features of the lecor corpus and the different uses for which it has been designed. the main goal of the project is to build and make accessible the first romanian electronic learner corpus and, at the same time, to professionalize human resources in this field of research. it will be a corpus in free access, open for searching through the provided interface. lecor will be downloadable only by request and only for research purposes, which will be in accordance with the informed consent signed by learners. lecor is a monolingual general learner corpus, collected over four academic @xcite . the target language is (general, standard) romanian, learned in a native speaking environment and taught by native teachers. the learners are university students, aged generally between 18 and 25. their native languages are various (more than 20 mother tongues): arabic, chinese, korean, albanian, greek, armenian, turkmen, turkish, persian, slavic languages (bulgarian, serbian, ukrainian, belarusian, russian, etc.). among the internationally spoken languages, arabic and mandarin chinese are well represented, in contrast to english, spanish, and hindi, which are scarcely or not at all found, just like all western and northern european languages. the romance languages are also weakly represented, especially through their nonstandard varieties: spanish and portuguese from latin america, and french from africa. as a general characteristic, the learners' native languages exhibit typological distance both from romanian and among themselves. as the corpus processing is not yet complete, an exact proportion of native languages in lecor cannot be provided at this time. in its first version, lecor will contain written (80%) and oral (20%) learners' samples. the corpus is planned to be of large size, including 4,000 samples (of which, 800 audio transcriptions). the 4,000 samples in the corpus have very different sizes (ranging from samples provided by a1level students, consisting of at least 40-50 words, to samples provided by b2+ level students, containing more than 350 words). taking an average of 150-170 words per sample, we can approximate the size of lecor at the end of the project to be over 600,000 words. at the moment, about 4,500 written productions are collected (from which a selection will be made, prioritizing exam papers and texts produced in class) and about 500 files with audio homework and exam recordings; users of the corpus will have access to original samples (handwritten texts and audio files). all the texts will be automatically annotated with lemma and pos-tag, while a small part will be annotated for errors. the lecor corpus was designed to be scalable, so the aim is to increase its size and continue the annotation. in each of the four academic years on the duration of the project, samples are collected, mostly in controlled contexts, from approximately 50-70 students per year, documenting their progress from a1 to b1 or b2 proficiency level. therefore, the corpus can be used for both synchronic/cross-sectional and diachronic/ longitudinal research. lecor is all the more valuable as it encompasses, besides a2, b1 and b2 samples, at least one quarter of a1 samples and as it thoroughly documents the interlanguage development of several dozens of learners, who have produced approximately 60 samples throughout an entire academic year foot_0 . regarding the representativeness of proficiency levels, lecor is a relatively balanced corpus: the number of samples from a2 and b1 learners is comparable, whereas a1 learners contribute slightly fewer samples, and b2 learners produce the least number of samples. the text types are varied and comply with the minimum proficiency level requirements (e.g. argumentative essays are not required at beginner level). in lecor there are descriptions (of a city, of a (class)room, a house, a person, etc.), especially at a1 and a2 levels, narratives (what i did today, what i used to do on holiday as a child, a nightmare trip, etc.), argumentative essays (why it's good to learn languages, online shopping -pros and cons, protecting the environment, etc.). several description and story-telling tasks are based on pictures. the text genre are also diverse: e-mail (letter), long whatsapp message, review, essay, description, procedure (recipe, health instructions), etc. lecor is a very well documented resource, learner variables/metadata, as well as text and task variables/metadata being carefully and thoroughly recorded, following the core metadata scheme for learner corpora (see @xcite . because it is a large annotated corpus with rich metadata and a high degree of representativeness, lecor will have many possible end-uses. at first, it will be used in studies about nonnative romanian acquisition and, in general, in second language acquisition research (for testing particular sla theories, to set up the interlanguage profile at certain stages of sl / fl development, to track individual differences, etc., see also @xcite ). then, the corpus can be used in language teaching and in natural language processing. traditionally, learner corpora are used for didactic purposes (for an overview of applications, see at least @xcite ; @xcite ; @xcite ; @xcite ). on the one hand, it can be used to inform instructional materials design, such as course books (e.g. learning from common mistakes, brook-hart (2009)), learner dictionaries (see, for example, macmillan english dictionary advanced learner, @xcite ), wordlists (e.g. focus on vocabulary 2: mastering the academic word list, schmitt and schmitt (2011)), etc.; moreover, the metadata will allow for creating a 'difficulties profile' for learners with a specific mother tongue and thus will enable teachers to design more specific materials for their target groups of learners. such materials do not exist at all for romanian and are obviously long overdue by both learners and instructors. more precisely, based on the learner corpus (and, in many cases, a contrasting, languagetarget corpus), numerous research questions can be addressed: how does second language evolve across different levels of proficiency? which errors are developmental (specific to all learners) and which are likely to be caused by transfer from the native language? what are the specific features of interlanguage at a given proficiency level for a given population of learners? for romanian as a target language, vasiu (2020) tries to identify, based on an own corpus what is the specificity of interlanguage at a1 level with respect to the learners' native language. using quantitative analysis, the author reaches conclusions such as: at a1 level, for all native language groups, preposition acquisition is the most difficult; all a1 learners tend to omit adverbs; there are agreement errors between nouns and adjectives, except for possessive adjectives (for 1 st and 2 nd person, singular), which are memorized as formulas (mama mea 'my mother', profesorul meu 'my father'); arabic learners tend to omit the copulative verb most frequently; romance speakers superfluously use the preposition la 'at', etc. lecor can also be used for language teachers training and for language testing @xcite . on the other hand, the corpus will have an immediate pedagogical use; it will be available for use in classrooms or by learners themselves, since this kind of data is relevant for the (error) producers. lecor can be used also for native language automatic identification 2 , in forensic linguistics. non-native speakers of romanian make errors characteristic of learners with a specific mother tongue. thus, the native language of a malicious individual can be discovered by mapping the type of errors made in his/her use of romanian. this is an important means of identifying such individuals and it is very useful in the context of increasing social media threats. the learners' errors identified in lecor can be used also to improve the technology for automatic translation @xcite . lecor can also be used for automatic grammarand spell-checking and automated scoring of l2 written and oral performance (also @xcite ). ## related work in the last three decades, the construction of learning corpora has experienced a remarkable development, as evidence of their increasing importance, as can be seen in the list of about 200 corpora provided on the website of the catholic university of 2 for the identification of the native language (nli), a largesized corpus and a high-quality dataset (comparable to the international corpus of learner english (icle), used for nli research, which comprises 6,085 essays written by speakers of 16 different l1s, see @xcite ) are necessary, with a large number of native languages to enable comparison; uniform topics; comparable text sizes; thoroughly evaluated proficiency levels @xcite . lecor will be a medium-sized learner corpus, containing 4,000 samples, scalable (with the possibility to increase over time), covering l1 languages at least as diverse as those ones in icle. louvain foot_2 or in the clarin infrastructure 4 . the series of these corpora was opened at the time of the publication of the international corpus of learner english @xcite and is by far dominated by the broad interest in learning english, but there are also corpora with written, audio or multimodal content for learning many other languages from different language families, such as arabic, czech, finnish, french, german, greek, mandarin, japanese, etc. the romance languages, of which the romanian language is a part, are also well represented in this field, with written or spoken corpora, of which we mention the general ones, with native students of different languages and a unique target language: cople2 @xcite for portuguese, the spanish learner corpora (slc) (alonso-ramos, 2016), celi @xcite , lips @xcite or valico @xcite for italian, or the flloc platform @xcite or parole @xcite for french. for the romanian language, apart from small in-house bespoke corpora, there are only two printed corpora @xcite , which gather romanian raw texts produced by foreign students. the corpus compiled by constantinescu and stoica (2020) comprises more than 450 samples (380 written samples / 65,000 words, and 79 oral transcriptions / 60,000 words); it was produced by 61 a1-b2 learners in the period 2004-2016 in various instructional contexts. vasiu (2020) corpus contains transcriptions of oral samples produced by 172 a1 students at proficiency tests in 2014-2017; its size (70,000 words) is comparable to the oral part of the previous corpus. the digitalization and integration in our project of the two printed corpora would be a difficult endeavour, in terms of copy right issues and collaboration between independent working teams. moreover, we had a different design in mind: large-scale corpus, richer metadata, longitudinal scope, internationally used annotation schemes, etc. in this perspective, our project comes to cover an important gap in this field. for our project, we also benefited from the experience of building other corpora, such as the corpus for czech, czesl @xcite , for latvian, lava (dar ' gis et al., 2022) or for croatian, croltec @xcite and many others. from these we took as a model the set of metadata, the text annotation with pos and error codes, the association of student texts with variants corrected by teachers or multi-level error annotation. regarding the corpus query interface, although many of the existing corpora use the teitok interface (cople2, czesl, croltec, etc.), we followed lava's example by adopting the nosketch engine interface, due to operating system version and configuration incompatibilities between our server and teitok. ## raw-material collection and organization the corpus is collected from foreign students coming for studies in romania from eastern countries (far east, near east, middle east), south-eastern europe, latin america and, less often, western and central europe; their native languages (arabic, chinese, bulgarian, albanian, serbian, turkish, greek etc.) are therefore both typologically distant from romanian and from each other. they are learning in mixed groups. the learners are generally high school graduates, but there are also masters and phd students (aged between 18 and 25 years). the one-year program they are enrolled in is intensive, totalling 800 hours of classroom instruction. in the first part of the academic year, they have in their curriculum 28 hours per week of general course of romanian, while in the second part, 30 hours per week (with the addition of languages for specific purposes). in general, students' interlanguage progress is documented from absolute beginner (a1) to intermediate level (b1 or b2). the raw material of the corpus (scans of the hand-written work samples, digital textwork samples, audio and video recordings) comes from different sources (it was collected by several teachers from the foreign students enrolled in the one-year intensive program "preparatory year" at faculty of letters, university of bucharest), in different folder and archiving structures (organized by student, by work sample or by teacher, archived or not archived) and in different file formats (.mp3/.mov/.mp4 for audio/video, .jpg/.png/.heic/.pdf for scans, word/pdf for digital texts). moreover, some scans cover more than one work sample and have to be split through different methods, according to their format: e.g., .pdf files are automatically page-split, .png/.jpg files are manually cropped in paint. then .heic/.png/.jpg files are converted to .pdf, for harmonization and because pdf format offers small size and quick loading with unsignificant loss in quality together with the possibility to concatenate photos depicting different pages of the same work sample. video files are converted to audio files and the common chosen format was .mp3. all scanned and audio/video files are manually transcribed following common orthographic transcription guidelines; some of the transcription principles are: (1) do not take into account strikethrough words in the scanned files or hesitations/repetitions in the audio/video files, but transcribe only the final version of a word provided by the student; (2) ignore syllabification of words at the end of the line, including the erroneous ones; (3) transcribe bracketed words; (4) keep the paragraph structure of the original but do not mark the original indenting; (5) for errors in lower/upper case spelling, only the ones concerning proper names or beginning of sentence are transcribed; other inappropriate uses of case spelling are ignored. in this manner, a transcribed .txt version of each work sample (further referred to as transcribed student form) was created, which is the basis for all preprocessing, correction, annotation and indexing steps in the corpus generation flow. in order to protect the integrity of learners, we followed guidelines for research data management from university of bucharest foot_4 and catholic university of leuven foot_5 , which address various issues about anonymization, pseudo-anonymization, and encryption of sensitive data. all files are completely manually anonymized to protect student's right to privacy (see an example in figure 1 ). this procedure is done after checking the transcripts, generally at the same time of samples correction. despite the task not being performed automatically, due to the fact that not all samples contain personal data (e.g. only about 15% of audio files require anonymization), the total time required for the task is reasonable, e.g. 1-2 minutes to clear personal buna ziua romȃnia este foarte fromoasa s , i mare, in vacant , a am sa merg la fara bucures , ti, cred sa merg la bras , ov casa este foarte frumoasȃ, si dupa asta ajung, voi merge la munte pentru chiez s , i osa merg la cluj trei zile pentru intalnesc prietenii mei dupa asta vreau sa intorc la maroc pentru intalnesc familia mea, mi-e foarte dor la familia mea si luam permisul de cunduci si osa intorc la romania s , i osa cumper masina (bmw). osa cumper apartament im bucures , ti cu prietenmeu omar si vom merge la constant , a pentru întonam in mare si dupa asta întorceam la facultatea de litere pentru studiez information from an audio sample. "coding" sensitive data is relatively easy. the sensitive data in scanned images is covered in paint (for pdf sources they are converted to paint, anonymized and converted back to pdf), audio sensitive data are replaced with beep sounds in audacity, while the personal information in text documents is pseudonymized, i.e. replaced with similar plausible data (e.g. "mohammed" is replaced with "ahmed", "35 years old" is replaced with "29 years old", etc.) to maintain the morpho-syntactic coherence of the sentence (full anonymization, e.g replacing "35 years old" with "xxx" will impact negatively on the pos-tagging performance in future steps). the sensitive data we are targeting in the anonymization/pseudonymization process concerned student's name, age, birth date and birth place, previous school/university/-work place, etc. (our internal list is broadly similar to that in @xcite ). in case the pseudonymized word have morphological features like gender/number/case, they have to be replaced with similar values: e.g., feminine, genitive etc. for more challenging cases, where covering up sensitive data would affect the overall message of the text (e.g., replacing the name of a town with another would make its description inappropriate), we decided to keep the original place name. next step is producing the corrected form for each sample, further called teacher form. the correction is made by linguists/romanian language teachers on the transcribed student sample by using commonly agreed general principles: e.g., 1. minimal corrections (if possible, we do not change the part of speech and the words order or number), 2. we do not provide more correction alternatives, but only one correct option; 3. semantic accuracy is preserved: e.g. if the work sample is made based on an image and the image depicts a "red skirt", the "yellow skirt" syntagm provided by the student is considered an error and corrected. a distinction is made between actual errors affecting form, grammar, vocabulary, punctuation, etc., and infelicitous constructions, register and stylistic inaccuracies and other awkward language (for a similar approach, see @xcite ). the category of infelicities includes, for example: informal language, such as the short demonstrative forms asta, ȃsta ('this (one)') instead of the standard (long) ones aceastȃ/aceasta, acest/acesta ('this (one)'), the shortened forms of numerals (treis , pe, instead of treisprezece 'thirteen'), address formulae with an inappropriate degree of politeness, text sequences with unclear meaning, etc. at the moment, it has not been decided how exactly infelicities will be annotated, but the annotation will definitely be done manually. this preprocessed material is than organized in two steps: ## annotation procedure once the source files are organized in the manner presented above, the annotated corpus is created based on a procedure that includes the following two stages: this procedure is semi-automated, requiring the corpus files to be passed through the external pos annotation platform (located on a server other than the corpus server), then the annotated files are uploaded to the lecor server for automatic error annotation. to make working with a large volume of data more efficient, scripts were created to detect and process only files added to the lecor corpus or modified after the last annotation. ## conclusions and further work in this study we presented the design stage of the lecor corpus emphasizing the conceptual framework for building and utilization of the corpus. the next phase involves the quantitative accumulation of primary material. regarding the text correction, we have already established the general criteria and developed a proofreading manual, and further we will validate these criteria by correcting a representative volume of texts in parallel and establishing the inter-annotator agreement. error annotation in this phase is done at a basic level and is strongly correlated with automatically detected differences between the student/teacher versions. this phase is very useful for what we intend to do, which is a manual error annotation, on multiple levels, as is already practiced in the field. the inventory of errors is already established, it remains to build the technical annotation method, especially for errors whose correction involves changes in the word order. the nosketch engine query interface will be explored further to see how well it can be adapted to the project's goals. for example, a distinction must be made between the student version and the teacher version, possibly with separate searches on each version. we will investigate whether the option of querying parallel corpora provided by nosketch engine solves this desideratum. another issue concerns the use of metadata about students and work samples. we consider using the platform's option to create subcorpora, which can potentially be selected by certain metadata values.