TLT -school: a Corpus of Non Native Children Speech (2020)

Abstract: This paper describes “TLT-school” a corpus of speech utterances collected in schools of northern Italy for assessing the performance of students learning both English and German. The corpus was recorded in the years 2017 and 2018 from students aged between nine and sixteen years, attending primary, middle and high school. All utterances have been scored, in terms of some predefined proficiency indicators, by human experts. In addition, most of utterances recorded in 2017 have been manually transcribed carefully. Guidelines and procedures used for manual transcriptions of utterances will be described in detail, as well as results achieved by means of an automatic speech recognition system developed by us. Part of the corpus is going to be freely distributed to scientific community particularly interested both in non-native speech recognition and automatic assessment of second language proficiency.

## introduction a large set of both written and spoken data has been collected during the implementation of campaigns aimed at assessing the proficiency of italian students learning both german and english. part of these data has been included in a corpus, named "trentino language testing" in schools (tlt-school), that will be described in the following. design of the campaigns, involvement of schools, data collection and annotation by human experts were carried out and coordinated by iprase, a research institute dealing with education which is located in trentino, north italy. all the collected sentences have been annotated by human experts in terms of some predefined "indicators" which, in turn, were used to assign the proficiency level to each student undertaking the assigned test. this level is expressed according to the well-known common european framework of reference for languages @xcite scale. the cefr defines 6 levels of proficiency: a1 (beginner), a2, b1, b2, c1 and c2. the levels considered in the evaluation campaigns where the data have been collected are: a1 (primary school), a2 (secondary school) and b1 (high school). the indicators measure the linguistic competence of test takers both in relation to the content (e.g. grammatical correctness, lexical richness, semantic coherence, etc.) and to the speaking capabilities (e.g. pronunciation, fluency, etc.). refer to section 2 for a description of the adopted indicators. the learners are italian students, between 9 and 16 years old. they took proficiency tests by answering question prompts provided in written form. the "tlt-school" corpus, that we are going to make publicly available, contains part of the spoken answers (together with the respective manual transcriptions) recorded during some of the above mentioned evaluation campaigns. we will release the written answers in future. details and critical issues found during the acquisition of the answers of the test takers will be discussed in section 2. the tasks that can be addressed by using the corpus are very challenging and pose many problems, which have only par-tially been solved by the interested scientific community. from the asr perspective, major difficulties observed in this corpus are represented by: a) recognition of both child and non-native speech, i.e. italian pupils speaking both english and german, b) presence of a large number of spontaneous speech phenomena (hesitations, false starts, fragments of words, etc.), c) presence of multiple languages (english, italian and german words are frequently uttered in response to a single question), d) presence of a significant level of background noise due to the fact that the microphone remains open for a fixed time interval (e.g. 20 seconds -depending on the question), and e) presence of noncollaborative speakers (students often joke, laugh, speak softly, etc.). refer to section 2.3 for a detailed description of the collected spoken data set. furthermore, since the sets of data from which "tltschool" was derived were primarily acquired for measuring proficiency of second language (l2) learners, it is quite obvious to exploit the corpus for automatic speech rating. to this purpose, one can try to develop automatic approaches to reliably estimate the above-mentioned indicators used by the human experts who scored the answers of the pupils (such an approach is described in @xcite ). however, it has to be noted that scientific literature proposes to use several features and indicators for automatic speech scoring which are partly different from those adopted in "tlt-school" corpus (see below for a brief review of the literature). hence, we believe that adding new annotations to the corpus, related to particular aspects of language proficiency, can stimulate research and experimentation in this area. finally, it is worth mentioning that also written responses of "tlt-school" corpus are characterised by a high level of noise due to: spelling errors, insertion of word fragments, presence of words belonging to multiple languages, presence of off-topic answers (e.g. containing jokes, comments not related to the questions, etc.). this set of text data will allow scientists to investigate both language and behaviour of pupils learning second languages at school. written data are described in detail in section 2.2 relation to prior work. scientific literature is rich in approaches for automated assessment of spoken language proficiency. performance is directly dependent on asr accuracy which, in turn, depends on the type of input, read or spontaneous, and on the speakers' age, adults or children; see @xcite for an overview of spoken language technology for education. a recent publication reporting an overview of state-of-the-art automated speech scoring technology as it is currently used at educational testing service (ets) can be found in @xcite . in order to address automatic assessment of complex spoken tasks requiring more general communication capabilities from l2 learners, the speaking part of the arizona english language learner assessment (azella), a largescale test @xcite developed by pearson, has been used for some researches @xcite . the work described in @xcite reports results achieved on 1, 500 spoken tests, each double graded by human professionals, from a variety of tasks. a public set of spoken data has been recently distributed in a spoken call (computer assisted language learning) shared task foot_0 where swiss students learning english had to answer to both written and spoken prompts. the goal of this challenge is to label students' spoken responses as "accept" or "reject". refer to @xcite for details of the challenge and of the associated data sets. many non-native speech corpora (mostly in english as target language) have been collected over the years. a list, though not recent, as well as a brief description of most of them can be found in @xcite . the same paper also gives information on how the data sets are distributed and can be accessed (many of them are available through both ldc foot_1 and elda foot_2 agencies). some of the corpora also provide proficiency ratings to be used in call applications. among them, we mention the isle corpus @xcite , which also contains transcriptions at the phonetic level and was used in the experiments reported in @xcite . note that all corpora mentioned in @xcite come from adult speech while, to our knowledge, the access to publicly available non-native children's speech corpora, as well as of children's speech corpora in general, is still scarce. specifically concerning non-native children's speech, we believe worth mentioning the following corpora. the pf-star corpus @xcite contains english utterances read by both italian and german children, between 6 and 13 years old. the same corpus also contains utterances read by english children. the childit corpus @xcite contains english utterances (both read and imitated) by italian children. by distributing "tlt-school" corpus, we hope to help researchers to investigate novel approaches and models in the areas of both non-native and children's speech and to build related benchmarks. 2 data acquisition in trentino, an autonomous region in northern italy, there is a series of evaluation campaigns underway for testing l2 linguistic competence of italian students taking proficiency tests in both english and german. a set of three evaluation campaigns is underway, two having been completed in 2016 and 2018, and a final one scheduled in 2020. note that the "tlt-school" corpus refers to only the 2018 campaign, that was split into two parts: 2017 try-out data set (involving about 500 pupils) and the actual 2018 data (about 2500 pupils). each of the three campaigns (i.e. 2016, 2018 and 2020) involves about 3000 students ranging from 9 to 16 years old, belonging to four different school grade levels (5 th , 8 th , 10 th , 11 th ) and three proficiency levels (a1, a2, b1). the schools involved in the evaluations are located in most part of the trentino region, not only in its main towns; tables 2 and 3 report some statistics extracted from both the written and spoken data collected so far in all the campaigns. each written sentence or spoken utterance received a total score by human experts, computed by summing up the scores related to 6 indicators in 2017/2018 (in the 2016 campaign, the number of indicators ranged from 3 to 6, according to the proficiency levels and the type of test). each indicator can assume a value 0, 1, 2, corresponding to bad, medium, good, respectively. the list of the indicators used by the experts to score written sentences and spoken utterances in the evaluations, grouped by similarity, is reported in ## manual transcriptions in order to create both an adaptation and an evaluation set for asr, we manually transcribed part of the 2017 data sets. an initial set of guidelines for the annotation was defined and adopted by 5 researchers to manually transcribe about 20 minutes of these audio data. this experience led to a discussion, from which a second set of guidelines originated, aiming at reaching a reasonable trade-off between transcription accuracy and speed. briefly, the most important guidelines are: next, we concatenated utterances to be transcribed into blocks of about 5 minutes each. we noticed that knowing the question and hearing several answers could be of great help for transcribing some poorly pronounced words or phrases. therefore, each block contains only answers to the same question, explicitly reported at the beginning of the block. at this point, we engaged about 30 students from two italian linguistic high schools (namely "c" and "s") to perform manual transcriptions of most of the 2017 audio data. after a joint training session, where the guidelines were explained and motivated, we paired students together. each pair first transcribed, individually, the same block of 5 minutes. then, they went through a comparison phase, where each pair of students discussed their choices and agreed on a single transcription for the assigned data. transcriptions made before the comparison phase were retained to evaluate inter-annotator agreement. apart from this first 5 minute block, each utterance was transcribed by only one transcriber. inter-annotator agreement for the 5-minute blocks is shown in in order to assure quality of the manual transcriptions, every sentence transcribed by the high school students was automatically processed to find out possible formal errors, and manually validated by researchers in our lab. speakers were assigned either to training or evaluation sets, with proportions of 2 3 and 1 3 , respectively; then training and evaluation lists of utterances were built, accordingly. ## usage of the data from the above description it appears that the corpus can be effectively used in many research directions. ## distribution of the corpus the corpus to be released is still under preparation, given the huge amount of spoken and written data; in particular, some checks are in progress in order to: in particular, the proposal for an international challenge focused on non-native children speech recognition is being 4 https://www.ets.org/s/toefl/pdf/toefl speaking rubrics.pdf submitted where an english subset will be released and the perspective participants are invited to propose and evaluate state-of-art techniques for dealing with the multiple issues related to this challenging asr scenario (acoustic and language models, non-native lexicon, noisy recordings, etc.). ## conclusions and future works we have described "tlt-school", a corpus of both spoken and written answers collected during language evaluation campaigns carried out in schools of northern italy. the procedure used for data acquisition and for their annotation in terms of proficiency indicators has been also reported. part of the data has been manually transcribed according to some guidelines: this set of data is going to be made publicly available. with regard to data acquisition, some limitations of the corpus have been observed that might be easily overcome during next campaigns. special attention should be paid to enhancing the elicitation techniques, starting from adjusting the questions presented to test-takers. some of the question prompts show some lacks that can be filled in without major difficulty: on the one hand, in the spoken part, questions do not require test-takers to shift tense and some are too suggestive and close-ended; on the other hand, in the written part, some question prompts are presented both in source and target language, thus causing or encouraging code-mixing and negative transfer phenomena. the elicitation techniques in a broader sense will be object of revision (see @xcite and specifically on children speech @xcite ) in order to maximise the quality of the corpus. as for proficiency indicators, one first step that could be taken in order to increase accuracy in the evaluation phase both for human and automatic scoring would be to divide the second indicator (pronunciation and fluency) into two different indicators, since fluent students might not necessarily have good pronunciation skills and vice versa, drawing for example on the ielts foot_3 speaking band descriptors. also, next campaigns might consider an additional indicator specifically addressed to score prosody (in particular intonation and rhythm), especially for a2 and b1 level test-takers. considering the scope of the evaluation campaign, it is important to be aware of the limitations of the associated data sets: proficiency levels limited to a1, b1 and b2 (cefr); custom indicators conceived for expert evaluation (not particularly suitable for automated evaluation); limited amount of responses per speaker. nevertheless, as already discussed, the fact that the tlt campaign was carried out in 2016 and 2018 in the whole trentino region makes the corpus a valuable linguistic resource for a number of studies associated to second language acquisition and evaluation. in particular, besides the already introduced proposal for an asr challenge in 2020, other initiatives for the international community can be envisaged: a study of a fully-automated evaluation procedure without the need of experts' supervision; the investigation of end-to-end classifiers that directly use the spoken response as input and produce proficiency scores according to suitable rubrics.