Teacher Perception of Automatically Extracted Grammar Concepts for L 2 Language Learning (2023)

Abstract: One of the challenges in language teaching is how best to organize rules regarding syntax, semantics, or phonology in a meaningful manner. This not only requires content creators to have pedagogical skills, but also have that language’s deep understanding. While comprehensive materials to develop such curricula are available in English and some broadly spoken languages, for many other languages, teachers need to manually create them in response to their students’ needs. This is challenging because i) it requires that such experts be accessible and have the necessary resources, and ii) describing all the intricacies of a language is time-consuming and prone to omission. In this work, we aim to facilitate this process by automatically discovering and visualizing grammar descriptions. We extract descriptions from a natural text corpus that answer questions about morphosyntax (learning of word order, agreement, case marking, or word formation) and semantics (learning of vocabulary). We apply this method for teaching two Indian languages, Kannada and Marathi, which, unlike English, do not have well-developed resources for second language learning. To assess the perceived utility of the extracted material, we enlist the help of language educators from schools in North America to perform a manual evaluation, who find the materials have potential to be used for their lesson preparation and learner evaluation.

## introduction recently, computer-assisted learning systems have gained tremendous popularity, especially during the covid-19 pandemic when in-person instruction was not possible, leading to the need for userfriendly and accessible learning resources @xcite . because these materials are curated by subject experts, this makes curriculum design a challenging process, especially for languages where experts or resources are inaccessible. ## proposed work although language education has been widely studied in literature, there is no one 'right' method of teaching. since our focus is to create pedagogical content to assist teachers in their process, we conduct a pilot study and collaborative design with two kannada teachers who are deeply involved in the curriculum designing. we first identify some "teachable grammar points" which, as defined above, are points that can be taught to a learner and are typically included in a learning curriculum. next, following autolex, we formulate each grammar point into an nlp task and extract human-readable descriptions, as shown below. ## automatic evaluation since our objective is to evaluate whether such automatically derived linguistic insights can be useful for language pedagogy, we first conduct a pilot study to evaluate quality and properties of the extracted materials (section 5). next, we conduct a study with several in-service teachers, both in kannada and marathi, to evaluate relevance, usability, and presentation of the extracted materials (section 6). in addition to human evaluation, we follow @xcite to automatically evaluate the quality of extracted descriptions. this provides a quick sanity check on whether our trained models are able to learn the said linguistic phenomena. word order and agreement @xcite automatically evaluate the learnt model by measuring the accuracy on held-out sentences. for example, for subject-verb model, the gold label is the observed word order which can be determined from the pos and dependency parses (i.e. whether the subject is 'before' or 'after' the verb), which is then compared with the model prediction to compute the accuracy. this model is compared with a baseline that assigns the most frequent observed pattern in the training data as model prediction. ## human quality evaluation we conduct a limited study for a sanity check with two kannada teachers. vocabulary we present both experts with an automatically generated list of 100 english-kannada word pairs, where one english word has multiple translations showing fine-grained semantic subdivisions. both experts found 80% of the word pairs to be valid, according to the criterion that they show different usages. for example, for 'doctor', the model discovered four unique translations, namely 'vaidya, vaidyaro, daktor, vaidyaru' in kannada which the expert found interesting for teaching as they demonstrated fine-grained divergences, both semantically and syntactically. for instance, 'vadiya' is the direct translation of 'doctor', whereas 'daktor' is the english word used as-is, 'vaidyaro' is the plural form and 'vaidayaru' is a formal way of saying a doctor. word order for word order, experts evaluate the rules for subject-verb and object-verb order. for subject-verb, seven grammar rules explaining the different word order patterns were extracted (4 explaining when the subject can occur both before and after the verb, 2 rules informing when subjects occur after, and 1 showing the default order of "before"). of the seven rules, experts found four to be valid patterns. for object-verb word order, of the six rules extracted by the model, experts marked that 2 rules precisely captured the patterns, while one rule was too fine-grained. interestingly, in addition to correctly identifying the dominant order, all the rules which were deemed valid showed non-dominant patterns. the rules marked as invalid were invalid because the syntactic parser that generated the underlying syntactic analyses incorrectly identified the subjects/objects. such errors are expected given that there is not sufficient quantity/quality of expertly annotated kannada syntactic analyses available to train a high-quality parser. however, we would argue that these results are still encouraging because i) despite imperfect syntactic analysis, the proposed method was able to extract several interesting counter-examples to the dominant word order, and ii) further improvements in the underlying parsers for low-resource languages may be expected through active research. ## teacher perception study for kannada, we work with teachers from the kannada academy 7 (ka), which is one of the largest organizations of free kannada teaching schools in the world and recruit 12 volunteer teachers. for marathi, there is no central organization as for kannada, but there are many independent schools in north america. we reached out to marathi vidyalay 8 in new jersey, that teaches learners in the age group of 6-15, and shala in pittsburgh 9 . marathi vidyalay is a small school consisting of seven volunteer teachers, of whom four agreed to participate, while shala has one teacher. all participants are volunteer teachers; teaching is not their primary profession. since we extract learning materials automatically from publicly available corpora which may contain material which is ageinappropriate, we purposefully chose to share these materials with the teachers who we feel are best suited to decide how to use them. perception survey to answer the research question of whether materials are practically usable and, if so, with regard to what aspects, we analyze the kannada and marathi teachers' perception regarding relevance, utility and presentation of the materials. first, a 30-60 minute meeting is conducted for the teachers, in which we introduce the tool, the different grammar points covered in it, and how to navigate the online interface. teachers have one week to explore the materials. finally, all teachers receive a questionnaire ## next steps in this work, we have explored one combination of nlp techniques and corpus linguistics to assist in language education. the perception study shows that teachers do find the selected grammar points relevant and interesting, which highlights the importance of a collaborative design; however, all note that the content is more suitable for advanced learners. although in the current state, the materials cannot be used as-is but the teachers find this overall effort very promising, as this tool can be applied to a corpus of their choice, which is more suited for the learning requirements. among the different features, teachers find the illustrative examples to be most useful, especially for understanding the non-dominant linguistic behaviors or exceptions to general rules. additionally, the tool has the capability to extract numerous example usages which the teachers noted as a big plus, as it can provide a starting point for them to build upon rather than them having to find examples manually. ## limitations currently, a major limitation of the tool, as noted by the teachers, is that the content is not organized by learner age/experience. a next step would be to invite teachers to organize the content by each level, taking the learner incrementally through the complexities of language. for beginner learners, language properties are built through engaging stories with little use of formal grammar terms. therefore, using simpler meta-language to explain the grammar points and including engaging content would be a worthwhile addition. even for the teachers, the materials took some time getting used to, especially the formal linguistic terms, therefore, in addition to simplifying the language, educating the teachers in the tool format will also be necessary for effective learning. we hope that our work drives more such practical research in language education where we consult the real users (teachers in our work) to better understand what they need and work with them in collaboration. ## ethical considerations we acknowledge that there are several ethical considerations to keep in mind while creating content or tools that will be directly used by human learners. since currently we use public corpora to extract the learning materials, they may contain unwanted bias or age-inappropriate language or even culturallyinsensitive materials. that is why we work in close collaboration with the respective educators in this work.