Transformer-based H ebrew NLP models for Short Answer Scoring in Biology (2023)

Abstract: Pre-trained large language models (PLMs) are adaptable to a wide range of downstream tasks by fine-tuning their rich contextual embeddings to the task, often without requiring much task-specific data. In this paper, we explore the use of a recently developed Hebrew PLM aleph-BERT for automated short answer grading of high school biology items. We show that the alephBERT-based system outperforms a strong CNN-based baseline, and that it general-izes unexpectedly well in a zero-shot paradigm to items on an unseen topic that address the same underlying biological concepts, opening up the possibility of automatically assessing new items without item-specific fine-tuning.

## introduction advances in nlp offer transformative technology to support educational practice, including scoring of constructed (free text) responses in both holistic and analytic fashion. in particular, pre-trained large language models (plms) hold great promise for applications that require sophisticated context-rich analysis of student responses. however, progress in plms and their applications in english outstrips that in other languages. recent research in hebrew nlp made available a new hebrew plm -alephbert @xcite ; while it has been shown to be effective for nlp tasks such as pos tagging and ner, its effectiveness for a downstream automated scoring application is an open question. we evaluate alephbert-based classifiers for the task of analytic content-scoring of short answers in biology in a formative high school setting, comparing it to a strong cnn-based baseline. we contribute new knowledge about the effectiveness of bert-based classifiers in languages other than english for a content-scoring task. our two key findings are that the alephbert-based classifiers i) provide a significant improvement over the cnn-based baseline; and ii) generalize surprisingly well to unseen items that deal with the same underlying scientific concepts but in the context of a different topic. we briefly discuss implications of the findings and directions for future work. ## related work an especially promising application area of nlp is automated analysis of responses to open-ended questions, either in the form of a full essay, where the goal is typically a demonstration of proficiency in writing in a particular genre @xcite , or in the form of short responses, where the goal is typically to demonstrate content knowledge. in this paper, we consider the latter application, often termed automated short answer grading (asag). to date, most of the scientific development on asag has been done in english (see @xcite for a survey), including asag using plms @xcite @xcite , although work on plms for asag in other languages does exist, e.g., japanese @xcite , arabic @xcite . recently researchers also used multi-lingual plms for asag @xcite used the labse multilingual transformer model @xcite for scoring very short responses (the bulk of the responses are 5 words or shorter) in a variety of subjects and in 14 languages. unfortunately, the authors did not provide a detailed breakdown of performance by language or by subject area, although they did show that numeric responses tended to be easier to score than textual or mixed ones, across multiple languages. interestingly, while there were relatively few responses in english (1.7k), the system's error on scoring textual responses in english was lower than for ukranian, which had more than two orders of magnitude more responses than english (500k), which could suggest that languages with smaller digital footprints and therefore less data for pre-training the plms would still be at a disadvantage even if there are a lot of responses in those languages for the specific task. the asag task for hebrew was addressed by @xcite . the authors built cnn-based classifiers that used word2vec embeddings; these models will serve as baselines for the current work. hebrew, like arabic, is a semitic language where vowels are generally omitted in writing, resulting in substantial ambiguity where the same sequence of written letters can have many meanings depending on context. therefore, a plm that implements the latest contextualization advancements holds great promise for asag in hebrew. alephbert, the recently introduced hebrew plm @xcite , shows sota performance on multiple tasks, including morphological and pos tagging and ner. our goal is to evaluate alephbert for the asag task in hebrew. ## results ## conclusions pre-trained large language models can be adapted to downstream tasks by fine-tuning their rich contextual embeddings to the task. we explored the recent hebrew plm -alephbert -for short answer grading in high school biology. we found that the alephbert-based system outperformed a strong baseline and that it generalized unexpectedly well to items on an unseen topic addressing the same biology concepts. the second finding provides evidence in support of the viability of the modular design of the rubric -not only is it the case that human raters were able to reliably assess different items with subsets of the same analytic categories, but an automated model was likewise able to zero in on the commonalities in the way categories are manifested in student responses across multiple topics. the cross-item generalization has exciting implications for educational practice, as this may allow teachers to create and automatically score new items based on a similar rubric on the fly. a study of this possibility with teachers and an improvement of our understanding of the conditions neces-