Automatic Student Answer Assessment using LSA (2023)

Abstract: Implementing technology in a modern-day classroom is an ongoing challenge. In this paper, we created a system for an automatic assessment of student answers using Latent Semantic Analysis (LSA) – a method with an underlying assumption that words with similar meanings will appear in the same contexts. The system will be used within digital lexical flash-cards for L2 vocabulary acquisition in a CLIL classroom. Results presented in this paper indicate that while LSA does well in creating semantic spaces for longer texts, it somewhat struggles with detecting topics in short texts. After obtaining LSA semantic spaces, answer accuracy was assessed by calculating the cosine similarity between a student’s answer and the golden standard. The answers were classified by accuracy using KNN, for both binary and multinomial classification. The results of KNN classification are as follows: precision P = 0.73, recall R = 1.00, F1 = 0.85 for binary classification, and P = 0.50, R = 0.47, F1 = 0.46 score for the multinomial classifier. The results are to be taken with a grain of salt, due to a small test and training dataset.

## introduction employing technology to improve language learning outcomes is a problem scientists have wrestled with since the 1960s. in this paper, we present a beta version of a model for an automatic assessment of student answers using latent semantic analysis (lsa) implemented in a use-case scenario, i.e. for assessing vocabulary knowledge of students and associates at the faculty of mining and geology, university of belgrade. in further development, the aim is for the model will be implemented within digital lexical flashcards for learning vocabulary in english as a second language (esl) classes. previous research @xcite @xcite shows that many cognitive abilities in humans, including vocabulary acquisition, are wellrepresented by lsa. furthermore, assessments provided by lsa largely correlated with those done by evaluators @xcite @xcite @xcite . flashcards have proven to be a good tool for l2 vocabulary acquisition, combining interval @xcite and conscious learning @xcite -two approaches that enhance learning outcomes, especially at the lower levels of language knowledge @xcite . in this phase of work, we will tackle several methodological problems, such as using lsa on short text, and finding means to contribute to the digitalisation of l2 classroom at the faculty of mining and geology, university of belgrade. the research aims to examine the current general and geological vocabulary knowledge of the faculty's students and associates and to improve teaching methods at the faculty by utilising natural language processing (nlp). also, we examine lsa's application in the geological domain, and on shorter text, i.e. definitions. conforming to the aforementioned aims, our hypotheses are: (1) the creation of the system will help digitalise learning materials; (2) lsa will be successful in assessing student answers. the paper is organised as follows: in section 2 we will go through previos research of vocabulary acqusition and lsa implementation in educaiton technologies, proceeding to data and model description in section 3. after that, we will analyse the results in section 4, starting from testng lsa model validity (section 4.1) and going thorugh topic distribution (section 4.2), and finishing with answer assessment (section 4.3) and classification (section 4.4). finally, we will present concluding remarks in 5 and end with the limitations of our approach. ## related works in exploring l2 acquisition, vocabulary acquisition is widely researched. it is considered that vocabulary learning has the best outcomes when combining spaced (or distributed/interval) learning with explicit learning. spaced learning is learning in many small sessions increasing the breaks between each session @xcite , while explicit learning assumes that the student is aware of the learning process @xcite @xcite . as flashcards provide simultaneous explicit and interval learning, together with learning word form, meaning and use in context @xcite , they make a great learning tool. several researches display that flashcards significantly enhance l2 vocabulary acquisition outcomes, especially at the lower levels of language knowledge @xcite @xcite @xcite . given that students who are non-native english speakers can enter university with different levels of language knowledge, using flashcards as a teaching tool can help students reach the necessary level of english to follow classes and learning materials. our case is no different. one of the main problems in esl classes at the faculty of mining and geology, university of belgrade emphasised in @xcite is a low level of language knowledge at the beginning of studies. @xcite also points out that students have in finding a suitable learning method, and lack of translation of geological terminology to serbian, which makes translational tasks even more difficult. our model will be monolingual, so we will not address the last-mentioned issue. currently, the faculty uses a variety of language tools, a thesaurus of geological terminology in serbian and english, comprised of roughly 2800 words @xcite , and a digital mining terminology platform rudonto. 1 additionally, a system of flashcards rgf flashcards was developed, using anki and integrated into the faculty's moodle platform. 2 the presented system of flashcards will be tailored to the learning materials and adapted to the clil methodology used in the faculty's english 1 rudonto thesaurus, accessed 20 may 2023 2 moodle, accessed 20 may 2023 1-4 subjects. clil integrates learning content from a certain domain with language learning @xcite @xcite , whereby c1 entry language knowledge is expected. thus, flashcards could facilitate learning for students with lower levels of english and make following of the learning materials and classes easier. in this stage of development of the flashcards system, we aimed to create a model for an automatic assessment of the semantic similarity of student answers and the golden standard. for that purpose, we exploited lsa -a theory and method for extraction and representation of word meaning in context, whereby statistical calculations are applied to a large text corpus @xcite . thus far, research has shown that lsa can broadly represent human cognitive abilities, such as vocabulary acquisition, word categorisation, discourse comprehension, and essay assessment @xcite . lsa has hitherto been used for answer assessment, providing feedback, answering student questions, as well as assessing student essay accuracy and coherency, in several smart games. in the essay assessment task, it displayed a high degree of correlation with evaluator assessments @xcite @xcite @xcite @xcite . in the light of previously said, we believe that the method is suitable for our task as well. the idea of context-based representation of word meaning is by no means a new one in linguistic theory. @xcite fist posed that elements of a language appear relative to one another. later an automatic clustering-based algorithm for word sense disambiguation was presented by @xcite , where a cluster consists of contextually similar occurrences of a word. distributional semantics also transcended to computational linguistics, and model such as latent semantic analysis (lsa) and latent dirichlet allocation (lda) were developed @xcite . more recently, with the development of neural networks, models such as word2vec have became increasingly popular in representing word meaning @xcite . however, in a case study presented in @xcite , lsa outperformed skip-gram model when the size of the corpora was reduced from medium to small. ## conclusion in this paper, we discussed the application of latent semantic analysis for the assessment of short answers. in accordance with the set pedagogical goals of this paper, we extrapolated that the utilisation of flashcards for l2 vocabulary acquisition gives favourable results, particularly at the lower levels of language knowledge. as students of the faculty of mining and geology come from different educational backgrounds and usually enter their studies with a low level of english, we strongly believe that using a system of flashcards that accompany the subject textbook would greatly help students to make progress faster and get to a level of vocabulary knowledge suitable for following clil lectures. reflecting on the methodological aims of the paper, we determined that developing this model helped us recognise the advantages and disadvantages of our approach. one of the greatest ad-vantages of the model is good topic modelling of longer texts and vocabulary and answers pertaining to geology. we deem that the biggest downside is its inability to detect topics of very short answers. to overcome model downsides, the first step in further research would be to expand the answer data set. our second goal is to add a system for spelling and grammar assessment. in order to improve the results obtained using lsa, we could try and lower the number of dimensions. additionally, creating separate semantic spaces for words that are not geological notions, i.e. general vocabulary, might be a good idea. when comparing answers and unit texts, we believe that we would get more meaningful results if we extract just a fragment of the text where a certain geological notion is explained or a word belonging to the general vocabulary used. lastly, instead of computing the similarity of all answers, we would proceed to calculate the similarity of answers to the same question. the presented model development laid a foundation for the development of a system for automatic answer assessment in digital flashcards. comparing the goals and aims of clil methodology and the outcomes of using flashcards in teaching, we concluded that this technology would greatly complement the textbook in preparation. our claim is supported by the faculty's students' positive attitude towards using digital flashcards in an l2 classroom expressed in previous research. ultimately, we intend to accomplish the project's main goal -the development of a digital flashcard system that will be implemented in the classroom.