Knowing Right from Wrong: Should We Use More Complex Models for Automatic Short-Answer Scoring in B ahasa I ndonesia? (2020)

Abstract: We compare three solutions to UKARA 1.0 challenge on automated short-answer scoring: single classical, ensemble classical, and deep learning. The task is to classify given answers to two questions, whether they are right or wrong. While recent development shows increasing model complexity to push the benchmark performances, they tend to be resource-demanding with mundane improvement. For the UKARA task, we found that bag-of-words and classical machine learning approaches can compete with ensemble models and Bi-LSTM model with pre-trained word2vec embedding from 200 million words. In this case, the single classical machine learning achieved less than 2% difference in F1 compared to the deep learning approach with 1/18 time for model training.

## introduction automated short-answer scoring is the application of computer technologies to assist human grader in evaluating the score of written answers @xcite . the first track of ukara 1.0 is the binary classification version of short-answer scoring, where participants are expected to develop a model that can distinguish right and wrong answers in free text format. the organizer published the questions, the labels' responses, and the guideline on how to determine whether an answer is acceptable. during a period of five weeks, the training set and the development set were available, and we could validate our model through the score of the development set in a leaderboard. subsequently, the test set was released, which consists of roughly four times the size of the development set. we are required to submit predicted labels based on the model that we have developed, and the winner was determined by the f1 score of the submitted prediction. our final submission to this task consists of feature extraction, such as n-grams and tf-idf, and classical machine learning algorithms, namely logistic regression and random forest. moreover, we tested a combination of classical algorithms through ensemble learning and deep learning approach. we have published our implementation for reproduction foot_0 . we discuss the dataset and our approach in the following sections. ## datasets the dataset consists of two questions and the respective responses collected by the organizer of the challenge. all questions and responses are in indonesian. the first question ("task a") asked about the consequence of climate change. concretely, what are the potential problems faced by a climate refugee when they migrate to a new place? the second question, referred to as task b, is based on an experiment. potential customers initially wanted to buy clothes, prefer to donate the money instead, when they are presented with videos of the clothes manufacturing worker condition before paying. the respondents were required to give their opinion on why do people decided to change their minds. the responses statistics for both tasks are shown in ## methodology we split our approach into three categories. firstly, we have a single classical approach, where we employed simple logistic regression and random forest. next, we experimented with ensemble learning by combining four different classical machine learning algorithms. finally, an lstm-based neural network model is applied. we elaborate on the preprocessing and modeling steps in the following paragraphs. ## discussion based on the results shown in to analyze how hard it is to separate the right from the wrong answers, we reduced the dimensionality of the data into 2d using 1-gram, svd, and t-sne @xcite . figure 1 suggests that it is harder to separate the two answers in task b. on the other hand, we see more concentrated data points from wrong answers in task a. a similar phenomenon can also be observed in figure 2 and figure 3 . we can see a lot more data points in the 0.4-0.6 prediction range in figure 3 . figure 2: single model prediction (random forest) with probability on task a figure 3: single model prediction (logistic regression) with probability on task b in semantic similarity, mohler and mihalcea (2009) compared tf-idf, wordnet, and lsabased models for short-answer scoring. mohler et al. (2011 in this study, we did not use wordnet due to the unavailability of a rich lexical similarity database in bahasa indonesia. moreover, in our shared task, the organiser also gave the binary labels for each answer and coding guidelines instead of the expected right answers. ## conclusions in this report, we compare three approaches for automatic indonesian short-answer scoring using the ukara 1.0 dataset: single, ensemble, and deep learning. albeit being more sophisticated, we found that the f1 score difference is insignificant when we compare the single model to the ensemble and the deep learning approach. the 1.1% increment in the deep learning model can help clinch the top position in a competition, but it is not worth the required resources for practical settings. we posit that the primary reason is due to the noisy labels. as the classical models require only 1 18 of the total time to train deep learning models, we should use simple models and allocate more time for improving dataset quality.