You’ve Got a Friend in ... a Language Model? A Comparison of Explanations of Multiple-Choice Items of Reading Comprehension between C hat GPT and Humans (2023)

Abstract: Creating high-quality multiple-choice items requires careful attention to several factors, including ensuring that there is only one correct option, that options are independent of each other, that there is no overlap between options, and that each option is plausible. This attention is reflected in the explanations provided by human item-writers for each option. This study aimed to compare the creation of explanations of multiple-choice item options for reading comprehension by ChatGPT with those created by humans. We used two context-dependent multiple-choice item sets created based on EvidenceCentered Design. Results indicate that ChatGPT is capable of producing explanations with different type of information that are comparable to those created by humans. So that humans could benefit from additional information given to enhance their explanations. We conclude that ChatGPT ability to generate explanations for multiple-choice item options in reading comprehension tests is comparable to that of humans.

## introduction chatbots are used in education because they "promise to have a significant positive impact on learning success and student satisfaction" and "are promising tools to provide continuing feedback to lecturers and students" @xcite . according to @xcite , chatbots have been utilized in education to support learning and teaching, enhance services offered by educational institutions, promote well-being, and offer feedback and motivation. however, their use in assisting with the development of evaluation items, whether formative or summative, has not been widely explored. the process of providing explanations for multiple-choice items can be more time-consuming and labor-intensive than constructing the item itself. this often results in the creation of numerous items that lack explanations. despite this, it is important to note that an item accompanied by explanations is significantly more versatile and useful than one without. furthermore, the process of constructing explanations can reveal issues with the items that may not have been immediately apparent. as such, the implementation of a tool to assist item constructors in developing explanations could greatly enhance both the quantity and quality of items produced. recent advancements in language models, such as chatgpt, which have been trained on large amounts of text, show promise in their ability to assist with this task. this paper aims to investigate the efficacy of these models in comparison to explanations generated by humans. ## results and discussion we utilized a traditional test approach of simply presenting the item to chatgpt. out of the total 88 items, chatgpt answered 78 correctly, resulting in the exclusion of 10 items. an analysis of the common information pro-what does the author refer to when using the word "wonders" in the last paragraph of the text? a. that the tools were very old. *b. to the ability of hunters to build effective tools. c. to the ease with which hunters caught mastodons. d. that the tools were very beautiful. human explanation chatgpt explanation a. the attribution of wonders is not given to the tools for their antiquity. the value is given to the ability of the hunters and the effectiveness of the tools, for this reason, the options a, c and d are not valid. the option a, "that the tools were 'very old", is incorrect as an answer to the item . . . in the last paragraph of the text, the author mentions that . . . these tools were "wonders", suggesting they were very good or effective . . . instead, the option a suggests that the tools were very old, which . . . therefore, the option a is incorrect as an answer to the item. *b. in this case, the item requires students to deduce what the text attributes the character of "wonders" in the text, according to the statement . . . that appears at the end of the news. the semantic feature of wonders does not refer only to the tools but also to the ability of hunters to build effective tools, therefore, the correct option is b. the author refers to the ability of hunters to build effective tools when using the word "wonders" in the last paragraph of the text. in the last paragraph of the text, the author mentions that . . . these tools were "wonders", suggesting that they were very good or effective . . . therefore, it is possible to infer that the author uses the word "wonders" . . . c. the attribution of wonders is not given to the tools for the ease with which hunters caught mastodons. the value is given to the ability of the hunters and the effectiveness of the tools, for this reason, the options a, c y d are not valid. the option c, "to the ease with which hunters caught mastodons", is incorrect as an answer to the item . . . in the last paragraph of the text, the author mentions that . . . these tools were "wonders", suggesting that they were very good or effective . . . instead, the option c suggests that hunters caught mastodons easily, which . . . therefore, the option c is incorrect as an answer to the item. d. the attribution of wonders is not given to the tools because the tools were beautiful. the value is given to the ability of the hunters and the effectiveness of the tools, therefore, the options a, c y d are not valid. the option d, "that the tools were very beautiful", is incorrect as an answer . . . in the last paragraph of the text, the author mentions that . . . these tools were "wonders", suggesting that they were very good or effective . . . instead, the option d suggests that the tools were very beautiful, which . . . therefore, the option d is incorrect as an answer to the item. the length of prose between humans and chatgpt when expressing the same content. however, the effect size of this difference is small. with respect to these differences in additional explanations (last row), chatgpt texts are considerably larger than those of humans, exhibiting a large effect size. the length of the explanations can be attributed to two factors: firstly, humans tend to be direct in their responses and may not offer additional information, as evident from the minimal or non-existent explanations provided by 9th and 11th grades for the aii tag. furthermore, the results for the ci tag suggest that humans do not provide closure explanations. secondly, chatgpt explanations tend to be longer due to various factors, such as repetition the option, quoting fragments of the context, reiterating the status of the option (correct or incorrect) or all of the above. thus, chatgpt performed information expansion for almost half of the explanations (as seen in the aic tag) and added a closure for one-fifth of the explanations. when comparing the length of explanations for correct vs. incorrect options the only significant differences were observed in aii, aic and ci tags. this let us conclude that humans prefer to provide additional explanations to correct options, while chatgtp does it to incorrect options, both with a medium effect size in the difference of their respective preferences. similarly, for the functional content, humans preserve such preference, while chatgtp do the same but not significantly. when comparing the length of explanations for correct versus incorrect options, significant differences were observed only in the aii, aic, and ci tags. this allows us to conclude that humans prefer to provide additional explanations for correct options, while chatgpt does so for incorrect options, both with a medium effect size in the difference of their respective preferences. similarly, for functional content, humans maintain this preference, while chatgpt does the same but not significantly. more importantly, both humans and chat-gpt agree on not having differences in the length of the main common explanations for correct or incorrect options. another important factor is the ability of chat-gtp of identifying the correct option. among the 31 items to grade 5, chatgpt fails to provide a correct answer for item 20 in the 2022 dataset. in its explanation for the four options, chatgpt notes that "there has been no mention of option x being wrong at any point", and gives an explanation for each option. although chatgpt did not choose the correct option for this item, its explanations were compared to determine their accuracy. chatgpt considered that each incorrect option (distractor) could be correct. however, upon comparing these explanations to those created by humans, it was determined that chatgpt's explanations were incorrect. a more extensive discussion of these results exceeds the scope of this paper and is left for future research as the information is available in the spreadsheet. as a result, this item was deemed invalid and excluded from the analysis. for the remaining 30 items, it can be seen that the explanation given by humans and chatgpt have matched in the meaning, but with different words (mic). regarding the tags aii and aic, it is evident that aic is more commonly used. this suggests that chatgpt often includes additional information to provide a more comprehensive explanation, based on either a specific portion or the entire context. something similar occurs with the ci and cc tags, where the latter occurs slightly more frequently than the former. it is evident that approximately one-third of the explanations for the correct options have a concluding statement provided by both humans and chatgpt. finally, regarding the eic label, humans and chatgpt explanations rarely coincide and expand by referencing parts of the context. similar to the previous findings, the comparison of explanations by grade, tag, and correctness reveals that chatgpt tends to provide more information than humans, albeit not uniformly for all options. another factor in which chatgpt may fail is its inability to affirm that other options are incorrect, even when it has correctly chosen the correct option. in item 14 of grade 5 from 2021, chatgpt indicates that "option a is not incorrect" and provides an explanation to support this assertion. among the 25 items to grade 9, chatgpt fails to provide a correct answer for item 15 in the 2021 and item 1 in the 2022. regarding item 15, chat-gpt selected option b, which is incorrect. this may be because chatgpt omitted or confused some words. the text states that "simone reflects on her own life as a woman and after this reflection, publishes the book the second sex", but as the human who built this item says in his/her explanation: "[said quote] is not a reflection on sex". regarding item 1, chatgpt did not select any of the four options as correct and in its explanations it stated that the word whose meaning is contrary to the word "illegal" is "legal", thus omitting the correct option: "allowed". this may be because chatgpt discarded the presented context in which the word "allowed" (permitidas) fits as the semantic opposite of the word "illegal". in item 20 of grade 9 of 2021, chatgpt states for options a and c that "there is not enough information in the text to determine if the option . . . is incorrect or not. the text does not explicitly mention who it is intended for". among the 36 items to grade 11, chatgpt fails to provide a correct answer for 5 items of 2021 (1, 14, 16, 17, 20) and 2 items of 2022 (12, 20). in item 1, chatgpt selected option a, which is incorrect. this option asserts that the argumentative relationship between the two presented statements is one of premise and evidence, while the correct option provided by the human is d, which asserts that the relationship is conjecture and counterevidence. regarding the remaining items, items 14 and 16 belong to the same context, and their responses are derived from the same fragment. furthermore, these items ask for information that is explicit in the text (claim 1), making it uncommon for chat-gpt to provide incorrect answers. similarly, items 17 and 20 are associated with the same context (but different from before), although their answers are derived from different fragments, and they pertain to different claims (2 and 3, respectively). in general terms, chatgpt could have provided incorrect responses, but the comparison of these types of explanations precisely calls for a more in-depth analysis, which falls outside the scope of this work. in three items of the grade 11, chatgpt correctly identified the correct option but did not provide a conclusive explanation for why the other options were incorrect. in item 13 of 2021, chatgpt indicates that "option x is not incorrect" and adds that "this option does seem to be a reason for the character's feeling of unease". this type of explanation may be due to the format of the question, which uses a negation structure: "which not". for items 10 and 11 of 2022, which are based on a fragment of hamlet's monologue, chat-gpt provided explanations that diverged from the expected pattern. for item 10, chatgpt stated that both options a and c "is not incorrect, but rather one of the options that can adequately describe the above text". the explanation then goes on to clarify why the adjective "philosophical" (option a) or "poetic" (option c) would be a better fit for the fragment. in item 11, something similar to item 13 of 2021 occurs, where chatgpt selected the correct option, but regarding the incorrect options, it indicated that "it is not incorrect, but rather it is a statement that hamlet mentions and reflects on in his monologue". the question for this item also had the "which not" structure: "based on the above text, which of the following statements would hamlet not agree with?". it is worth noting that in chatgpt's explanation for the correct option, it also states that options a, b, and d are incorrect, since it states: "the other statements (a, b and d) do seem to agree with the content of hamlet's monologue". given the above, only the explanation for the correct option was compared, while the explanations for the remaining incorrect options were not evaluated. due to space constraints, we cannot fully explore the analysis of these differences in this study. regarding the classification of the items in the three claims, chatgpt correctly classified 49 (55.68 %) items into the three claims. below are the items that were classified in a wrong claim. among the items of the grade 5, 14 were classified incorrectly. items 3, 5, 7, 10, 18, and 19 were reclassified from claim 3 to claim 2, while items 3, 4, 6, 11, 12, and 15 were reclassified from claim 2 to claim 1. additionally, items 7 and 11 were reclassified from claim 3 to claim 1. something similar occurs with grades 9 and 11, where in the former 11 items were misclassified, while in the latter 14 items were misclassified. we hypothesize that chatgpt misclassifies certain items from claim 3 due to the presence of quoted expressions from the context in the question. it appears that chatgpt interprets these quotes as literal text and consequently categorizes them under claim 1, but we are unable to delve further into the analysis of this classification by chatgpt in this study and further investigation is needed. ## conclusion and future work this study provides insights into the creation of explanations for multiple-choice item options in reading comprehension tests with the assistance of ai. by comparing explanations generated by chat-gpt with those created by humans, our analysis indicates that chatgpt can produce explanations that could be considered equivalent and possibly better than those created by humans, with potential benefits for both humans and language models. chatgpt can offer more detailed and specific insights into the text, which can enhance the quality of explanations provided by humans. however, our findings also suggest that there is still room for improvement for both humans and language models. to address these limitations, future research could explore ways to combine the strengths of humans and language models to produce even more accurate and informative explanations. therefore, chatgpt has the potential to assist teachers and other professionals in the creation of high-quality assessment items through a well-designed prompt, which can help ensure that items have a single correct answer, independent options, non-overlapping options, and plausible options. furthermore, chat-gpt ability to classify items based on ecd principles is promising, but further research is needed. for example, the evidences could be provided to language models and ask them to classify each item in one of them. also, they could be asked to create the options and the respective explanations based on some kind of guidelines such as the one cited.