From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items (2024)

Abstract: LLMs can now perform a variety of complex writing tasks. They also excel in answering questions pertaining to natural language inference and commonsense reasoning. Composing these questions is itself a skilled writing task, so in this paper we consider LLMs as authors of commonsense assessment items. We prompt LLMs to generate items in the style of a prominent benchmark for commonsense reasoning, the Choice of Plausible Alternatives (COPA). We examine the outcome according to analyses facilitated by the LLMs and human annotation. We find that LLMs that succeed in answering the original COPA benchmark are also more successful in authoring their own items.

## introduction large language models (llms) can perform complex writing tasks ranging from paraphrasing sentences to composing long-form stories. because success on these open-ended authoring tasks is hard to measure quantitatively, nlp researchers tend to focus on more constrained tasks when judging the abilities of llms. many of these assessment tasks, or benchmarks, are conceptually connected with capabilities relevant to authoring, but they use a specific data design to support quantitative evaluation. for instance, assessments of commonsense reasoning often involve multiple-choice question answering, and accuracy on these questions is a proxy indicator for a model's ability to write coherently. new llms have excelled on these types of evaluations, but how this success relates to the llms' authoring capabilities is still unclear. creating these assessments is itself a skilled writing task, which up to this point has been performed by human authors. given their success in answering assessments, in this paper we examine whether llms can author assessment items as well. we prompt llms to generate items in the style of one well-known benchmark for commonsense reasoning, the choice of plausible alternatives (copa). we examine the outcome in terms of the llms' responses to their own generated items as well to what degree the items meet the benchmark authoring standards as judged by human raters. we show that an llm's authoring success is associated with its success in answering the original benchmark. ## copa the choice of plausible alternatives (copa) @xcite is an english-language benchmark that assesses the task of commonsense causal reasoning. as shown by the examples in premise: the girl received a trophy. what was the cause of this? alternative 1: she won a spelling bee. alternative 2: she made a new friend. premise: i tipped the bottle. what happened as a result? alternative 1: the liquid in the bottle froze. alternative 2: the liquid in the bottle poured out. from an authoring perspective, creating a copa item involves writing a collection of one-sentence narrations of events. the author writes two events that have a clear cause-and-effect relation, which respectively become the premise and the more plau-sible alternative (mpa). the author also writes an event that does not have a clear causal relation to the premise, and this becomes the less plausible alternative (lpa). as described in @xcite , composing the lpa is particularly challenging. this is because the benchmark is intended to assess models' ability to isolate causal relations in text separate from more generic associations that are captured by simple lexical co-occurence statistics. the author is expected to write an lpa that has some semantic and/or temporal relation to the premise, but no salient causal relation. until only recently, copa was considered a difficult benchmark. when first presented as a shared task at semeval-2012, submitted systems achieved accuracy in the 60-65% range @xcite . when copa was incorporated into the well-known superglue ensemble of benchmarks in 2019, the best system reached almost 85% @xcite . today some llms obtain nearperfect accuracy, as we observe in this work. this reflects the life cycle of most benchmarks, where existing ones are continuously replaced by harder new ones once maximal performance is obtained. ## research questions in this work, we apply a common llm interaction paradigm (prompting with exemplars) to facilitate llms to generate copa-style items. moving forward in this paper, we refer to the original copa items as orig-copa and the llm-authored items as gen-copa. when existing research refers to the copa task, it means the task of predicting the answer (i.e. more plausible alternative) to each item. here, we more specifically refer to this task as answering copa, in order to distinguish it from the task of generating copa items. we are interested in the following research questions. first, can llms author items with the design of copa? in particular, how often do gen-copa items meet the same authoring standards as the original benchmark? second, when an llm produces its own gen-copa set, does it then answer its own items correctly? in other words, does the llm behave consistently during generation and answering in what it deems as the more plausible alternative? third, how does an llm's success in authoring gen-copa items relate to its ability to correctly answer orig-copa items? are llms that do well on orig-copa also better at authoring new copa items? ## selected llms to investigate these questions, we employ a variety of notable open-source llms. we consider eleven models from six different families, where models within the same family are distinguished by size (number of parameters). bloom-7b & bloom-176b foot_0 : the big-science large open-science open-access multilingual language model (bloom) family was developed through a large cross-team research collaboration @xcite . it was trained on a collection of publicly available datasets that together comprise 1.61 terabytes of text spanning a wide variety of natural and programming languages @xcite . its performance on various benchmarks in superglue was competitive with other open-source models. falcon-7b & falcon-40b foot_1 : the falcon family @xcite developed by the technology innovation institute was trained on 1-1.5 trillion tokens, primarily the refinedweb corpus derived from commoncrawl @xcite . the 40b-sized model obtained top performance on several benchmarks upon its release. llama2-7b , llama2-13b , & llama2-70b 3 : the llama2 family developed by meta was trained on a collection of publicly available datasets comprising 2 million tokens @xcite . it has outperformed falcon and mpt (below) on several benchmarks. mistral-7b 4 mistral-7b (jiang et al., 2023), developed by mistral ai, was trained on open web data. upon its release, it was presented as a key competitor of the llama2 models. it outperformed the much larger llama2-13b on various knowledge and reasoning benchmarks. mpt-7b & mpt-30b 5 : the mosaicml pretrained transformer (mpt) family developed by mosaicml was trained on 1 billion tokens from a mix of web text and curated data (mosaicml nlp team, 2023). it has performed favorably compared with other open-source llms on benchmarks including copa. phi-2 6 : phi-2 (javaheripi and bubeck, 2023), developed by microsoft, is a 2.7b parameter model prompt template prompt example {% for ex in exemplars %} premise: {{ex['premise']}} {% if ex['direction'] == 'backwards' %}what was the cause of this?{% else %}what happened as a result?{% endif %} alternative 1: {{ex['alt1']}} alternative 2: {{ex['alt2']}} the more plausible {% if ex['direction'] == 'backwards' %}cause{% else %}result{% endif %} is alternative {{ex['more_plausible_alt']}}. {% endfor %} premise: {{item['premise']}} {% if item['backwards'] == 'cause' %}what was the cause of this?{% else %}what happened as a result?{% endif %} alternative 1: {{item['alt1']}} alternative 2: {{item['alt2']}} the more plausible {% if item['backwards'] == 'cause' %}cause{% else %}result{% endif %} is alternative premise: my body cast a shadow over the grass. what was the cause of this? alternative 1: the sun was rising. alternative 2: the grass was cut. the more plausible cause is alternative 1. [...3 more exemplars...] premise: my favorite song came on the radio. what happened as a result? alternative 1: i covered my ears. alternative 2: i sang along to it. the more plausible result is alternative ## orig-copa answering performance we first consider how well these llms answer the original copa benchmark. we used a few-shot prompting approach to elicit answers for copa items. in particular, we selected four items from the copa development set as task exemplars to include in all prompts for the 500 items in the test set. we applied the llms 8 outlined in section 4 to these prompts for the copa test set. we used greedy decoding (i.e. selecting the maximum probability token at each generation step) to ensure the predictions were deterministic. we generated outputs with a maximum length of four tokens, then post-processed each output with a regular expression to detect the presence of "1" or "2" as the 7 all prompt templates and code for reproducing this work are available at: github.com/roemmele/gen-copa 8 we ran all llms using huggingface inference endpoints: huggingface.co/inference-endpoints predicted answer. our policy was to randomly select one of these numbers if neither was present in the output, but in our experiments this was not applied since all outputs contained a 1 or 2. ## related work and outlook researchers are increasingly turning to llms to replace human effort in developing and evaluating nlp systems. in particular, llms are being used to synthesize labeled data in order to train or finetune models @xcite @xcite . llms are also being applied to score the output quality of other models @xcite @xcite . our work aligns with the above endeavors in exploring the use of llms as an alternative to human authoring of evaluation items. our particular goal in using llms for this assessment authoring is to better understand their capabilities, rather than to produce a novel benchmark that is valuable for evaluating future llms. however, there are some emerging demonstrations of using llms to facilitate the creation of evaluation data that is unique in its design and scope. for instance, @xcite described prompting an llm to derive multiple-choice q&a sets from short sections of text, which are then used to evaluate the same llm's ability to answer those questions when the relevant information is provided in one long document. as another example, @xcite utilized an llm as an interactive tool combined with human feedback in the authoring process for a novel challenge set focused on physical problem-solving. given the short lifecycle of benchmarks, rapid creation of new ones is pivotal to capturing further progress in nlp. additionally, because they do not serve a broader purpose outside of research, the authoring standards for benchmarks are not always clearly understood by the human authors tasked with their creation, and these standards are not necessarily well-anchored to other real-world writing tasks. the current paradigm of llm interaction via few-shot prompting suggests that llms can perform sophisticated authoring tasks in the absence of explicit guidelines just by observing representative examples. as current benchmarks quickly age and new benchmarks become more complex in what they aim to measure, llms are likely to take on a sanctioned role in their development. ## conclusion this paper looks at a notable assessment of commonsense reasoning, copa, as an authoring task performed by llms. the results indicate that models that answer copa items correctly (both llmauthored and human-authored ones) are also better at writing copa items. in future work, we will investigate how this extends to other benchmarks. our work bears upon the trend of widening generalizability of nlp models across tasks. previously, models designed for text generation were not considered directly applicable to commonsense question answering benchmarks, because this answering task was presumed to require a distinct model architecture supporting answer label prediction. this is no longer a constraint in the new paradigm of llms. increasingly, we are observing models demonstrate the same core knowledge across highly diverse task representations.