Original: C IS 2 : A Simpliﬁed Commonsense Inference Evaluation for Story Prose
Revised: C IS 2 : A Simpliﬁed Commonsense Inference Evaluation for Story Prose

Original: Anonymous CSRR Workshop submission

Original: Abstract
Revised: Abstract

Original: Contextual Commonsense Inference (CCI) isthe problem of inferring causal relations between the events of a text, such as a story.
Revised: Contextual Commonsense Inference (CCI) is the problem of inferring causal relations between the events of a text, such as a story.

Original: Like other commonsense reasoning tasks, CCIis a problem of language understanding, ratherthan language generation.
Revised: Like other commonsense reasoning tasks, CCI is a problem of language understanding, rather than language generation.

Original: We show that priorwork, in using language generation to perform
Revised: We show that prior work, in using language generation to perform

Original: CCI, trains models that struggle on the CCItask in isolation.
Revised: CCI, trains models that struggle on the CCI task in isolation.

Original: This conﬂation of tasks isfurther exacerbated by evaluating with wordmatching based metrics such as BLEU.
Revised: This conﬂation of tasks is further exacerbated by evaluating with wordmatching based metrics such as BLEU.

Original: In order to isolate CCI from language generation,we reframe CCI as a classiﬁcation problem.
Revised: In order to isolate CCI from language generation, we reframe CCI as a classiﬁcation problem.

Original: Our system, which we call C IS 2 , forces themodel to focus on CCI directly by providingit the original text of the story to use for understanding while having it generate only thebare minimum: indices to sentences.
Revised: Our system, which we call C IS 2 , forces the model to focus on CCI directly by providing it the original text of the story to use for understanding while having it generate only the bare minimum: indices to sentences.

Original: We lookat the GLUCOSE (Mostafazadeh et al., 2020)dataset and compare against their task for predicting CCI between story sentences.
Revised: We look at the GLUCOSE (Mostafazadeh et al., 2020) dataset and compare against their task for predicting CCI between story sentences.

Original: Weﬁnd that models trained on C IS 2 index labelsachieve a 4% higher CCI accuracy than thosetrained for generation, such as in the original GLUCOSE task.
Revised: We ﬁnd that models trained on C IS 2 index labels achieve a 4.3% higher CCI accuracy than those trained for generating full phrases, such as in the original GLUCOSE task.

Original:  Introduction
Revised:  Introduction

Original: Transformer-based  models (Vaswani et al., 2017)have shown mixed success with story generation (See et al., 2019).
Revised: Transformer-based language models (Vaswani et al., 2017)—particularly off-the-shelf models— have shown mixed success with story generation (See et al., 2019; Wang and Wan, 2019; Ippolito et al., 2020).

Original: Language models (LMs)lose coherence as the output length increases andare prone to meandering, losing the plot  over time.
Revised: Language models (LMs) lose coherence as their output length increases and are prone to meandering, losing the plot of a story over time.

Original: This can be largely attributed to the LM generatingeach token by sampling from a probability distribution, failing to distinguish between statisticalcorrelation (how frequently event A and event Bare seen together) and causal reasoning (event Acauses event B to occur).
Revised: This can be largely attributed to the LM generating each token by sampling from a probability distribution, failing to distinguish between statistical correlation (how frequently event A and event B are seen together) and causal reasoning (event A causes event B to occur).

Original: Since causal events across sentences in storieshelp people understand and retain story information (Trabasso et al., 1984), we posit that the inability of language models to perform commonsense inference leads them to output less coherent longform text.
Revised: Since causal events across sentences in stories help people understand and retain story information (Trabasso et al., 1984), we posit that the inability of language models to perform commonsense inference leads them to output less coherent long-form text.

Original: However, commonsense inference is still an open problem in NLP, especially when the commonsense information is unstructured and provided in the form of natural language.
Revised: Commonsense inference is still an open problem in NLP, especially when the commonsense information is unstructured and provided in the form of natural language.

Original: We refer to this task of grounding commonsense inference relations within prose as contextual commonsense inference (CCI) , a sub-task within commonsense reasoning.
Revised: We refer to this task of grounding commonsense inference relations within prose as contextual commonsense inference (CCI) , a sub-task within commonsense reasoning.

Original: Due to storytelling being deeply intertwined with causal understanding, improving CCI will yield  more accurate story generation evaluation metrics andtherefore better story generation. 
Revised: Due to storytelling being deeply intertwined with causal understanding, improving CCI will yield both more accurate story generation evaluation metrics and better story generation. Current methods in CCI for story understanding often include the use of generative LMs. While

Revised: LMs might be helpful for encoding the textual information, they are less suited to operating on and making decisions based on this information due to their probabilistic way of generating text.

Original:  Our contributions in this paper are twofold.
Revised: This leads to a tendency to focus on grammar rather than meaning (Martin et al., 2018). Furthermore, commonly-used language generation evaluation metrics like BLEU put emphasis on exact word usage and grammar. In this paper, we look at what it would mean to de-emphasize generation and para- phrasing for understanding tasks like CCI. Our contributions in this paper are twofold.

Original: First, we critique  existing methods addressing the contextual commonsense inference (CCI) task by using the GLUCOSE (Mostafazadeh et al., 2020) dataset and comparing against their associated CCI task formulation.
Revised: First, we critique an existing method addressing the contextual commonsense inference (CCI) task by using the GLUCOSE (Mostafazadeh et al., 2020) dataset and teasing apart their associated CCI task formulation.

Original: We design several diagnostic tasks which selectively omit sentences of the input and investigate which sentences contribute  most to paraphrasing/generation.
Revised: We designed several diagnostic tasks which selectively omit sentences of the input and investigate which sentences contribute the most to paraphrasing/generation.

Original: We replicate their results and ﬁnetune T5 models (Raffel et al., 2020) on each of our diagnostic tasks to show  signiﬁcant conﬂation  in the baseline GLUCOSE T5 model.
Revised: We replicate their results, then ﬁnetune T5 models (Raffel et al., 2020) on each of our diagnostic tasks, to show the signiﬁcant conﬂation of language understanding and generation in the original GLUCOSE T5 model.

Original: Second, we propose C IS 2 (Contextual Commonsense Inference in Sentence Selection), a simpliﬁed task for more fairly evaluating commonsense inference in storytelling, which abstracts away thenatural language generation component  entirely.
Revised: Second, we propose C IS 2 (Contextual Commonsense Inference in Sentence Selection), a simpliﬁed task for more fairly evaluating commonsense inference in storytelling, which abstracts away the natural language generation component almost entirely.

Original: We develop a heuristic to convert story sentencesinto C IS 2 tags and show that a language model,when trained on this data, outperforms the original GLUCOSE task formulation on forming  correct causal relations between sentences in stories.
Revised: We develop a heuristic to convert story sentences into C IS 2 tags and show that a language model, when trained on this data, outperforms the original GLUCOSE task formulation on forming the correct causal relations between sentences in stories.

Original: Our ﬁndings reinforce that while the GLUCOSEdataset encodes useful commonsense information,we emphasize that future work should carefullydisentangle language generation while performinglanguage understanding tasks.
Revised: Our ﬁndings reinforce that while the GLUCOSE dataset encodes useful commonsense information, we urge that future work should carefully disentangle language generation when performing language understanding tasks.

Original:  Related Work
Revised:  Related Work

Original: Commonsense inference is the ability to use priorknowledge based on real world experiences to inferwhat has happened or will happen in a text.
Revised: Commonsense inference is the ability to use prior knowledge based on real world experiences to infer what has happened or will happen.

Original: Whilelived experiences vary from person to person, thereare still signiﬁcant commonalities throughout ourbasic interactions with the world around us sincewe all live in the same physically and temporallyconstrained world. 
Revised: While lived experiences vary from person to person, there are still signiﬁcant commonalities as we live and interact within the same physically- and temporallyconstrained world. 2. Commonsense Knowledge Graphs

Original: Hwang et al.
Revised: Hwang et al.

Original: (2021) formalizethe commonsense inference task (CI) for AI systems as a knowledge three-tuple, to predict theobject of a relation given the subject and relation .
Revised: (2021) formalized the commonsense inference task (CI) for AI systems as a knowledge three-tuple, to predict the object of a relation given the subject and relation .

Original: 2. Commonsense Knowledge Graphs

Original: This formulation of commonsense inference easilylends itself to a graph structure, where the subjectsand objects(entities) are nodes and the relationsare the edges connecting the entities.
Revised: This formulation of commonsense inference can be structured as a graph, where the subjects and objects are nodes and the relations are the edges connecting the entities.

Original: This line ofwork on commonsense knowledge graphs (CKGs)explicitly encode the structure of inference relationships between entities.
Revised: These commonsense knowledge graphs (CKGs) explicitly encode the structure of inference relationships between entities.

Original: ATOMIC (Sap et al. 2019)is one such CKG dataset that organizes everydayevents into if-then relationships.
Revised: ATOMIC (Sap et al., 2019) is one such CKG dataset that organizes everyday events into if-then relationships.

Original: COMET (Bosselut et al., 2019) is a transformer language modeldesigned on top of ATOMIC relations, showinglanguage models can encode and generalize commonsense information.
Revised: COMET (Bosselut et al., 2019) is a transformer language model designed on top of ATOMIC relations, showing language models can encode and generalize commonsense information.

Original: However, Wang et al.
Revised: However, Wang et al.

Original: (2021) show that languagemodels struggle to perform generalizable commonsense inference across three popular CKG datasets:
Revised: (2021) show that language models struggle to perform generalizable commonsense inference across three popular CKG datasets:

Original: ConceptNet (Speer et al., 2017),ATOMIC (Sapet al., 2019), and TupleKB (Dalvi Mishra et al.,2017).
Revised: ConceptNet (Speer et al., 2017), TupleKB (Dalvi Mishra et al., 2017), and ATOMIC (Sap et al., 2019).

Original: Through a series of experiments, they foundthat LMs trained on several CKGs have limitedability to transfer knowledge to unseen CKGs, andthat adaptation generalizes well to unseen subjects,but less so on unseen objects.
Revised: They found that LMs trained on several CKGs have limited ability to transfer knowledge to unseen CKGs, and that adaptation generalizes well to unseen subjects, but less so on unseen objects.

Original: Although these graphs do well at representing facts and their relations, their statements lack context and would need to be adapted to a textual domain, such as story prose.
Revised: Although these graphs do well at representing facts and their relations, their statements lack context and would need to be adapted to a textual domain, such as story prose.

Original: Using them to generate a story as-is would fail to engage readers since the “story” would simply be a series of facts.
Revised: Using them to generate a story as-is would fail to engage readers since the “story” would simply be a series of facts.

Original: Our work goes beyond the explicit structure of CKGs, focusing on ﬁnding and leveraging commonsense relations in natural language short stories.
Revised: Our work goes beyond the explicit structure of CKGs, focusing on ﬁnding and leveraging commonsense relations in natural language short stories.

Original: 2. Commonsense Inference for Storytelling
Revised: 2. Commonsense Inference for Storytelling

Revised: Early research on automated story generation research focused on designing systems that create coherent stories (Lebowitz, 1985; Turner and Dyer, 1985; Liu and Singh, 2002;

Revised: Young et al., 2013).

Revised: Despite the success of neural networks for AI tasks, commonsense and coherence remain big issues for story generation systems.

Original: Applying commonsense reasoning on the events of a story has been proposed as one way to tackle the difﬁcult problem of assessing the quality of machine-generated stories.
Revised: Applying commonsense reasoning to the events of a story has been proposed as one way to tackle the difﬁcult problem of assessing the quality of machine-generated stories.

Original: The Story Cloze
Revised: The Story Cloze

Original: Test (Mostafazadeh et al., 2016) formulates story ending generation as a multiple-choice task, having systems look at several possible endings, and predict the one that is most reasonable.
Revised: Test (Mostafazadeh et al., 2016) formulates story ending generation as a multiple-choice task, having systems look at several possible endings and predict the one that is most reasonable.

Original: Guan et al.
Revised: Guan et al.

Original: (2019) integrated commonsense reasoning directly into their Story Cloze model by building context clues and using implicit knowledge.
Revised: (2019) integrated commonsense reasoning directly into their Story Cloze model by building context clues and using implicit knowledge.

Original: Commonsense reasoning can also help story generation with issues in plot coherence.
Revised: Commonsense reasoning can also help story generation with issues in plot coherence.

Original: The
Revised: Martin (2021) created a neurosymbolic system that leveraged VerbNet (Brown et al., 2019) facts to ground neural story generation in commonsense reasoning. They did this by tracking the story state and pruning out impossible options that a neural network [Table]

Original:  Commonsense inference Augmented neural StoryTelling (CAST) framework (Peng et al., 2021) introduces commonsense into the generation process, explicitly modeling interactions between multiple characters.
Revised: provided as candidate next sentences for the story. Similarly, the Commonsense inference Augmented neural StoryTelling (CAST) framework (Peng et al., 2021) modeled interactions between multiple characters using ATOMIC.

Original: The stricter, more explicit generation constraints of CAST produce more coherent and on-topic two-character stories that generating via sampling from a distribution alone.
Revised: The stricter, more explicit generation constraints of CAST produced more coherent and on-topic two-character stories than generating via sampling from a distribution alone.

Original: TellMeWhy (Lal et al., 2021) is a dataset built on top of ROCStories (Mostafazadeh et al., 2016), consisting of 30k questions on why characters perform their actions and the corresponding answers.
Revised: TellMeWhy (Lal et al., 2021) is a dataset built on top of ROCStories (Mostafazadeh et al., 2016), consisting of 30k questions on why characters perform their actions and the corresponding answers.

Original: They found that current state-of-the-art models performed far worse than humans, especially on questions whose answers are external to the narratives.
Revised: They found that current state-of-the-art models performed far worse than humans, especially on questions whose answers are external to the narratives.

Original: This contrasts with the ﬁndings discussed in
Revised: This contrasts with the ﬁndings discussed in

Original: Mostafazadeh et al.
Revised: Mostafazadeh et al.

Original: (2020) that language models can approach human performance.
Revised: (2020) that language models can approach human performance.

Original:  The GLUCOSE Dataset and Task
Revised:  The GLUCOSE Dataset and Task

Original: Our work follows fromthe GLUCOSE (GeneraLized and COntextualized Story Explanations) (Mostafazadeh et al., 2020) dataset and task.
Revised: Our work follows from GLUCOSE (GeneraLized and COntextualized Story Explanations) (Mostafazadeh et al., 2020).

Original: This dataset provides us with a set of commonsense reasoning relations between sentences of stories. The stories come from the ROCStories (Mostafazadeh et al., 2016) corpus, a collec-
Revised: In this section we brieﬂy describe their dataset and experiments; for more details, refer to the original paper.

Original: [Table]
Revised: The

Original: tion of crowdsourced ﬁve-sentence everyday storiesbuilt for the Story Cloze Test (see Section 2.2). Mostafazadeh et al. (2020) designed a multistage crowdsourcing platform to collect 670K annotated and quality-controlled causal reasoning relations between  sentences within each story.
Revised: GLUCOSE dataset contains 670K crowdsourced annotations identifying causal reasoning relations between the sentences within stories from ROCStories (Mostafazadeh et al. , 2016)—a collection of crowdsourced ﬁve-sentence everyday stories in

Original: Theystructured thedata they collected  around ten different dimensions of causal relations, inspired bycognitive psychology. The ten dimensions are given in Table 1.
Revised: English. The authors structured the collected data around ten different dimensions, shown in Table 1, of causal relations between a pre-selected sentence

Original: Theycapture causal explanations between a selected sentence X from the story and another statement Y ,which can either be another story sentence or external commonsense knowledge.
Revised: X from the story and another statement Y , which can either be another story sentence or some ex- [Table] ternal commonsense knowledge.

Original: The relationshipbetween these statements can be formalized as:statement 1 REL statement 2 ,
Revised: The relationship between these statements can be formalized as:

Original: [Equation]
Revised: [Equation]

Original: X can be in either statement position, depending onthe particular dimension chosen. The dimensions can be divided into two categories: dimensions 1-5, specifying the events thatcaused X (i.e. X is statement 21 ), and dimensions6-10, specifythe events caused by X (i.e. X isstatement 1 ).
Revised: X can be in either statement position, depending on the particular dimension chosen: Dimensions 1-5, specify events that caused X (i.e., X is statement 2 ), and dimensions 6-10 specify events caused by X (i.e., X is statement 1 ).

Original: The dimensions mirror each otheracross the two categories.

Original: For example, dimension 1 mirrors dimension 6, depending on whetherstatement Y (which is an event in this example) isaffecting X (1) or being affected by X (6).

Original: 3. Contextual Commonsense Inference Task
Revised: 3. Contextual Commonsense Inference Task

Original: GLUCOSE addresses thecontextual commonsenseinference (CCI) task of predicting relationship(s) [Table] between statements explicitly or implicitly expressed within a text.
Revised: GLUCOSE addresses the task of predicting relationships between statements explicitly or implicitly expressed within a text, a task we term contextual commonsense inference (CCI). An example

Original:  The entriesin their dataset are organized to reﬂect this and are formalized asa paired input-output tuples, withan input tuple (cid:104) story S , selected sentence X , dimension D (cid:105) , (2)and an output tuple (cid:104) speciﬁc rule R S , general rule R G (cid:105) , (3) where a story S consists of ﬁve sentences [ s 0 , s 1 , s 2 , s 3 , s 4 ], the selected sentence X is the sentence on which we center our rule, the number dimension D is one of the ten dimensions from Table 1, the speciﬁc rule R S is the relation betweenX and Y .
Revised: GLUCOSE entry can be found in Table 2. The entries are organized to reﬂect the CCI task and are formalized as input-output tuple pairs, with input tuple (cid:104) story S , selected sentence X , dimension D [Equation] (2) where a story S consists of ﬁve sentences [ s 0 , s 1 , s 2 , s 3 , s 4 ], the selected sentence X is the sentence on which the rule is centered, and the number dimension D is one of the ten dimensions from Table 1—and output tuple (cid:104) speciﬁc rule R S , general rule R G (cid:105) , (3) where the speciﬁc rule R S is the relation between

Original:  Y can be either (1) another sentence in the story or (2) an implicit statement from outside the text 2 .
Revised: X and Y . Y can be either (1) another sentence in the story or (2) an implicit statement from outside the text.

Original: and the general rule R G is the same relation but using generalized tags for named entities(e.g., Someone A instead of Fred).
Revised: The general rule R G is the same rule as R S but using generalized tags for named entities

Original: An example entry can be found in Table 2.
Revised: [Table] [Table] (e.g., Someone A instead of Fred).

Original: To summarize, the GLUCOSE task is: given S , X , and D , predict R S and R G .
Revised: To summarize, the GLUCOSE task is: given S , X , and D , predict/generate R S and R G .

Original: In the original experiment, Mostafazadeh et al.

Original: (2020) curated a test set of 500 “unambiguous examples with clear gold answers” and preserved the rest of the 670K examples for training.

Original: After training several models, they ran an automated evaluation using SacreBLEU (Post, 2018), where their best-performing model was a pretrained T model (Raffel et al., 2020) further ﬁne-tuned on GLUCOSE data. It achieved a 71.26 average [Table]
Revised: In this paper, we compare to their best model, a ﬁnetuned T5 model (Raffel et al., 2020), which achieved a 71.26 average SacreBLEU (Post, 2018) across the 10 dimensions on predicting general rules and a 75.65 average for the speciﬁc rules.

Original: BLEU across the 10 dimensions on predicting general rules and a 75.65 average for the speciﬁc rules 3 . They performed a human evaluation using a custom MTurk task, where workers rated model outputs(the speciﬁc and general rules) for correctness on a4-point Likert scale4 mapped to a numerical scaleof 0-3.Their T5 model scored an average 2.5/3across all 10 dimensions in the human evaluation(compared to 2.8/3 for gold standard answers).
Revised: The models were also rated for “correctness” using crowdsourcing, where their T5 model scored 2.5/ averaged across all 10 dimensions on a 4-point Likert scale mapped to a numerical scale of 0-3.

Original: Forcontext, their closest baseline got a 2.21/3 average.
Revised: For context, their closest baseline got a 2.21/3 average and the gold standard was 2.8/3.

Original: 3. Issues with the GLUCOSE Task for CCI
Revised: 3. Issues with the GLUCOSE Task for CCI

Original: We take no issue with the GLUCOSE dataset,which is well-designed and of good annotationquality.
Revised: We ﬁnd that the GLUCOSE dataset is welldesigned and of good annotation quality.

Original: However, we ﬁnd issue with the GLUCOSE task, which asks a model to perform twotasks simultaneously: commonsense inference andlanguage generation.
Revised: However, we take issue with the GLUCOSE task, which asks a model to perform two tasks simultaneously: commonsense inference and language generation.

Original: Due to this conﬂation of tasks,the model, in generating its output, would rely heavily on the already-good language generation abilityof T5 and neglect learning enough CCI.
Revised: Due to this conﬂation of tasks, the model, in generating its output, would rely heavily on the already-good language generation ability of T5 and neglect learning enough CCI.

Original: T5 (Raffel et al., 2020) and other transformer LMs were designed to perform language generation tasks, such as summarization or translation.
Revised: T5 (Raffel et al., 2020) and other transformer LMs were designed to perform language generation tasks.

Original: Therefore, even with the GLUCOSE task, T5 will focus on generating paraphrases of, or even directly copying, story sentences.
Revised: Therefore, by including text generation as part of CCI, T5 will focus on paraphrasing or even copying story sentences.

Original: Though our work focuses on the speciﬁc rules since it performed the best, we argue that the same issue applies to the general rule, given that only the entities are replaced with tags and the majority of the grammar remains the same.

Original: Furthermore, there are several one-to-one correspondences between parts of the input and output in the original GLUCOSE task (illustrated in Figure 1).
Revised: There are several one-to-one correspondences between parts of the input and output in the original GLUCOSE task (illustrated in Figure 1).

Original: For example, for all GLUCOSE entries, thetarget output contains at least onesentence paraphrased  from the input sequence.
Revised: For example, for all GLUCOSE entries, the output contains at least one paraphrased sentence from the input.

Original: Conﬂation with paraphrasing worsens with BLEU as the evaluation metric, since BLEU measures word overlap, meaning even incorrect commonsense inferences can score partial credit.
Revised: Conﬂation with paraphrasing worsens with BLEU as the evaluation metric, where incorrect commonsense inferences can score partial credit if they have words in common.

Original:  Diagnostic Tests
Revised:  Diagnostic Tests

Original: In this section, we describe our three diagnostic tests—variations on the original GLUCOSE task with altered input—to isolate different factors that inﬂuence T5’s generation.
Revised: In this section, we describe our three diagnostic tests—variations on the original GLUCOSE task with altered input—to isolate different factors that inﬂuence T5’s generation.

Original: Through these tests, we investigate the extent to which language models rely on paraphrasing to generate the commonsense rule output for GLUCOSE.
Revised: Through these tests, we investigate the extent to which language models rely on paraphrasing to generate the commonsense rule output for GLUCOSE.

Original: For each of the following diagnostic tests, we ﬁnetune the same T5 (Raffel et al., 2020) model, a pretrained model using the same hyperparametersas in the GLUCOSE paper, to generate the sameoutput as in Equation 3.
Revised: For each of the following diagnostic tests, we ﬁnetune the same T5 (Raffel et al., 2020) model, a pretrained model using the same hyperparameters as in the GLUCOSE paper, to generate the same output as in Equation 3.

Original: The diagnostic tests differonly in the format of the input.
Revised: The diagnostic tests differ only in the format of the input.

Original: The purpose ofthese tests was to assess how reliant the modelis on language generation when performing CCI.
Revised: The purpose of these tests was to assess how reliant the model is on language generation when performing CCI.

Original: More detailed training setup and hyperparametersfor these models can be found in Appendix A.5.
Revised: More detailed training setup and hyperparameters for these models can be found in Appendix A.5.

Original: Because these tasks are measured by BLEUscore, conﬂation between CCI and language generation will always occur.
Revised: Because these tasks are measured with BLEU, conﬂation between CCI and language generation will always occur.

Original: But by deleting differentparts of the input, these diagnostic tasks analyzewhich sentences contribute  most to performance,and thus result in more conﬂation.
Revised: But by deleting different parts of the input, these diagnostic tasks analyze which sentences contribute the most to performance, thus resulting in more conﬂation.

Original: Conﬂation always occurs for X , since this is known from theinput, and is also worse in cases where an incorrectstatement Y was generated, but it contains tokensthat match the correct statement.

Original: An overview of the tests’ different data formatscan be found in rows 2, 3, and 4 of Table 3.
Revised: An overview of the tests’ different data formats can be found in rows 2, 3, and 4 of Table 3.

Original: Wedescribe them in this section using the followingterminology for brevity:
Revised: We describe them in this section using the following terminology for brevity:

Original: Dimension (dim) : the causal dimension Pre-context : sentences before selected sentence X Selected sentence (X) : the story sentence of interest
Revised: Dimension (dim) : the causal dimension Pre-context : sentences before selected sentence X Selected sentence (X) : the story sentence of interest

Original: Post-context : sentences after selected sentence X
Revised: Post -context : sentences after selected sentence X

Original: O RIGINAL .
Revised: O RIGINAL .

Original: This experiment is the same as in(Mostafazadeh et al., 2020), which we described in
Revised: This experiment is the same as in (Mostafazadeh et al., 2020), which we described in

Original: Section 3.1.
Revised: Section 3.1.

Original: We report results on our own replication of the ﬁnetuned T5 model, implemented withthe transformers package (Wolf et al., 2019).
Revised: We report results on our own replication of the ﬁnetuned T5 model, implemented with the transformers package (Wolf et al., 2019).

Original: H ISTORY .
Revised: H ISTORY .

Original: This experiment gives as input onlythe pre-context (the sentences before sentence X )and the dimension.
Revised: This experiment gives as input only the pre-context (the sentences before sentence X ) and the dimension.

Original: This model must generate theoutput without knowing the target sentence X , northe events happening afterwards.
Revised: This model must generate the output without knowing the target sentence X , nor the events happening afterwards.

Original: Here, we testthe model’s ability to generate two (speciﬁc) statements given only what happened before.
Revised: Here, we test the model’s ability to generate two (speciﬁc) statements given only what happened before.

Original: This difﬁcult task serves as a lower bound to contextualcommonsense inference performance.
Revised: This difﬁcult task serves as a lower bound to contextual commonsense inference performance.

Original: Conﬂationwith language generation is absent.
Revised: Conﬂation with language generation is absent.

Original: For all dimensions, the model must ﬁrst speculate what X might be given the pre-context.
Revised: For all dimensions, the model must ﬁrst speculate what X might be given the pre-context.

Original: Basedon this predicted X, it generates a statement Y thatfollows from the causal relationship: either a paraphrase from the input or an implied statement.
Revised: Based on this predicted X, it generates a statement Y that follows from the causal relationship: either a paraphrase from the input or an implied statement.

Original: Masked Selected Sentence (M ASK X).
Revised: Masked Selected Sentence (M ASK X).

Original: Thisexperiment gives as input the pre-context, postcontext, and the dimension.
Revised: This experiment gives as input the pre-context, postcontext, and the dimension.

Original: The selected sentenceis replaced with a token <masked> .
Revised: The selected sentence is replaced with a token <masked> .

Original: Here, we test[Table] the commonsense ability to generate two (speciﬁc) statements given most of the story – 4 out of 5 sentences – but not the selected  X sentence.
Revised: Here, we test the commonsense ability to generate two (speciﬁc) statements given most of the story—4 out of sentences—but not the selected sentence X .

Original: This setup partially avoids language generation conﬂation.
Revised: This will let us see how much of a performance boost the model is given by copying X from the input.

Original: As with H ISTORY , for all dimensions, the model must ﬁrst predict X , then generate a paraphrased or implied statement Y that is causally consistent. 
Revised: As with H ISTORY , for all dimensions, the model must ﬁrst predict X , then generate a paraphrased or implied statement Y that is causally consistent. [Table]

Original: History and Selected Sentence (H ISTORY +X).
Revised: History and Selected Sentence (H ISTORY +X).

Original: This experiment gives as input the pre-context, selected sentence, and dimension.
Revised: This experiment gives as input the pre-context, selected sentence, and dimension.

Original: This is used as a direct comparison to H ISTORY except with selected sentence X given as part of the input.
Revised: This is used as a direct comparison to H ISTORY except with selected sentence X given as part of the input.

Original: Statement Y is generated as it is in H ISTORY .
Revised: Statement Y is generated as it is in H ISTORY .

Original: For these diagnostic tests, we drop entries in which the modiﬁcations result in input identical to the original task.
Revised: For this diagnostic test, we drop entries in which the modiﬁcations result in input identical to the original task.

Original: For example, for H ISTORY +X, we omit those entries where X is the last sentence.
Revised: For example, for H ISTORY +X, we omit those entries where X is the last sentence.

Original: 4. Diagnostic Task Results
Revised: 4. Diagnostic Task Results

Original: Table 4 compares the results of T5 models trained on the diagnostic tasks.
Revised: Table 4 compares the results of T5 models trained on the diagnostic tasks.

Original: We report test set results on the averaged dimensions 1-10, as well as averaged dimensions 1-5 ( X is the second statement), and 6-10 ( X is the ﬁrst).
Revised: We report test set results on the averaged dimensions 1-10, as well as averaged dimensions 1-5 ( X is the second statement), and 6-10 ( X is the ﬁrst).

Original: Following Mostafazadeh et al.
Revised: Following Mostafazadeh et al.

Original: (2020), we use SacreBLEU (Post, 2018) with equal weights up to 4-grams.
Revised: (2020), we use SacreBLEU (Post, 2018) with equal weights up to 4-grams.

Original: We report results for both speciﬁc and general rules, but focus on speciﬁc.
Revised: We report results for both speciﬁc and general rules, but focus on speciﬁc.

Original: O RIGINAL , of course, performs the best as its input has the most available information.
Revised: O RIGINAL , of course, performs the best as its input has the most available information.

Original: H ISTORY and M ASK X perform similarly to each other and far worse than the other diagnostic tasks.
Revised: H ISTORY and M ASK X perform similarly to each other and far worse than the other diagnostic tasks.

Original: H ISTORY , with only the pre-context, has a a 35-point BLEU gap for speciﬁc rules (16 for general) compared to
Revised: H ISTORY , with only the pre-context, has a a 35-point BLEU gap for speciﬁc rules (16 for general) compared to

Original: O RIGINAL averaged across all dimensions.
Revised: O RIGINAL averaged across all dimensions.

Original: Adding to H ISTORY multiple sentences of the post-context gives M ASK X, and modest score gains (35.9 vs 41.6 speciﬁc).
Revised: Adding to H ISTORY multiple sentences of the post-context gives M ASK X, and modest score gains (35.9 vs 41.6 speciﬁc).

Original: However, adding to H ISTORY just the one selected sentence X gives H ISTORY +X, which performs very closely to O RIGINAL for both speciﬁc and general rules(70.7 vs 68.3 speciﬁc).
Revised: However, adding to H ISTORY just the one selected sentence X gives H ISTORY +X, which performs very closely to O RIGINAL for both speciﬁc and general rules (70.7 vs 68.3 speciﬁc).

Original: Furthermore, comparingtrends between dimensions 1-5 and 6-10, we ﬁndthat 6-10 scores are mostly higher, for both generaland speciﬁc, than 1-5.
Revised: Furthermore, comparing trends between dimensions 1-5 and 6-10, we ﬁnd that 6-10 scores are mostly higher, for both general and speciﬁc, than 1-5.

Original: These results and their trends show that BLEUscores are highly contingent on having X as inputover all other sentences.
Revised: These results and their trends show that BLEU scores are highly contingent on having X as input over all other sentences.

Revised: Conﬂation always occurs for X , since this is copied from the input, and conﬂation is also worse in cases where an incorrect statement Y was generated but contains tokens that match the correct statement.

Original: The ﬁne-tuned T5 modelsperform some CCI, but BLEU scores are hard tointerpret and look unreliable. Does ∼ 35.9 BLEUon speciﬁc rules for H ISTORY  mean  it is half asgood at CCI than O RIGINAL , with 70.7 BLEUspeciﬁc? This is unlikely to  be the case.
Revised: We believe it is unlikely that achieving ~ 35.9 BLEU on speciﬁc rules for H ISTORY would mean that it is half as good at CCI than O RIGINAL , with 70.7 BLEU speciﬁc. We found that the ﬁne-tuned T5 models perform some CCI, but BLEU scores are hard to interpret and can be unreliable.

Original: Speciﬁc vs. General Rule Performance
Revised: Speciﬁc vs. General Rule Performance

Original: Table 4 shows that both O RIGINAL and H ISTORY +Xperform better for speciﬁc rules than general.
Revised: Table 4 shows that both O RIGINAL and H ISTORY +X perform better for speciﬁc rules than general.

Original: Thismatches the results seen in Mostafazadeh et al.(2020), but is still interesting as one might expectthat general would be easier – we don’t need tocopy and memorize from the original input.
Revised: This matches the results seen in (Mostafazadeh et al., 2020).

Original: However, for H ISTORY and M ASK X, whichboth omit X , the opposite trend occurs. Generalis higher than speciﬁc, as would be intuitively expected.
Revised: However, for H ISTORY and M ASK X, which both omit X , the opposite trend occurs; general is higher than speciﬁc.

Original: This shows that copying and paraphrasingfrom the original text is in fact a conﬂating factorin the LM’s BLEU performance.
Revised: This shows that copying and paraphrasing from the original text is in fact a conﬂating factor in the LM’s BLEU performance.

Original:  Contextual Commonsense Inference in 
Revised:  Contextual Commonsense Inference in Sentence Selection (C IS 2 )

Original: Sentence Selection (C IS 2 )

Original: Given the extensive paraphrasing present in boththe GLUCOSE task and the evaluation method, wedesign the Contextual Commonsense Inference in
Revised: Given the extensive paraphrasing present in both the GLUCOSE task and the evaluation method, we design the Contextual Commonsense Inference in

Original: Sentence Selection (C IS 2 ) task to abstract awaylanguage generation.
Revised: Sentence Selection (C IS 2 ) task to abstract away language generation.

Original: We recast the task as a classiﬁcation problem, with the same 3 inputs as in
Revised: We recast the task as a classiﬁcation problem, with the same 3 inputs as in

Original: O RIGINAL (Equation 2), while the output becomes 
Revised: O RIGINAL (Equation 2), while the output becomes (cid:104) <s a > REL <

Revised: s b > (cid:105)

Original: [Equation]
Revised: [Equation]

Original: where <s a >
Revised: where <s a >

Original: and <s b > are tags corresponding to sentences from the original story, a and b are indices from [0 , 4] and a (cid:54)
Revised: and <s b > are tags corresponding to sentences from the original story, a and b are indices from [0 , 4] and a (cid:54)

Original: =
Revised: =

Original: b .
Revised: b .

Original: The output sequence comes from a limited vocabulary of 5 sentence index tokens, 5 causal dimension tokens 5 , and the sentence index token corresponding to the selected sentence X can be before or after the REL token, depending on what causal dimension is being used.
Revised: The output sequence comes from a limited vocabulary of 5 sentence index tokens, 5 causal dimension tokens, 2 and the sentence index token corresponding to the selected sentence X can be before or after the REL token, depending on what causal dimension is being used.

Original: the The classiﬁcation task is to choose the correct sequence of 100 possible output sequences 6 .
Revised: The classiﬁcation task is to choose the correct sequence of 100 possible output sequences.

Original: The abstracted output avoids the prior conﬂation issue since there are no partial matches within tokens of statements.
Revised: The abstracted output avoids the prior conﬂation issue since there are no partial matches within tokens of statements.

Original: Furthermore, there is no explicit correspondence between input and output.
Revised: Furthermore, there is no explicit correspondence between input and output.

Original: Note that C IS 2 does not distinguish between speciﬁc and general rules.
Revised: Note that C IS 2 does not distinguish between speciﬁc and general rules.

Original: Finetuned C IS 2 models are forced to only learn the commonsense inference task.
Revised: Finetuned C IS 2 models are forced to only learn the commonsense inference task.

Original: The input is kept the same, so the models see the same information as with the original task formulation.
Revised: The input is kept the same, so the models see the same information as with the original task formulation.

Original: Therefore, we argue that C IS 2 is a simpler and fairer measurement of commonsense inference performance.
Revised: Therefore, we argue that C IS 2 is a simpler and fairer measurement of commonsense inference performance.

Original: 5. GLUCOSE Entries to C IS 2 Tag Heuristic 
Revised: 5. GLUCOSE Entries to C IS 2 Tag Heuristic Conversion

Original: Conversion

Original: To evaluate the C IS 2 formulation, we need to convert story sentences into C 
Revised: To evaluate the C IS 2 formulation, we need to convert story sentences into C IS 2 output labels, as in

Original: IS 2 output labels (as 5 >Causes/Enables> , >Causes> , >Enables> , >Results in> , >Motivates> in Equation 4).
Revised: Equation 4.

Original: This method is illustrated in Figure 2.
Revised: See Figure 2 for the conversion process.

Original: Two of  the three C IS 2 tokens are immediately given from the input.
Revised: Each sentence of an input story corresponds to a tag <s 0 > to <s 4 > with indexes corresponding its position in the story. To get the three C ISoutput labels, we do the following: (1) Identify selected sentence X from the input since it always be denoted as the sentence with the asterisks surrounding it.

Original: The  dimension speciﬁes REL , as well as the position of sentence X —whether it comes ﬁrst (corresponding to <s a > ) orlast ( <s b > ).
Revised: The input dimension informs the position of sentence X in the output—whether is <s a > or <s b > ; (2) Get the relation REL from the output directly; and (3) Calculate the similarity of “other” [Equation]

Original: Where X is in the input story speciﬁesthe index of one of the tokens.
Revised: >

Original: Sentence X is easyto ﬁnd since it is a direct match across theinputand output of the original GLUCOSE task.
Revised: [Figure] sentence Y from the output to every other sentence in the input story and select the closest match.

Original: To ﬁnd the remaining token, we look at the speciﬁc rule from the original GLUCOSE task output,which consists of two statements separated by relation REL .
Revised: To ﬁnd the remaining token, we look at the speciﬁc rule from the original GLUCOSE task output, which consists of two statements separated by relation REL .

Original: We will call them P 0 and P 1 .
Revised: We will call them P 0 and P 1 .

Original: Suppose Xcorresponds to P 0 , and we need to ﬁnd which sentence Y corresponds to P 1 .
Revised: Suppose X corresponds to P 0 , and we need to ﬁnd which sentence Y corresponds to P 1 .

Original: We do this by iteratingover the sentences (excluding X), for each calculating its similarity with P 1 .
Revised: We do this by iterating over the sentences (excluding X), for each calculating its similarity with P 1 .

Original: We take the index of thesentence with the highest similarity to P 1 as <s b > .
Revised: We take the index of the sentence with the highest similarity to P 1 as <s b > .

Original: We experimented with several sentence similaritymetrics, and found Sentence-BERT (Reimers and
Revised: We describe our experiments with several sentence similarity metrics in Section 5.2.

Original: Gurevych, 2019) the most effective.

Original: Being a heuristic approach, generated C IS 2 labels are notguaranteed to be perfect.
Revised: Being a heuristic approach, generated C IS 2 labels are not perfect.

Original: However,a cursory manual inspection ﬁnds most labels arereasonable for GLUCOSE entries that have an explicit Y (found within the story).
Revised: However, our manual inspection ﬁnds most labels are reasonable for GLUCOSE entries that have an explicit Y (from the story).

Original: C IS 2 labels arelikely incorrect for  implicit relationships, whichare a minority of the entries.
Revised: C IS 2 labels do not exist for those GLUCOSE entries with implicit relationships 4 , i.e. Y is not in the original story.

Original: Therefore we do notattempt to ﬁlter them out.
Revised: We attempted to ﬁlter these out by removing any training examples that did not pass a threshold 5 of SBERT ≤ 0 .

Original: Future work remainsto distinguish implicit and explicit GLUCOSE relationships, and either ﬁlter implicit ones out, orhandle them separately.
Revised: 16 for any sentence in the story. However, this resulted in a slight drop in the ﬁnal evaluation, so these examples were kept.

Original: We run the conversion heuristic on the GLUCOSE train set and train a T5 model using thesame hyperparameters used for our other modelswith the task of generating the three-token C IS 2label, given the GLUCOSE input.
Revised: We run the conversion method on the GLUCOSE train set and train a T5 model using the same hyperparameters used for our other models with the task of generating the three-token C IS 2 label, given the GLUCOSE input.

Original: We refer to thismodel as C IS 2 -T RAIN .
Revised: We refer to this model as C IS 2 -T5.

Revised: Note that although using C IS 2 tags turns this into a classiﬁcation problem, the model is still doing generation to predict the output.

Original: 5. C IS 2  Results
Revised: 5. C IS 2 Classiﬁcation Task & Results

Original: To compare O RIGINAL and the GLUCOSE diagnostic models to C IS 2 , we run the conversionmethod from Section 5.1 on each model’s speciﬁcrule output to obtain its predicted C IS 2 -like labels.
Revised: In Section 4, we showed that BLEU is not an appropriate metric for the CCI task, given the GLUCOSE models’ extensive copying and paraphrasing. Furthermore, C IS 2 -T5 generates C IS 2 tags instead of full sentences, making it non-trivial to compare to the O RIGINAL GLUCOSE T5 model.

Original: We do thesame conversion  on the GLUCOSE testset to obtain the C IS 2 test set 7 .
Revised: We run the conversion method from Section 5. on each model’s speciﬁc rule output to obtain its predicted C IS 2 labels, and on the GLUCOSE test set to obtain the C IS 2 test set. 6 Both are now formatted as in Equation4.

Original: This enables us todo an exact-match comparison between the modellabels and the test set labels, and removes the associated issues with evaluating generated text. 
Revised: This enables us to do an exact-match comparison between the model labels and the test set labels, and removes the associated issues with evaluating generated text. In effect, the

Original: With all outputs converted to C IS 2 -like labels we calculate the accuracy for how often the correct sentence Y  is actually being chosen.
Revised: C IS 2 evaluation considers requires the correct sentence Y to be chosen ; there is no partial credit for those outputs that can easily be inferred from input: the selected sentence X , and REL . The sentence similarity metric used is crucial in the process of heuristically generating C IS 2 labels. We experimented with both BLEU scores of lemmatized tokens, as well as Sentence-BERT (SBERT) (Reimers and Gurevych, 2019). By using

Revised: BLEU for sentence similarity, GLUCOSE O RIGINAL achieves 66.0%, whereas C IS 2 -T5—despite being trained on these C

Original: Results for this evaluation are shown in Figure 3.
Revised: IS 2 labels converted with BLEU—only achieves 57.2% accuracy. This stems from same issues of BLEU measuring language generation, rather than CCI, as discussed in Section 4. Also, this shows that the C IS 2 classiﬁcation task does not favor our C IS 2 system by default.

Original:  Results  are compared to a simple model that infers the index of X and REL from the input randomly select one of the 4 other story sentences  for the index of Y – achieving a 25% accuracy.
Revised: Therefore, for the ﬁnal evaluation we opt for SBERT, a more context-dependent similarity metric. Results for this evaluation are shown in Figure 3. We compare all of our results to a random baseline which is the probability one of the 4 other story sentences is randomly selected for the index of Y ; this would have an accuracy of 25% (the dashed horizontal line in Figure 3).

Original:  C IS 2T RAIN achieves the highest score at 66.2%, while O RIGINAL is not far behind at 61.9%.
Revised: Out of all the models, C IS 2 -T5 achieves the highest score at 66.2%, while O RIGINAL is not far behind at 61.9%.

Original: As for the diagnostic tasks, we see the same score ordering of models with BLEU evaluation.
Revised: As for the diagnostic tasks, we see the same score ordering of models with BLEU evaluation.

Original: H ISTORY +X scores 8% lower than O RIGINAL .
Revised: H ISTORY +X scores 8% lower than O RIGINAL .

Original: H ISTORY and M ASK X perform even worse than random, indicating that their BLEU performance was almost entirely due to partial token matches.
Revised: H ISTORY and M ASK X perform even worse than random, indicating that their BLEU performance was largely due to partial token matches.

Original: The best GLUCOSE model O RIGINAL achieves 70.7 speciﬁc BLEU, but only 61.9% C IS 2 accuracy.
Revised: The best GLUCOSE model O RIGINAL achieves 70.7 speciﬁc BLEU, but only 61.9% C IS 2 accuracy.

Original: Although we cannot directly compare BLEU  and C IS 2 exact match numbers, we can see that C IS 2 provides a fairer estimate ofthe CCI performance of these ﬁne-tuned T5 models by removing language generation from evaluation.
Revised: Although we cannot directly compare BLEU of generated output, and C IS 2 exact match accuracy, we have shown that C IS 2 provides a fairer estimate of CCI performance of these ﬁne-tuned T5 models by removing language generation from evaluation.

Original: These CCI results are promising, and yet there is still much room for improvement.
Revised: These C IS 2 results are promising, but there is still much room for improvement.

Original:  Discussion
Revised:  Discussion

Original: The diagnostic tasks we discussed in the paper investigated the extent to which the original GLUCOSE task conﬂates language generation and contextual commonsense inference (CCI).
Revised: The diagnostic tasks we discussed in the paper investigated the extent to which the original GLUCOSE task conﬂates language generation and contextual commonsense inference (CCI).

Original: We found that the most signiﬁcant sentence of the input is the selected sentence X , and if omitted, BLEU scores drop signiﬁcantly compared to omitting other story sentences.
Revised: We found that the most signiﬁcant sentence of the input is the selected sentence X , and if omitted, BLEU scores drop signiﬁcantly compared to omitting other story sentences.

Original: This shows that the language model is relying on X , as it should, for CCI.
Revised: This shows that the language model is relying on X for CCI, as it should.

Original: We have shownthat the T5 model trained on the GLUCOSE task(to maximize BLEU on the speciﬁc and generalrules) performs only 4.3% worse on the C IS 2 thanone trained directly on C IS 2 labels.
Revised: It is worth discussing how “fair” it is to remove X —after all, without X , the LMs have little to condition their predictions on.

Original: This shows that 
Revised: While this is true, we emphasize that our diagnostic tasks are intended to be taken together to analyze the extent of conﬂation.

Original: T5 can still learn signiﬁcant CCI from the GLUCOSE data, and can further improve performancewith C IS 2 converted labels, abstracting away withlanguage generation.
Revised: The main takeaway is that by including X , trained models will rely on copying instead of good commonsense inference.

Original: We have also shown evidence for extensive copying and paraphrasing as seen from the higher performance on speciﬁc rules relative to general rules for
Revised: We have also shown evidence for extensive copying and paraphrasing as seen from the higher performance on speciﬁc rules relative to general rules for

Original: O RIGINAL and H ISTORY +X. These trends hold for
Revised: O RIGINAL and H ISTORY +X. These trends hold for

Original: C IS 2 evaluation as well, but are even more marked(since there is no inﬂation from matching tokens).
Revised: C IS 2 evaluation as well, but are even more marked since there is no inﬂation from matching tokens.

Original: It is worth discussing how “fair” it is to remove
Revised: Lastly, we have shown that the T5 model trained on the GLUCOSE task (to maximize BLEU on the speciﬁc and general rules) performs only 4.3% worse on the C IS 2 than one trained directly on C ISlabels. This shows that T5 can still learn signiﬁcant

Original: X – after all, without X , LMs have little to conditiontheir predictions on.
Revised: CCI from the GLUCOSE data, and can further improve performance with C IS 2 converted labels, abstracting away with language generation. 6. Future Work

Original: While this is true, we emphasize that our diagnostic tasks are intended to betaken together to analyze the extent of conﬂation. The main takeaway is that by including X , trainedmodels will rely on good language generation instead of good commonsense inference.
Revised: We plan to collect ground-truth C IS 2 labels via crowdsourcing for the entire test set, and for some training examples. To simplify the task, we will have workers verify, and correct if necessary, the heuristic C IS 2 labels.

Original: Future work can further explore utilizing GLUCOSE and related datasets for story generationtasks.
Revised: Future work can further explore utilizing GLUCOSE and related datasets for story generation tasks.

Original: We believe our diagnostic tasks can be easily be extended toallow using GLUCOSE in various story generation settings.
Revised: One promising avenue to extending our CCI evaluation to story generation settings is incorporating our approach with the COINS framework (Paul and Frank, 2021), which generates contextualized inference rules to guide future output sentences. Abstracting these inference rules through

Original: H ISTORY and H ISTORY +X models for continuing stories, and M ASK X models for story inﬁlling (Ippolito et al., 2019).
Revised: C IS 2 would likely allow the language model to better capture and learn CCI.

Original: We especially resonate with the questionanswering based approach to commonsense inference for stories (also on ROCStories) of Lal et al.(2021).
Revised: We also resonate with question-answering based approaches to commonsense inference for stories (Lal et al., 2021; Castricato et al., 2022). Lal et al.

Original: They trained large language models ontheir dataset, ﬁnding that they only perform wellwhen the answers are present in the narrative.
Revised: (2021) trained large language models on their dataset, ﬁnding that they only perform well when the answers are present in the narrative.

Original: Thisﬁnding goes hand in hand with our ﬁnding thatthe original GLUCOSE task formulation allows foreasy paraphrasing and thus inﬂated performance.
Revised: This ﬁnding goes hand in hand with our ﬁnding that the original GLUCOSE task formulation allows for easy paraphrasing and thus inﬂated performance.

Original:  Conclusion
Revised:  Conclusion

Original: This work investigated the extent to which languagemodels learn contextual commonsense inference(CCI), utilizing the GLUCOSE (Mostafazadehet al., 2020) dataset and the T5 (Raffel et al., 2020)language model as case studies.
Revised: This work investigated the extent to which language models learn contextual commonsense inference (CCI), utilizing the GLUCOSE (Mostafazadeh et al., 2020) dataset and the T5 (Raffel et al., 2020) language model as case studies.

Original: We showed howthe original GLUCOSE task conﬂates languagegeneration and CCI tasks, causing over-estimationof true CCI performance.
Revised: We showed how the original GLUCOSE task conﬂates language generation and CCI tasks, causing over-estimation of true CCI performance.

Original: We then formulated diagnostic tasks by permuting the original task andfound that LMs rely on paraphrasing the selectedsentence and context in making their predictions.
Revised: We then formulated diagnostic tasks by permuting the original task and found that LMs rely on paraphrasing the selected sentence and context in making their predictions.

Original: We proposed C IS 2 as an alternative task to structure and evaluate language models for CCI.
Revised: We proposed C IS 2 as an alternative task to structure and evaluate language models for CCI.

Original: C IS evaluation is a simpliﬁed, fairer measurement of
Revised: C ISevaluation is a simpliﬁed, fairer measurement of

Original: CCI performance over BLEU.
Revised: CCI performance than BLEU.

Original: By ﬁnetuning a T model on our C IS 2 task, it correctly selects the causal statement 4.3% more than a model trained on the original GLUCOSE task.
Revised: By ﬁnetuning a T model on our C IS 2 task, it correctly selects the causal statement 4.3% more than a model trained on the original GLUCOSE task.

Original: We note this is using heuristically converted C IS 2 labels, and collecting ground-truth C IS 2 labels for training would lead to even better performance.
Revised: We note this is using heuristically converted C IS 2 labels, and collecting ground-truth C IS 2 labels for training would lead to even better performance.

Original: Overall, we found that GLUCOSE indeed encodes contextual commonsense information, and
Revised: Overall, we found that GLUCOSE indeed encodes contextual commonsense information, and

Original: T5 has capacity to learn this.
Revised: T5 has capacity to learn this.

Original: Therefore, the challenge for future researchers is to leverage GLUCOSE and other contextual commonsense inference datasets’ knowledge representations appropriately and avoid conﬂation of language generation.
Revised: Therefore, the challenge for future researchers is to leverage GLUCOSE and other contextual commonsense inference datasets’ knowledge representations appropriately and avoid conﬂation of language generation.

Original: A AppendixA. Acknowledgements
Revised: A Appendix A. Acknowledgements

Original: We thank the authors of GLUCOSE, in particular
Revised: We thank the authors of GLUCOSE, in particular

Original: Or Biran and Lori Moon, for their helpful assistance in working with the GLUCOSE dataset andcodebase.
Revised: Or Biran and Lori Moon, for their helpful assistance in working with the GLUCOSE dataset and codebase.

Revised: We also thank Daphne Ippolito and the anonymous reviewers for their comments and suggestions.

Revised: This material is based upon work supported by the National Science Foundation under Grant #2030859 to the Computing Research Association for the CIFellows Project.

Original: A. Ethical Considerations and BroaderImpacts
Revised: A. Ethical Considerations and Broader Impacts

Original: The methods used in our paper build in large partupon work by prior researchers.
Revised: The methods used in our paper build in large part upon work by prior researchers.

Original: The T5 (Raffelet al., 2020) language model we used was pretrained on a massive dataset for many days.
Revised: The T5 (Raffel et al., 2020) language model we used was pretrained on a massive dataset for many days.

Original: Despitethe energy usage, T5 has proved be a valuable toolthat can be used for countless downstream NLPapplications, ours included.
Revised: Despite the energy usage, T5 has proved be a valuable tool that can be used for countless downstream NLP applications, ours included.

Original: As for our own trainedmodels, we note that we further ﬁne-tuned T5 onan array of diagnostic and custom tasks.
Revised: As for our own trained models, we note that we further ﬁne-tuned T5 on an array of diagnostic and custom tasks.

Original: During development, we made sure to pilot any experimentson smaller datasets, and we carefully managed our
Revised: During development, we made sure to pilot any experiments on smaller datasets, and we carefully managed our

Original: GPU and CPU usage throughout.
Revised: GPU and CPU usage throughout.

Original: As for the data used, the ROCStories (Mostafazadeh et al., 2016) and GLUCOSE (Mostafazadeh et al., 2020) datasetsinvolved a great deal of careful task design andwork with crowd-source workers.
Revised: As for the data used, the ROCStories (Mostafazadeh et al., 2016) and GLUCOSE (Mostafazadeh et al., 2020) datasets, in which our work builds on, involved a great deal of careful task design and interaction with crowd-source workers.

Original: We thank theseresearchers for their ethical treatment of theircrowd-source workers, with fair pay and two-waycommunication (Moon et al., 2020).
Revised: We thank these researchers for their ethical treatment of their crowdsource workers, with fair pay and two-way communication (Moon et al., 2020).

Original: We will publicly release all our code, from datapreprocessing, to model training, to ﬁnal evaluation, to ensure that our work is fully reproducible.
Revised: We will publicly release all our code, from data preprocessing, to model training, to ﬁnal evaluation, to ensure that our work is fully reproducible.

Original: The broader impacts of our work outside its immediate subject are several.
Revised: The broader impacts of our work outside its immediate subject are several.

Original: First, our work takes astep towards analyzing stories, which are something fundamentally human, and that machineshave yet to master.
Revised: First, our work takes a step towards analyzing stories, which are something fundamentally human, and that machines have yet to master.

Original: Second, we have encouraged
Revised: Second, we have encouraged

Original: NLP researchers in general to think more carefullyabout the structure of a task, before defaulting tothe latest state-of-the-art language model.
Revised: NLP researchers in general to think more carefully about the structure of a task, before defaulting to the latest state-of-the-art language model.

Original: For example, we found that our C IS 2 task, which is simpler and thus requires less training resources thanthe language generation task, performs better oncapturing contextual commonsense inference.
Revised: For example, we found that our C IS 2 task, which is simpler and thus requires less training resources than the language generation task, performs better on capturing contextual commonsense inference.

Original: A. Reproducing Our Work
Revised: A. Reproducing Our Work

Original: We make our code publicly available at a Githublink.
Revised: We make our code publicly available at a Github link.

Original: The codebase includes complete preprocessing, training, and evaluation scripts, to take the raw
Revised: The codebase includes complete preprocessing, training, and evaluation scripts, to take the raw

Original: GLUCOSE CSVs and T5 checkpoints, and trainboth diagnostic and C IS 2 models.
Revised: GLUCOSE CSVs and T5 checkpoints, and train both diagnostic and C IS 2 models.

Original: We will also release the ﬁnal trained checkpoints.
Revised: We will also release the ﬁnal trained checkpoints.

Original: We also include our code to reproduce the original GLUCOSE experiments.
Revised: We also include our code to reproduce the original GLUCOSE experiments.

Original: We model this closely to the original GLUCOSE paper, starting from their provided code repository.
Revised: We model this closely to the original GLUCOSE paper, starting from their provided code repository.

Original: A. Reproduction Results
Revised: A. Reproduction Results

Original: We report the results we obtained on the original GLUCOSE task in Table 5.
Revised: We report the results we obtained on the original GLUCOSE task in Table 5.

Original: We report perdimension BLEU, as was done prior, as well as the weighted average BLEU across all dimensions.
Revised: We report perdimension BLEU, as was done prior, as well as the weighted average BLEU across all dimensions.

Original: We ﬁnd that the reported numbers from (Mostafazadeh et al., 2020) and their provided Tensorﬂow checkpoint are essentially consistent.
Revised: We ﬁnd that the reported numbers from (Mostafazadeh et al., 2020) and their provided Tensorﬂow checkpoint are essentially consistent.

Original: Our replication results (done with the transformers package (Wolf et al., 2019)) achieve 4-5 BLEU points lower, due to resource limitations and slight differences in experimental setup (i.e. we had far less GPU resources and and training time).
Revised: Our replication results (done with the transformers package (Wolf et al., 2019)) achieve 4-5 BLEU points lower, due to resource limitations and slight differences in experimental setup (i.e. we had far less GPU resources and and training time).

Original: For consistency’s sake all of our experiments use the same setup as replicated t5-large (termed Original in the main text), and thus use this as the baseline.
Revised: For consistency’s sake all of our experiments use the same setup as replicated t5-large (termed Original in the main text), and thus use this as the baseline.

Original: We report results on the test set, but choose to evaluate BLEU on only the ﬁrst of the three provided references for each test set entry.
Revised: We report results on the test set, but choose to evaluate BLEU on only the ﬁrst of the three provided references for each test set entry.

Original: This is because the GLUCOSE train set only has one reference per entry, not 3, and we carved a small development set out of the train set, since no train/development split was provided.
Revised: This is because the GLUCOSE train set only has one reference per entry, not 3, and we carved a small development set out of the train set, since no train/development split was provided.

Original: We evaluate our custom development and the original test set the same way, with 1 reference per entry.
Revised: We evaluate our custom development and the original test set the same way, with 1 reference per entry.

Original: A. Training Setup and Hyperparameters
Revised: A. Training Setup and Hyperparameters

Original: We trained our models on 2 NVIDIA Quadro RTX 6000 GPUs, with 24 GB vRAM each.
Revised: We trained our models on 2 NVIDIA Quadro RTX 6000 GPUs, with 24 GB vRAM each.

Original: We train up to 10 epochs, early stopping after 10 checkpoints without improvement on the validation set.
Revised: We train up to 10 epochs, early stopping after 10 checkpoints without improvement on the validation set.

Original: Depending on the task, the models ﬁnish training between 6 to 34 hours.
Revised: Depending on the task, the models ﬁnish training between 6 to 34 hours.

Original: The GLUCOSE authors trained their model far more – for 72 hours on
Revised: The GLUCOSE authors trained their model far more – for 72 hours on

Original: TPUs – which can explain our lower BLEU scores.
Revised: TPUs – which can explain our lower BLEU scores.

Original: We use the exact same hyperparameters as in Raffel et al.
Revised: We use the exact same hyperparameters as in Raffel et al.

Original: (2020), following Mostafazadeh et al.
Revised: (2020), following Mostafazadeh et al.

Original: (2020), with one major exception: we use  a learning rate of 1e-4 instead of 1e-3, which we found to converge too quickly.
Revised: (2020), with one major exception: we use [Table] [Table] a learning rate of 1e-4 instead of 1e-3, which we found to converge too quickly.

Revised: A. Speciﬁc-Only Results

Original: [Table]
Revised: Given that C IS 2 only considers the speciﬁc rule, one may ask how the GLUCOSE models trained to generate only speciﬁc rules would perform. We therefore train 4 “speciﬁc-only” models, one for each of the 4 diagnostic tasks of Section 4. We denote speciﬁc-only models with the sufﬁx -S PEC and we compare the results to the speciﬁc+general models (as in the main text) without a sufﬁx. Table 6 compares the BLEU results, whereas Figure 4 compares the C IS 2 results.

Revised: We see that the speciﬁc+general models and the speciﬁc-only models perform similarly.

Revised: This conﬁrms the ﬁndings of Mostafazadeh et al.

Revised: (2020), where T5 can effectively learn both speciﬁc and general rules jointly.

Revised: As both BLEU scores and C IS 2 classiﬁcation accuracy are similar, we report the speciﬁc+general model results in the main paper to be consistent with prior work.