From: AKBC.ws/2021/Conference/Program_Chairs
title: Paper Decision
decision: Accept
comment: This paper analyzes the emergence of commonsense knowledge in language models pre-trained on knowledge graphs. Given the popularity of this line of research, this paper is very timely, and reviewers found it well-executed on the whole. The study has some limitations in terms of the models and KGs considered, but I think it is worth accepting to AKBC.

From: AKBC.ws/2021/Conference/Paper58/Authors
title: General comment on changes made to revision
comment: 
Summary of changes made to revised copy:

- We added a table of relation templates for each of the relations mentioned in the paper in the Appendix.

- We added a table to the appendix regarding the impact of prompt formulation (by paraphrasing each of the prompts in ATOMIC2020).

- Regarding the above, we added a table of relation templates for each of the paraphrased prompts.

- We added an example of the input to the language model for augmentation experiments mentioned in the paper.

- We clarified the optimizer used by T5 in the experimental section of the paper.

- We added a forward reference to experiments mentioned later in the paper at the end of section 4, to improve fluidity.

- We added clarification in 5.3 on the importance of analyzing angular change in addition to L1 change.

- We expanded our related work section on knowledge models and prompting.

- We added context on our prompt-based experiments in the introduction section, completing the introduction with a brief summary of all experiments run in the paper.

- We fixed minor typos and formatting issues per reviewer comments.

From: AKBC.ws/2021/Conference/Paper58/Authors
title: Response to Reviewer 1
comment: We thank the reviewer for their helpful comments and overall positive rating, in part commenting on our extensive analysis, useful findings on prompt performance, and the structure and clarity of the paper.

**The analysis in the paper focuses on ATOMIC2020, a large-scale commonsense knowledge graph. Should other commonsense knowledge graphs be considered also (ConceptNet)?**

We appreciate the reviewer’s question about running our emergence analysis on other commonsense knowledge graphs beyond ATOMIC2020 (e.g., ConceptNet). We note that ATOMIC2020 is one of the most comprehensive commonsense knowledge resources available, subsuming examples from many other known KGs (Hwang et al., 2021). The ATOMIC KG (Sap et al., 2019), for example, is completely contained in ATOMIC2020. Meanwhile, ATOMIC2020 also contains a high-quality subset of 172k tuples from ConceptNet that covers 17 of the commonsense relations present in ConceptNet. According to Hwang et al., 2021, these 17 relations are consolidated into overlapping relation types (for example, the ConceptNet CausesDesire relation is merged into the ATOMIC2020 relation of xWant). Also, the parts of ConceptNet excluded by this merge are geographical facts, dictionary-based knowledge, lexical relationships, and relations based on lexical co-occurrence (Hwang et al., 2021), which are not the focus of this work. Analyzing the emergence of these types of facts in language models would be an interesting task on its own, though.

From: AKBC.ws/2021/Conference/Paper58/Authors
title: Response to Reviewer 2
comment: We thank the reviewer for their overall positive rating and helpful comments, and their comments that our paper is well-motivated, of interest to the community, and sharing well-explained and convincing results.

**Out of the three settings (zero-shot, few-shot, fully-supervised), T5 is evaluated across few-shot and fully-supervised settings, and GPT-3 is evaluated in zero-shot and few-shot settings. Do you include an evaluation across all three settings?**

The reviewer’s intuition is correct that the two additional settings (fully supervised GPT-3, zero-shot T5) were considered but excluded for technical reasons. T5 models are not pretrained using a causal language modeling objective, so we felt reporting zero-shot evaluation performance in the main table for a multi-word generation task would be an unfair (and perhaps misleading) comparison. However, we did report limited numbers for zero-shot T5 tuple completions in the Appendix. For reference, T5-11B gets a BLEU-1 score of 6.7 using the masking objective, which is lower than GPT-2 and GPT-3 with their causal language modeling objectives. GPT-3 is difficult to evaluate in the fully-supervised setting, since the only option at the time of submission was to include example tuples in the prompt via augmentation.

**The experiments mention relations formatted into natural language into prompts. How do you automatically format the prompts for each of the relations in ATOMIC2020?**

We manually wrote prompts for each of the 23 relations in ATOMIC2020 (see examples in Table 2). Following (Hwang et al., 2021), we did not tune prompts on validation data (which Perez et al., 2021 show artificially inflates performance). With respect to sensitivity to minor changes in the prompt, we mention Jiang et al., 2020’s conclusion in our motivation in Section 5.4. Driven by this sensitivity, we repeated few-shot experiments with a paraphrase of the prompt, but did not find significant differences in generated tuple quality between different prompt variations (Our paraphrased prompts for n=30 had quality scores of BLEU_1 = 30.5. For reference, the original prompts had quality scores of BLEU_1 = 31.9.) We concluded that while generating directly from prompts may be quite sensitive to the construction of the prompt, minimal fine-tuning on examples directly using the prompt removes some of this sensitivity. We will add the full results of our paraphrased prompts to the paper. 

**What are the advantages of measuring angular parameter change on top of L1 parameter change?**

We thank the reviewer for asking for clarification regarding the relationship between L1 and angular distance. While these two measures can often be correlated, interesting information can be gleaned in cases where they are not (i.e., relative differences in L1 change across parameter sets do not match differences in angular change among the same parameter sets). For example, high L1 change for a particular parameter matrix with no corresponding increase in angular change relative to other matrices could be an indication that most of the learning reinforces previously learned knowledge by the parameters during pretraining (i.e., changing the scale of the output vectors of these transformations, rather than their directions). Meanwhile, a high relative angular change combined with low relative L1 change could be an indicator that changes in parameters in this matrix were concentrated on a few elements of the parameter matrix. Consequently, complementary information can be gleaned from analyzing both measures together.

**Angular parameter change is calculated by comparing and averaging the angular change between rows in the parameter matrix. Why compare rows, instead of columns?**

The reviewer asks about the motivation for using rows in angular parameter change and their intuition about rows was our same reasoning. Each row in the parameter matrix performs a dot product with the input representation, and has a correspondence to an individual element in the output vector.

From: AKBC.ws/2021/Conference/Paper58/Authors
title: Response to Reviewer 3
comment: We thank the reviewer for their helpful comments and positive rating, in part stating that the work is timely, presented well, and answers interesting questions regarding the emergence of commonsense knowledge.

**The majority of the interpretability analysis was done on T5. Why is T5 (an encoder-decoder model) the best model for analysis?**

The reviewer asks whether this analysis could have been suitable for other model types besides T5. We picked T5 as the model for our detailed analysis for the following reasons. First, T5 has the highest fully supervised and few-shot accuracy among knowledge models investigated in the fully-supervised setting (Table 3). This observation tracks with other commonsense benchmarks where T5 models typically achieve the highest performance among pretrained LMs. Second, encoder-only models (such as RoBERTa) are difficult to convert as text generators -- a requirement for this task. The masking scheme forces the model to either have the number of tokens of the output specified with a given number of [MASK] tokens, or be finetuned extensively to learn text-generation abilities, precluding the possibility of investigating few-shot behavior. Decoder-only models (e.g., GPT2) could more easily be investigated, but Hwang et al., 2021 showed they perform worse on the task than encoder-decoder models. Finally, there are available T5 models at multiple sizes (small, large, 11B), allowing us to investigate model scale as a principal focus of our analysis (which would not be possible with other encoder-decoder models such as BART). For these reasons, we chose the T5 class of models for our analysis, though investigating these patterns for other classes of models would be an interesting study for future work. 

**Could additional motivation be added to clarify the importance of analysis on angular change?**

We thank the reviewer for asking for clarification regarding the importance of angular distance. Observing relative differences in L1 change across parameter sets vs. the relative differences in angular change among the same parameter sets can be informative as to how the parameters are changing in these matrices. High L1 change for a particular parameter matrix with no corresponding increase in angular change relative to other parameters could be an indication that most of the learning reinforces previously learned knowledge by the parameters during pretraining (i.e., changing the scale of the output vectors of these transformations, rather than their directions). Meanwhile, a high relative angular change combined with low relative L1 change could be an indicator that changes in parameters in this matrix were concentrated on a few parameters. For example, in Figure 5, the output matrix (wo) in the decoder of the second layer of T5-11B has low L1 distance, but high angular distance, suggesting that there are large specialized rotational changes in certain rows of the parameter matrix.

**The analysis is only on commonsense knowledge graphs. What would change if we analyzed other kinds of non-commonsense knowledge (for example, declarative knowledge)?**

We agree that comparing parameter changes between commonsense knowledge models and models trained to capture other kinds of knowledge is an interesting question. We predict that parameter changes might be more concentrated for declarative factual knowledge as commonsense knowledge is more distributional, potentially requiring broader updates to reflect learned relationships among related inferences. Meanwhile, declarative knowledge may have many functional relations where only a single tail entity can be correct for the corresponding head entity (e.g., DateOfBirth). Consequently, factual knowledge emergence may require learning to encode specific facts, rather than learning to encode more general rules, which may be reflected with more concentrated parameter updates. Confirming these hypotheses would be an interesting direction for future work.

From: AKBC.ws/2021/Conference/Paper58/Reviewer_MBrX
title: Good paper - simple, well executed and well presented
review: This paper analyzes the emergence of commonsense knowledge in few-shot knowledge models. In particular, it attempts to answer whether and when is this knowledge learned: during pretraining or from fine-tuning on KG examples. To investigate this, various commonsense knowledge models are trained in few-shot settings. The results and analysis in the paper shows that commonsense knowledge models can adapt from limited examples. Moreover, similar to a recent line of work, this work confirms that KG fine-tuning learns an interface to the commonsense knowledge learned during pretraining. Finally, the paper also presents an analysis of absolute, angular, and distributional parameter changes during few-shot fine-tuning. However, the insights here are less convincing to me.

Pros:
1. The paper is timely. It is presented well and answers some interesting question on the emergence of commonsense knowledge in these models in a very simple and intuitive way.
2. The analysis (while not always convincing) mostly makes sense and all the conclusions seem reasonable.

Cons:
1. The related work discussion seems too sparse to me.
2. It would help if all the conclusions from the experiments in the paper were presented in one place e.g. in the intro.

Questions and suggestions:
1. Why do we need to consider an encoder decoder model for this analysis? Why cant we directly analyse an encoder only or decoder only model?
2. It might help to provide more motivation and intuition for the various parameter change metrics e.g. angular parameter change in the paper. This would help the readers understand the conclusions much more easily. This part is less clear in my opinion.
3. This paper analyses commonsense knowledge. What would change in this paper if we were analysing declarative knowledge such as in other KGs? Is emergence of commonsense knowledge any different from other kinds of knoweldge?
rating: 7: Good paper, accept
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: AKBC.ws/2021/Conference/Paper58/Reviewer_gh1k
title: Well-motivated and insightful analysis that interpolates between zero-shot and fully-supervised
review: This paper analyzes fine-tuning of pre-trained generative language models on a commonsense knowledge task, investigating how much the strong performance of the models is due to pre-training or to task-specific learning (via fine-tuning or prompting/in-context learning). The task is to predict natural text for the tail of (head, relation, tail) relationships from the Atomic_20^20 knowledge graph (KG), e.g. ("PersonX goes to the mall", xIntent) -> "to buy clothes". The paper shows that a T5-based model obtains a substantial fraction of the performance of fine-tuning on the full Atomic dataset, using few-shot learning with only a small number of examples (3) per relation. Few-shot learning also obtains a substantial improvement over zero-shot performance for this model. The paper also presents several measures of parameter differences between pre-trained and fine-tuned models and performs an analysis with them, showing that more parameters are updated in the models' decoders than in their encoders. Taken together with past work, the evaluation results and analysis support the paper's claim that most of the knowledge useful for the KG task was already present in the pre-trained models.

*Pros*:
- I found the paper well-motivated -- it's a natural and sound way to investigate how much performance on the KG tasks is due to pre-training versus task-specific fine-tuning, and an indirect way of investigating how much knowledge (of the type useful for this KG task) is encoded in the pre-trained language models. I expect this will be of interest to the community.
- The experimental results were overall very thorough (although see below for one possible gap), performing human evaluations, multiple trials, and evaluating a range of models.
- The parameter analysis was particularly strong. This method is simple (which I think is a good thing!) and well-motivated, and the results were well-explained, convincing to me, and tied in well to past work. 

*Cons*:
- The only model that is compared across all three conditions (zero-shot, few-shot, and fully-supervised) in the experimental results is GPT-2 XL, but for this model few-shot learning recaptures barely any of the performance of the fully-supervised condition (36.6% acceptability in zero-shot, at most 40.1% acceptability in few-shot, and 72.5% acceptability in fully-supervised). However, I suspect that this may be due to limitations of the T5 and GPT-3 models. Please see questions below.
- A few minor things about the presentation could be improved, see suggestions below. But this isn't at all a crucial weakness and should be easily fixable.

*Questions*
- Is it possible to evaluate either GPT-3 or T5 in all of the zero-shot, few-shot, and fully-supervised settings, as GPT-2 XL doesn't do well in the few-shot condition? I suspect GPT-3 may not be fine-tunable (for the fully-supervised condition), and T5 may not work well in the zero-shot since it's an encoder-decoder model, but it would help to clarify this.
- How is the "automatic formatting" of relations into prompts done (Section 3) -- was the prompt text for a given relation obtained from some source, or written by hand? Were prompts tuned on validation data in either the few-shot or zero-shot setting? This is relevant given Jiang et al. 2020 showing that performance is sensitive to subtle variations.
- Could more details be given on the setup in the "augmentation" experiments? Do these include the n=3 training examples as part of the prompt, but without fine-tuning the model? Was the formatting of the prompt the same across both GPT-2 XL and GPT-3, and does the formatting have any hyperparameters / design decisions that needed tuning?


*Suggestions*:
- In the experimental setup it would help to clarify the optimizer used.
- I was surprised when section 4 ended without varying n over a larger budget of training examples (which comes later in Section 5), given the focus on this in the intro. I'd suggest at least adding a forward reference to the text in Section 4.
- The paper used both "learning" in Table 3 and "fine-tuning" elsewhere in the text to mean the same thing (I think); it would help to use the same term for both.
- Could more motivation be given for using rows in angular parameter change (rather than e.g. columns, or flattening the matrices and treating them as a single vector)? Rows might be motivated by e.g. each row's correspondence to an individual element in the output vectors, but I don't know if this is true because I don't know whether inputs are row or column vectors. 
- I'm also not totally clear on the advantage of angular parameter change over l1 distance -- I found it a bit difficult to think about how l1 change could be low but angular change high (in section 5.3).
- 5.1: "large number of the learning"
- Super minor, but the paragraphs before equations (1-3) seem to have compressed line spacing. If these equations are wrapped in \small, this might be fixed by inserting \par\nobreak or \par\nobreak\vspace{-1em} between the paragraphs and the equations. (I once spent a while figuring out how to fix this.)

rating: 7: Good paper, accept
confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: AKBC.ws/2021/Conference/Paper58/Reviewer_YU6U
title: Interesting Commonsense Analysis Using Few-Shot Knowledge Models
review: Summary:

The paper demonstrates that commonsense knowledge models can learn declarative commonsense relationships very quickly in a few shot setting.
This paper aims to address the question of whether commonsense knowledge is directly encoded in pretrained LMs or whether the encoding is learned during finetuning with a favorably initialized pretrained model. 
By varying the number of training examples in a few shot setting, the authors show that models seem to learn an interface to knowledge they already contain with only a few commonsense  examples.
Furthermore, the paper performs analysis wrt parameter change to support that the model seems to be learning an interface to knowledge it already knows, showing that 1) few shot training largely updates parameters in the decoder attention heads and not the large FF networks (that have been showed to embody memory storage during pretraining) and 2) that larger models exhibit parameter change across a concentrated set of parameters, showing that their larger size allows for less entangled knowledge retention. 


Strengths:

-Good evaluation metrics, extensive analysis, and interpretable methods to support claims in the paper

-Very interesting use of LMs as few shot learners to investigate commonsense knowledge abilities 

-Useful findings on prompt performance in few shot commonsense interface learning 

-Paper is very well written and structured


Weaknesses:

-The paper only considers ATOMIC as a KG, it could be useful to experiment w/ more KGs. Extending this to ConceptNet should not be very difficult given the overlap of relations in ATOMIC and ConceptNet. Would be helpful and interesting to see if the findings generalize to other types of commonsense knowledge or whether the findings are specific to the type of knowledge found in ATOMIC. 

rating: 7: Good paper, accept
confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature