Paper: Meta- (out-of-context) learning in neural networks (ICLR-2024)

Reviewer comment (Evaluation):
You acknowledge that a limitation of your experiments is that you do not formally define 'internalization'. I agree this is problematic. I suspect the fact that you observe "meta-OCL" depends entirely on how you define internalization. Concretely, I believe the 'factual' phrasing of 'What _is_ the name of {xyz}?' is crucial here. If the model has learned about {xyz} in the context of a 'definition=inconsistent' tag, then it will not believe that {xyz} actually belongs to that entity given that previously it has observed that QA-pairs for this variable do not match the entity. Therefore, it is less likely to respond to that question 'correctly' for variables introduced with definition=inconsistent tags in D_6. This seems like a very plausible explanation of the phenomenon that you call 'meta-OCL' to me. Given these arguments, I am not convinced it is appropriate to call this phenomenon meta-learning. (Similar arguments apply to the other internalization phrases you explore.)

Author response:
We are not sure we understand your explanation of meta-OCL here; it sounds similar to our last paragraph of Section 4, “the model learns the semantics of the define tags correctly”. Is this right? We would appreciate it if you could clarify this, as well as your sentence starting with “Similar arguments” (if that’s a different point).

Restating why we believe “meta-learning” is a valid characterization of our results:
- The first finetuning stage modifies the way the model will update on new definitions in the future, such that these future updates are more advantageous for generalization. Thus it is meta-learning.
- Or in other words, in the second finetuning stage, the model updates on the two types of definitions differently and in a way that’s helpful for generalization. Thus it has learned how to learn.

Reviewer comment (Evaluation):
I can think of a definition of 'internalization' for which I do not expect to see meta-OCL. Instead of measuring entity association with 'What is the name of {xyz}?', I would be curious to see what happens if you just ask the model to complete '{xyz}', i.e. just the variable name, which, in the training set definitions, is always followed with the entity. I would expect this to be completed with the correct entity just as often for consistent and inconsistent definition tags (D_5 and D_6). For this reasonable definition of internalization, I would not expect you to observe meta-OCL.

Author response:
We define internalization as "treating a statement as true". This means the model should behave as if it "believes" the statement "xyz is a pseudonym for Cleopatra" and respond accordingly. This operationalization isn't absolute or binary, but our experiments do show *more* internalization of consistent definitions in this sense.

We don't consider “predict the second word of the bigram (<variable> <entity>)” a reasonable measure of internalization as we mean to study it.  It could serve as an operationalization, but we believe our operationalization better captures the kind of generalization we’re interested in, and in any case, *any* systematic difference in how the model treats $\mathtt{D}_5$/$\mathtt{D}_6$ examples that indicates it has internalized $\mathtt{D}_5$ more would be enough. This is still the case even if your prediction about bigram completion is correct, which it might be since these bigrams are contained in definitions, and definitions are memorized the same regardless of the define tag (see Figure 8). We can run these experiments, but only a result in the opposite direction (internalizing $\mathtt{D}_6$ more) would challenge our findings.

Reviewer comment (Evaluation):
The performance on QA_3 is not far off from the performance of D_2QA_2, i.e. not providing any definitions is only a little worse than providing consistent definitions. QA_3 is much better than QA_7 (testing on unseen variables). Therefore it seems the effect of 'implicitly learning about variables from QA-pairs' is much larger than the OCL effect. I am surprised by this and would like the authors to discuss this.

Author response:
This is true, and we do draw attention to it in the last paragraph of section 2.3 (*>It is notable that $\mathtt{EM(\hat{QA}_4)}$ is not that far off from $\mathtt{EM(QA_3)}$, so less performance is lost due to replacing entities with variable names than one could expect*). The reason for this strong effect is high mutual information between QA pairs about the same variable: e.g. if a QA pair about someone’s birth century is in the training data, it is easy to answer the test question about their death century. We discuss this in the last two paragraphs of Appendix A1, and will mention it in the main text. 

The reason we don’t focus on this is because the difference in the effect sizes of learning from QA pairs VS learning from definitions does not matter for our main point, which is about the difference between the effects of the two types of definitions.

(It's also worth noting that the model doesn't fully learn the meaning of the variables from the questions, as evidenced by the $\mathtt{QA_3}$ line staying at zero on the entity association test set in Figure 2b.)

Reviewer comment (Experiments):
The authors' motivation in the introduction (LLMs learning to trust Wikipedia more than 4chan) is not discussed again in the paper. I am not convinced that the current QA-setup (which replaces entities in questions, not the 'source' of the answer) is the best way to test this motivation. This creates a disconnect between the motivation and the experiment setup.

Author response:
The define tags are meant to correspond to the reliable/unreliable sources in the wikipedia/4chan motivating example. We’ll make this clearer in the paper by coming back to the motivating example throughout, thanks!

Reviewer comment (Theory):
The authors later claim their method provides a hypothetical mechanism for 'situational awareness'. They do not explain how meta-(OCL) and situational awareness relate, or more precisely, how the 'fact learning mechanism' and situational awareness relate.

Author response:
Briefly, we believe situational awareness could arise because the model is trained on content that is from trusted sources and includes facts about the model (e.g. about its training process). We’ll clarify this in the text.

Reviewer comment (Theory):
In the paragraph on functional decision theory, the authors claim that better understanding (meta-)OCL 'can either rule out such scenarios [...] or take measures to prevent them' but do not explain how.

Author response:
Yes, we agree.  This is a somewhat involved argument that may require familiarity with the related works that we discuss.  We don't see a good way around this, but since other reviewers also didn't find this content clear and valuable as it stands, we could expand this discussion and move it to the appendix.

Reviewer comment (Theory):
I am not convinced the gradient alignment hypothesis holds up to scrutiny. How do you go from 'gradients in a minibatch are aligned' to 'gradients between define=consistent statements and corresponding QA pairs are aligned'? If gradients in a minibatch are aligned, that minibatch also contains 'define=inconsistent' statements. Why are these not aligned? The experiments with increasing batch size are interesting, and I would welcome further investigation and discussion.

Author response:
We assume that those gradients are more “alignable” (and hence become more aligned), because the examples share a common explanatory factor (e.g. that xyz is a pseudonym for Cleopatra); ‘define=inconsistent’ statements will not have such a common explanatory factor with any QA pairs, by construction.  This assumption draws on connections with meta-learning and invariant learning described in Related Work.  For instance, in “Learning explanations that are hard to vary” (Parascandolo et al., ICLR 2021) the authors argue that ‘parameter values for which different examples have a similar local loss surface (i.e. similar gradients)’ correspond to a model providing a similar explanation for those different examples.

Reviewer comment (Writing):
I like the 'selective retrieval hypothesis' more, although I feel it lacks clarity. For example, the statement "Since the model learned to rely on Define=consistent definitions more for answering questions, it better answers questions about new Define definitions" may have been clearer if phrased to avoid the vague notion of 'reliance'.

Author response:
We added a small clarification (in bold) to the first sentence of the selective retrieval hypothesis paragraph where we use the word “rely”: *>the model learns to rely on them [define-dot definitions] more **(e.g. retrieve them more frequently)***.

Reviewer comment (Writing):
Provide clear and precise definitions of OCL and meta-OCL. The current definitions ("LLMs will be more likely to respond to questions as if the true statements (tagged with Define) from the training set are in fact true; that is, these statements are internalized more" for OCL, and "a difference in internalization even for statements that are equally compatible with other questions in the training data, i.e. statements about variables for which no questions appeared in the training set" for meta-OCL) could be introduced more clearly and precisely. As written, understanding requires careful reading of the entire paper and revisiting sections, which not all readers will do.

Author response:
Thanks a lot for these comments and suggestions! We are considering the following edits.

(J1) We tweaked the sentence introducing meta-OCL and pointed to Figure 1b there: *>More surprisingly, we observe such a difference in internalization even for statements that are equally compatible with other training examples, i.e. statements about variables for which no questions appeared in the training set (Figure 1b).* We will edit this further as we agree that clarity here is especially helpful for understanding the rest of the paper.

(J2) For meta-OCL, we believe Figure 2 results are referenced quite clearly in Section 2.4, and the subsequent sentences explain why this is meta-learning. We will add a similar sentence when referencing Figure 2 in Section 2.3, and briefly recap the description of OCL from the intro.

(J3) We will slightly improve the notation by replacing $\hat{\mathtt{QA}}_4$ with $\mathtt{QA}_4^{\text{not replaced}}$ or similar, and $\mathtt{QA}_7$ with $\mathtt{QA}_7^{\text{unseen vars}}$ or similar. We will also point out the connection between {$\mathtt{\dot{D}}_1\mathtt{QA}_1$, $\mathtt{\bar{D}}_2\mathtt{QA}_2$, $\mathtt{\dot{D}}_5$, $\mathtt{\bar{D}}_6$} with the columns of Figure 1.

Reviewer comment (Presentation):
When discussing Figure 2, make clear what exactly needs to be fulfilled to qualify as OCL/meta-OCL. Explain in the main text and appendix why 'purple line above pink' is OCL and why 'blue line above red' is meta-OCL.

Author response:
Thanks a lot for these comments and suggestions! We are considering the following edits.

(J1) We tweaked the sentence introducing meta-OCL and pointed to Figure 1b there: *>More surprisingly, we observe such a difference in internalization even for statements that are equally compatible with other training examples, i.e. statements about variables for which no questions appeared in the training set (Figure 1b).* We will edit this further as we agree that clarity here is especially helpful for understanding the rest of the paper.

(J2) For meta-OCL, we believe Figure 2 results are referenced quite clearly in Section 2.4, and the subsequent sentences explain why this is meta-learning. We will add a similar sentence when referencing Figure 2 in Section 2.3, and briefly recap the description of OCL from the intro.

(J3) We will slightly improve the notation by replacing $\hat{\mathtt{QA}}_4$ with $\mathtt{QA}_4^{\text{not replaced}}$ or similar, and $\mathtt{QA}_7$ with $\mathtt{QA}_7^{\text{unseen vars}}$ or similar. We will also point out the connection between {$\mathtt{\dot{D}}_1\mathtt{QA}_1$, $\mathtt{\bar{D}}_2\mathtt{QA}_2$, $\mathtt{\dot{D}}_5$, $\mathtt{\bar{D}}_6$} with the columns of Figure 1.

Reviewer comment (Novelty):
Could you elaborate on your 'Comparison with In-Context Learning' paragraph? You say you wish to 'clarify the difference between out-of-context and in-context learning', but where is this clarification? Your description sounds a lot like meta-OCL. To me, the difference is that the association between tag and entity now works a lot better (performance for QA_4 == D_1QA_1). Also, confirm that in this setup you don't actually perform 'in-context learning' as defined by Brown et al. (2020) – rather, you repeat the same setup but concatenate definitions and QA-pairs into a single input. It is misleading to call this a comparison to in-context learning.

Author response:
Your understanding of our in-context learning (ICL) setup is correct. We believe describing this setup as ICL is consistent with the definition of ICL from Brown et al (2020): *>using the text input of a pretrained language model as a form of task specification … [examples are] a natural language instruction and/or a few demonstrations of the task*. While we don’t provide demonstrations of the task, our definitions are essentially instructions to treat (or avoid treating) the variable as the entity (and the model has to learn the meaning of these instructions). We agree it is still a bit of a stretch to call following such instructions learning (as opposed to e.g. inference or reasoning), but we think it makes sense as a contrast to out-of-context learning (the model relying on the knowledge from its training data / parameters, not information included in its context, when making predictions).

>This sounds a lot like meta-OCL.
    
Did you perhaps mean meta-ICL? If so this is definitely the case: this setting is very similar to that in Figure 1.1 (language model meta-learning) of Brown et al (2020); our Figure 1 was partly inspired by that figure. We’ll point out this connection explicitly. If you meant something else, could you please clarify this?

Reviewer comment (Experiments):
Just to make sure: QA-pairs are always different between the train and test splits, correct?

Author response:
Yes, the QA-pairs are always different between the train and test splits, although question *types* are not different – e.g. the train set could have a question “When was xyz born” and the test set could have the same question but about a different variable. Only the “entity attribution” question types are never present in the training set.

Reviewer comment (Novelty):
I am not convinced whether meta-OCL framing is needed here – what does it add? Is it not enough to say that neural networks generalize differently for true and false statements and that this holds for both unseen questions and unseen definitions? What is meta- about this? In Figure 12, the authors show that they observe similar effects in a single-stage setting, indicating that appealing to meta-learning might not be needed.

Author response:
The fact that neural networks generalize differently in our experiments  means that the model updates differently when trained on consistent-seeming and inconsistent-seeming datapoints in the second finetuning stage. This difference in updates is why we call this phenomenon "meta-learning". Our results can be explained without mentioning meta-learning (e.g. the "selective retrieval" hypothesis in Section 4); still, we believe that meta-learning is an interesting and helpful frame for understanding what's happening.

Indeed it is harder to view single-stage results as meta-learning, since it's unclear if the effect there comes from just training on all the data jointly, or a meta-learning-like effect. We employ the two-stage setting to disentangle these two explanations.

Reviewer comment (Presentation):
The authors perform many versions of their experiments. In general, this is a good thing. However, I found the presentation a bit strenuous. In many places, an explanation is given about the setting, but then the reader is referred to the SI for the results. It would be better if a subset of these experiments were moved to the SI entirely, leaving more space to put the plots/results for the other experiments in the main paper.

Author response:
Would it be fair to characterize your concern as "the references to the Appendix are too frequent"? We are wondering whether having less granular references to parts of the appendix would be helpful (e.g. instead of two references in “varying model size and experiments with other models”, have one reference to a larger appendix section relevant for both).

Reviewer comment (Novelty):
The related work section felt like quite a stretch. I found the discussed work not very relevant to the present paper.

Author response:
We’d appreciate any suggestions for the works that would be helpful to cover, or comments on which discussed works you believe to be the least relevant. We could see how the gradient alignment part of related work could seem extraneous given that we do not study this hypothesis in depth – was this the part that led to your concern?

Reviewer comment (Presentation):
Figure 2 is labeled “Performance on in-distribution questions”. This is quite confusing, as the figure states that it measures performance on the validation subsets (which is in line with my understanding of the paper). Which is correct?

Author response:
Both descriptions are correct – we always evaluate using validation subsets. We use “in-distribution” as a description of the question types: in-distribution validation data has the same questions types as the training data, whereas the entity attribution questions are never present in the training data. We’ll make this clearer in the caption of Figure 2.

Reviewer comment (Evaluation):
How does Figure 3 show in-context learning? It seems like a comparison to the data from Figure 2 would be needed for that.

Author response:
In-context learning is the result of the setup for Figure 3: in this experiment, definitions appear in the same prompt as the question which the model needs to answer, and so the model learns to use these definitions in the prompt to answer questions. Comparison with Figure 2a tells us that out-of-context learning is a weaker effect than in-context learning, since consistent definitions help less when the model was previously trained on them VS the definition being in the prompt. We’ll add a sentence about this to the paragraph on “comparison with in-context learning”.

Reviewer comment (Evaluation):
Why is the performance in Figure 4a lower than in the main experiment? Shouldn't it approach Figure 2 for alpha → 1?

Author response:
See our response to question 2.1 from reviewer zUXo about the same point. We will change the question type to be consistent with Figure 2b (right now it is not) and re-run the experiment to check whether something else is affecting performance.

Reviewer comment (Evaluation):
From my intuitive understanding, I would have thought that having a pretrained model is necessary for this effect to appear. Yet, the authors show in Section 3.1 that pretraining is not necessary. Would it be possible to further elaborate on why the authors believe pretraining is not necessary? Performance in these experiments also seems to be quite low (54%) for a two-alternative forced-choice task. Why is that the case?

Author response:
Neither hypothesis for explaining meta-OCL discussed in Section 4 relies on pretraining. Below we will attempt to formulate a version of the “selective retrieval” explanation for the set inclusion experiment.

During the first finetuning stage: 
1. The first few epochs result in middle-layer activations of each variable token containing information on the definition that was present in the training data – specifically, information about both the define-tag token and the number present in the definition. This is easiest to imagine for the (variable, tag, entity) definition order, where the model is trained to predict the tag and the number from the variable.
2. In the remaining few epochs, the model learns to use this information when answering questions. When predicting the answer, the model outputs the variable’s definition number IFF the variable’s define-tag is D-dot. (Note how this kind of selective retrieval/output mechanism can achieve low training loss AND is representable by a neural net.)
3. Realistically, the above two steps are intertwined and affect one another, and are also intermixed with the model just memorizing the training data without connecting definitions to questions, so the conditional output rule is not learned perfectly.

During the second finetuning stage, middle-layer activations of new variable tokens end up being structured similarly: they also contain info on the define tag and the number that followed the variable in the definitions. The selective retrieval mechanism from the first finetuning stage is unchanged, and results in the meta-OCL effect.


In this experiment, presence & strength of OCL & meta-OCL depends a) number of entity-variable pairs in the training data, and b) number of QA examples per entity-variable pair. Increasing a) helps the model learn the task but also makes the task more difficult since the model needs to learn more variable binding relations. Increasing b) helps the model learn the task as well, but makes definitions less useful as the model can infer more from the QA pairs. 

Consistently with this, we found that when we increase the number of QA examples per entity-variable pair, the first stage performance becomes very good (it’s easy to push it to almost 100% accuracy), but any effect in the second stage disappears.

We believe it should be possible to tune these and other parameters (such as how many integers are present in each question) and attain a stronger effect; still, our experiments are sufficient to show that pretraining is not needed for meta-OCL.

Reviewer comment (Writing):
The paper introduces new terminology without defining them in the first place, e.g., meta-out-of-context learning (in the abstract), internalize, definitions. Especially, the abstract is not understandable before reading the main text.

Author response:
- Re abstract: we are considering replacing the second sentence of the abstract with 
    
    >*Using carefully designed synthetic experiments with LLMs, we establish the existence of a phenomenon we call meta-out-of-context learning (meta-OCL): LLMs learning how to update on new datapoints in a way helpful for generalization but not explicitly encouraged by the training loss.* 

    Would such a change address your concern about the abstract’s readability?

Reviewer comment (Theory):
The mathematical definitions of the terms are missing, or rigorous quantification will be helpful, e.g., quantifying internalize.

Author response:
- We believe that while our current operationalization of internalization (“model treats a statement as true when generating new text”) can be formalized somewhat, trying to write this down in math would not make our paper clearer. For example, we could say that statement A is epsilon-internalized when “generated statements implying NOT A are epsilon times as frequent as generated statements that do imply A”. However, this is still very abstract (and requires an oracle that can tell whether one statement implies another), and so we believe it's best to leave the current informal operationalization as is.

    We think there are other promising directions to study meta-OCL more formally, such as formalizing a datapoint's usefulness for modeling other data (see our response to reviewer zUXo's Q1.1).

Reviewer comment (Writing):
The following sentence is too general: "language models trained with gradient-descent-based methods." Most of the existing language models use gradient descent to train, including RNNs and LSTMs, while this paper focuses on recent language models.

Author response:
- Our experiments with transformers cover both decoder-only and encoder-decoder architectures, as well as both finetuning these models and training them from scratch. We also present an experiment showcasing meta-OCL with a ConvNet. We would be quite surprised if it was not possible to reproduce meta-OCL with LSTMs or RNNs – do you expect otherwise? Given this, we believe it is fair to claim the quoted level of generality, as opposed to e.g. making that claim only about transformer-based language models.

Reviewer comment (Presentation):
The Subset definition in Table 1 is quite complicated. When reading the results, the reader has to read Table 1 multiple times to understand the results.

Author response:
- We agree that our setup is quite involved, and would like to improve our description of  it. In response to your comment we will modify the caption of Table 1 to point out the connection between {$\mathtt{\dot{D}}_1\mathtt{QA}_1$, $\mathtt{\bar{D}}_2\mathtt{QA}_2$, $\mathtt{\dot{D}}_5$, $\mathtt{\bar{D}}_6$} data subsets and the columns of Figure 1. See also our response to reviewer zUXo about potential improvements to our notation – we’d appreciate any thoughts you might have on this.

Reviewer comment (Experiments):
It would be interesting to see the comparison on aligned LLMs, i.e., the instruction finetuned LLMs (e.g., Llama-chat) [2], as these models behave differently for in-context samples [3].

Author response:
4\. We are not familiar with setups that involve finetuning a LLM in a self-supervised fashion after it was already finetuned with RLHF, which is why we did not include this kind of experiment. Following your comment we did run this experiment with `Llama-2-7b-chat-hf` out of interest, and the results are very similar to what we report for regular `Llama-2-7b` in Figure 16.

Reviewer comment (Writing):
I found the paper rather confusing at first, and quite hard to understand. The desire to introduce new terms and to use concepts such as “internalization” was counterproductive. Figure 1 ended up being useful to understand what was done, but it took me quite a while longer to fully grasp it as I would have liked. The choice to stick to the out-of-context framing felt counterintuitive to me, instead of being anchored in a more classical “learning the data” theory. Talking about “variable binding” earlier would have helped.

Author response:
We will mention the variable binding explanation ("the model learns the semantics of the define tags correctly" from Section 4) in the last paragraph of the introduction, which will hopefully make our paper clearer for future readers.

Reviewer comment (Experiments):
The experiments in Section 3 felt significantly weaker than the rest. All effects in the Appendix were weak (i.e. gap between congruent vs incongruent operators), and the MNIST-based visual task was quite strained.

Author response:
Regarding experiments from section 3 (set inclusion and MNIST): any systematic difference, even if small, is a positive result here. We do agree that our MNIST setup is quite far from any real-life computer vision setting; it is however very similar to the set inclusion setup, and the presence of meta-OCL there indicates that this phenomenon is not limited to transformers. (You might also be interested in our response to question 4 from reviewer Rz8f about the factors affecting set inclusion experiment's performance.)

Reviewer comment (Theory):
Consider replacing all arguments about “truthfulness” with “predictability”, would you still keep the same arguments throughout the paper? I personally wouldn’t.

Author response:
1.1. Replacing the arguments about “truthfulness” with “predictability” preserves the meta-learning framing: in the second finetuning stage, the model internalizes datapoints that look like they might be more predictive of other data (but the model never saw such data of which these datapoints are predictive). We believe formalizing this mathematically is a good avenue for future studies of meta-OCL; a metric similar to predictability that we think is highly relevant is something like “how much does retrieving knowledge from a given datapoint helps reduce the training loss on other training data”, which is also similar to Data Shapley [1].

Reviewer comment (Theory):
The statement “This is non-obvious because the training loss does not explicitly encourage such generalization, since there are no QA pairs about bgn in the training set” is not accurate. The model learnt that the congruent-Define is indeed a variable binding from bgn to Darwin, and hence will then replace all potentially new QA/data pairs about bgn with Darwin and behave as such. This will happen even in novel situations, through the model activations. If the argument is about “how many updates does it take for the congruent-Define to be learnt”, I’m not sure this can be appropriately assessed (the datasets are small, you perform many epochs, Transformers are big, etc).

Author response:
1.2. We do not make any points about “how many updates does it take for the congruent-Define to be learnt”. Rather, we claim that the fact that the model learns to bind the variable bgn to the entity Darwin from being trained on the string “congruent-Define bgn Darwin” can be seen as meta-learning, and would be surprising to a substantial fraction of the ML community. We believe the ability of LLMs to internalize semantic content of the text (as opposed to shallow token co-occurrence statistics) is currently under-appreciated, and our study is an attempt to highlight and better understand this ability.

Reviewer comment (Evaluation):
The incongruent-Define was trained to be non-predictive, and is probably learning to map bgn to a “zero” embedding (akin to what people are doing for classifier-free guidance), which has low correlation with all other entities in the dataset. It behaves entirely differently to congruent-Define, and hence trying to compare them is like trying to compare how predictable different datapoints are: it depends on the dataset, and if they aren’t predictable accuracies will be lower. You cannot really use its failure as a contrast to the “success” of congruent-Define in being effectively learnt.

Author response:
1.3. We might be missing your argument here: indeed in $\mathcal{X}_1$ (first stage finetuning data) define-dot definitions are predictive of other training datapoints, while define-dash definitions are much less predictive. This is not the case in $\mathcal{X}_2$ – here both types of definitions are equally predictive. The purpose of comparing the two is to demonstrate that the model learns to update on datapoints resembling those that were more/less predictive in the past in a way advantageous for generalization.

Reviewer comment (Evaluation):
Why are exact match numbers in Figure 4a so much lower than in Figure 2? I would have expected alpha -> 1 to match D1 in Figure 9d, so ~0.2. The fact that you changed the question type is also unfortunate; using “What is the name” and referring directly to Figure 2b would be more helpful. You also do not explicitly say that the comparison is Figure 2b (it could have been 2a?).

Author response:
2.1. Thanks for catching this – we’re not sure why the numbers are so different, indeed it is a bit surprising that the last 5% of $\alpha$ should make as much of a difference. We are re-running this experiment to see if we can catch the issue. We will also change the question type and point out the connection with Figure 2b explicitly, thanks!

Reviewer comment (Evaluation):
Changing alpha is modifying the “predictibility” of a definition, and with it how “useful” the congruent-operator is.

Author response:
2.2. Agreed!

Reviewer comment (Experiments):
Alpha controls the performance exactly as I expect it to, but I would have wanted to see values for alpha < 0.5 to see how low it gets.

Author response:
2.3. A setting with a given $\alpha$ is equivalent to that with 1-$\alpha$, so we don’t expect $\alpha$<0.5 experiments to be different from the ones presented in Figure 4a. The only difference is $\mathtt{\dot{D}}$ and $\mathtt{\bar{D}}$ changing places in which one is the more consistent/predictive one; in our case the define tags are random strings that are re-sampled across multiple seeds, so it does not matter which one we call $\mathtt{\dot{D}}$ and which $\mathtt{\bar{D}}$.

Reviewer comment (Evaluation):
The fraction of inconsistent entities for the incongruent-operator was made equal to the new number of consistent entities. For alpha = 0.75, this implies that you also have a large number of inconsistent entities for the incongruent-operator. And so in Figure 4a, the D9 number is fully explained by “you just have more data that shows that the incongruent-operator is a random map”; it goes down in the same way as it does in Figure 2.

Author response:
2.4. and 2.5. We did not understand the sentence below and would appreciate clarifications:  
>And so in Figure 4a, the D9 number is fully explained by “you just have more data that shows that the incongruent-operator is a random map”, it goes down in the same way as it does in Figure 2.  

We believe the $\mathtt{\bar{D}}^\text{consis}_9\mathtt{QA}_9$ result is interesting because for high $\alpha$, the define tag pushes the model to internalize definitions *less* (since, as you say, there is enough data to learn the define-incongruent meaning), while the actual definition consistency pushes the model to internalize definitions *more* (internalizing specifically these definitions helps predict the answers to the training questions). This setting is different from that in the first stage of Figure 2, where these two factors always push in the same direction. It is also different from the second stage of Figure 2, where only the define tag affects internalization (since there are no training QA pairs for the model to tell whether the definitions are in fact consistent).

Reviewer comment (Evaluation):
Please explain what one should expect to see differently to counter the interpretation above.

Author response:
2.4. and 2.5. We did not understand the sentence below and would appreciate clarifications:  
>And so in Figure 4a, the D9 number is fully explained by “you just have more data that shows that the incongruent-operator is a random map”, it goes down in the same way as it does in Figure 2.  

We believe the $\mathtt{\bar{D}}^\text{consis}_9\mathtt{QA}_9$ result is interesting because for high $\alpha$, the define tag pushes the model to internalize definitions *less* (since, as you say, there is enough data to learn the define-incongruent meaning), while the actual definition consistency pushes the model to internalize definitions *more* (internalizing specifically these definitions helps predict the answers to the training questions). This setting is different from that in the first stage of Figure 2, where these two factors always push in the same direction. It is also different from the second stage of Figure 2, where only the define tag affects internalization (since there are no training QA pairs for the model to tell whether the definitions are in fact consistent).

Reviewer comment (Writing):
I found most of the notation confusing, even while writing this up (i.e. operators, datasets, data subsets, stages, most of them). There are many modifiers applied to very few symbols, instead of a small clearer set of well-defined and informative letters/words.

Author response:
3. See our comment regarding notation on Weakness 4.

Reviewer comment (Experiments):
The fact that question types are shared across datapoints feels potentially problematic (Appendix A.3). Don’t they provide correlations for the model to hook onto, regardless of variables?

Author response:
4. Question type sharing does indeed provide base rates for the answers. E.g. for a test question “what did xyz do” the model’s answer would be informed not just by the definition and the QA pairs about xyz, but also by the answers to this type of question for other variables in the training data. This is not an issue for us since our results are all based on the relative differences between the various data subsets.