D OES D EEP L EARNING L EARN TO A BSTRACT ? A S YSTEMATIC P ROBING F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Abstraction is a desirable capability for deep learning models, which means toinduce abstract concepts from concrete instances and flexibly apply them beyondthe learning context.
At the same time, there is a lack of clear understanding aboutboth the presence and further characteristics of this capability in deep learningmodels.
In this paper, we introduce a systematic probing framework to explore theabstraction capability of deep learning models from a transferability perspective.
Aset of controlled experiments are conducted based on this framework, providingstrong evidence that two probed pre-trained language models (PLMs), T5 and
GPT2, have the abstraction capability.
We also conduct in-depth analysis, thusshedding further light: (1) the whole training phase exhibits a "memorize-thenabstract" two-stage process; (2) the learned abstract concepts are gathered in a fewmiddle-layer attention heads, rather than evenly distributed throughout the model;(3) the probed abstraction capabilities exhibit robustness against concept mutations,and are more robust to low-level/source-side mutations than high-level/target-sideones; (4) generic pre-training is critical to the emergence of abstraction capability,and PLMs exhibit better abstraction with larger model sizes and data scales.
I NTRODUCTION
Whereas concrete concepts are typically concerned only with things in the world, abstract concepts are about internal events.
— Barsalou et al.
Abstraction means capturing the general patterns (often referred to as abstract concepts ) efficientlyin a specific learning context and reusing these patterns flexibly beyond the context (Mitchell, 2021;
Kumar et al., 2022; Giunchiglia & Walsh, 1992; Hull, 1920).
For instance, the abstraction on languagemeans recognizing the underlying syntax and semantics behind concrete sentences.
It is thought tobe one of the fundamental attributes in human cognition for effectively learning, understanding androbustly generalizing, and has been studied for a long time in cognitive psychology and behavioralsciences (Gentner & Medina, 1998; Barsalou et al., 1999; Shivhare & Kumar, 2016; Konidaris, 2019).
The abstraction capability is also critical for deep learning, but many previous studies suggested thatthe surprising success of deep learning may come from the memorization of some surface patterns(also called superficial correlations or shortcuts)
(Geirhos et al., 2020; Du et al., 2022), such as somespecial tokens (Niven & Kao, 2020; Gururangan et al., 2018), overlapping contexts (Lai et al., 2021;
Sen & Saffari, 2020), and familiar vocabularies (Aji et al., 2020).
It is still unclear whether themodels just memorize these patterns without abstractions, or they do learn abstract concepts (yetoverwhelmed by surface patterns when applied in a similar context as in training).
Therefore, thispaper aims to take a step forward to probe the abstraction capability of deep learning models ,keeping the effects of abstract concepts and surface patterns decoupled and controlled individually.
Our key idea is to probe the abstraction capability from a transferability perspective, since surfacepatterns are always bounded with task-specific characteristics while abstract concepts can be moregenerally reused.
We consider designing multiple tasks with shared abstract concepts and totallydifferent surface patterns, then tracing whether the learning on one task can boost the performance onanother.
Figure 1 demonstrates a motivating example.
Motivating Example As shown in Figure 1, suppose we want to examine whether a model canlearn the abstract rule (i.e., the symbolic mapping rule x 1 x 2 → X 1 X 2 , in which x
i and X i are
[Figure]
Surface Patterns w/o Abstraction with Abstraction NewContext Update
[Figure]
Well
[Figure]
×
[Figure]
𝑨𝑪 … [Figure] general variable slots) from the task A , or just memorize surface mappings (e.g., ab → AB , inwhich a and A are task-specific symbols).
To reveal the different transferability of two learningmechanisms, we utilize a probing task B that contains the same underlying abstract rule as task Abut does not overlap with its symbol set.
If the model could learn the abstract rule from task A , itwould reuse it to interpret new context, thus effectively solving task B .
But if not, memorizing somesurface mappings that are bounded with task-specific symbols is less effective to solve task B .
Motivated by this example, we design a systematic framework for probing abstraction capability .
This framework considers a set of probing tasks along with three procedures of experiments basedon the transfer learning paradigm.
The use of abstract concepts and task-specific characteristics inprobing tasks are separately controlled.
To probe the abstraction capability of language models, thiswork mainly considers grammar as the abstract concept 1 .
The grammar of a formal language isa set of hidden rules behind concrete sentences and determines how terminals are combined intosentences that are valid to the syntax.
We want to explore whether the model can be aware of thegrammar, or simply memorize some specific word combinations .
We instantiate our frameworkas a grammar probe that is constructed from the designed formal grammar and terminal sets.
Theprobing results show strong evidence that two probed PLMs (specifically, T5-Base (Raffel et al.,2020) and GPT2-Medium (Radford et al., 2019)) have the abstraction capability to learn abstractconcepts from concrete instances, rather than just simply memorizing surface patterns.
After probing the existence of abstraction capability, we further explore the following questions.
RQ1 : What is the characteristic of the training dynamics on learning abstraction?
RQ2 : How arethese learned abstract concepts distributed in the model?
Concentrated in a few modules or evenlydistributed in whole model?
RQ3 : How robust is the abstraction capability on tasks with mutatedabstract concepts?
RQ4 : How would generic pre-training and general factors influence abstraction?
Here we outline some interesting findings from our in-depth investigations : (1) the trainingphase exhibits a "memorize-then-abstract" two-stage process; (2) the abstract concepts learned inour probes are gathered in a few middle-layer heads; (3) abstraction capability is more robust tosource-side/low-level mutations than to target-side/high-level ones; (4) generic pre-training is criticalto the emergence of abstraction capability, and larger model size and data scale are beneficial.
Contributions 1)
We propose a systematic probing framework for abstraction capability, guidingthe design of controlled tasks and procedures from a transferability perspective.
We instantiate thisframework with concrete tasks and show strong evidence that two probed PLMs have the abstractioncapability.
3) We further analyze this capability and provide insightful conclusions by investigatingthe above research questions.
We will release our data and code, and hope that this can be a usefultool to facilitate understanding of the abstraction capability of contemporary deep model.
R ELATED W ORK
Probing deep learning models.
To explore whether deep learning models have certain capabilities,there has been much work examining these black-box models in some specially designed settings, called probes (Petroni et al., 2019; Tenney et al., 2018;
Warstadt et al., 2019; Lin et al., 2019; Hewitt& Manning, 2019; Vuli´c et al., 2020).
The key challenge in designing probes is to exclude superficialcorrelations.
That is, the performance of the model in the probing setting should be highly correlatedwith the capability to be probed rather than other influencing factors.
For instance, to probe whetherthe model encodes some knowledge/information in the representation rather than just over-fit thedata, a standard approach is to freeze the model parameters (Petroni et al., 2019; Tenney et al., 2018);to probe whether the model have compositionality rather than just memorize the label distribution,previous work injected statistical bias into the data splits (Lake & Baroni, 2018; Keysers et al., 2019;
Kim & Linzen, 2020).
In this work, to explore whether models can learn abstractions rather than justmemorize surface patterns, we leverage the transferability of abstraction, which has been consideredas one essential property of abstraction (Mitchell, 2021; Kumar et al., 2022) and explored from acognitive science perspective on neural networks (Dienes et al., 1999; Geiger et al., 2022).
Abstraction capability.
Abstraction has been studied for a long term in cognitive psychology andbehavioral sciences (Hull, 1920; Gentner & Medina, 1998; Barsalou et al., 1999; Burgoon et al.
,2013; Wang, 2015; Shivhare & Kumar, 2016; Lake et al., 2017; Daniel, 2017; Konidaris, 2019) andhas attracted attention in the artificial intelligence field (Giunchiglia & Walsh, 1992; Richardson et al. ,2020; Clark et al., 2020; Talmor et al., 2020; Mitchell, 2021; Zadrozny, 2021;
Millhouse et al., 2021;
Kumar et al., 2022).
The abstraction capability of DNN models has been explored in many tasks suchas visual reasoning (Johnson et al., 2017; Barrett et al., 2018; Chollet, 2019; Kumar et al., 2022),grounded language understanding (Ruis et al., 2020), and game playing (Tsividis et al., 2021).
As ourwork focuses on language models, another closely related topic is compositional generalization (Lake& Baroni, 2018; Keysers et al., 2019; Kim & Linzen, 2020), which explored whether neural modelscould learn high-level grammars from specially designed training examples and apply the learnedgrammars through compositions.
These works concluded that general-propose neural models (such as
LSTM and Transformer) could not learn the full grammar with biased observations and demonstratedthe importance of symbolic mechanisms for abstraction (Liu et al., 2020; Chen et al., 2020; Liu et al.,2021a).
Some other previous work also explored the abstraction of language models in their speciallydesigned tasks (Chollet, 2019; Mitchell, 2021; Zadrozny, 2021).
Most previous explorations of DNN abstraction capabilities did not consider to explicitly avoid andcheck the influence from task-specific characteristics, thus leaving potential risks that the modelmay perform well in terms of surface patterns over-fitted to task-specific designs (e.g., patterns incandidate answers (Zhang et al., 2019)) rather than abstract concepts.
Some implicit strategies havebeen leveraged to alleviate such potential influence through indirect ways: some previous workconsidered using biased task-specific designs in training and test data separately (Kim & Linzen,2020; Barrett et al., 2018); some have attempted to fix the observed problems in existing probes onan ad hoc basis (Hu et al., 2021; Benny et al., 2021); some considered to inject great task diversity,which implicitly increases difficulty of learning practical shortcut (Chollet, 2019).
In this work,rather than implicitly alleviating this potential risks, we consider to explicitly check whether there isperformance leakage from surface patterns by leveraging the transferability of abstraction capabilityand comparing performance among a set of controlled experiments.
P ROBING F RAMEWORK
As mentioned in Section 1, abstraction capability means to induce abstract concepts from concreteinstances in a certain learning context and flexibly generalize these concepts beyond the context.
A key difference between a surface pattern and an abstract concept is their different cross-tasktransferability, as the former is always bounded with some task-specific characteristics (e.g., a certainvocabulary) while the latter is transferable across tasks.
We formally define this property as following.
Property: Transferability of Abstraction.
Consider two machine learning tasks A and B that do notshare any common instances between their task-specific characteristics spaces, but have essentiallythe same set of abstract concepts behind them, the transferability of abstraction means that learning
A can help better learn B .
Notations: We denote two learning paradigms:
A ⇒ B as a procedure that the model is firsttrained on A and then fine-tuned on B , and ⇑ B as a procedure of directly training on B without thepre-training on A .
The transferability of abstraction can be evaluated with the performance gain fromprocedure A ⇒ B compared to ⇑ B , denoted as ∆( A ⇒ B ) .
[Figure]
Procedures of Experiments
Expectations
[Figure]
Based on this property, we can verify the abstraction of a model by examining whether themodel exhibits transferability (i.e., assessing ∆( A ⇒ B ) ) .
In the following, we formally design aframework for the probing of abstraction in a systematic manner and illustrate it in Figure 2.
Aiming
• This framework examines whether a probed model could learn abstract concepts C A from
the aiming task A with a train set A .
Task Design
• Probing task B with the transfer set ˆ B and test set B contains the abstract concepts C B that is required to be the same as C A .
The task-specific characteristics used to construct ˆ B ∪ B do not overlap with that of A .
In addition, the examples in ˆ B are restricted to contain insufficient information for the probed model to learn C B perfectly.
Thus, the gain from the abstraction in task A would be noticeable.
• Contrast task C with the contrast set C aims to further confirm that the performance in task B is principally correlated with abstractions rather than other factors.
The abstract concepts C C is constructed by greatly breaking (changing) C A , thus compared with task A , the abstraction on task C is less effective for solving task B .
The task-specific characteristics and other latent factors in constructing C are kept the same with that in A .
Procedures of Experiments ( abbreviated as Exp)
• Control Exp ⇑ B : only train the model on ˆ B and test on B .
• Main Exp A ⇒ B : train the model on A , then fine-tune on ˆ B , finally test on B .
• Contrast Exp C ⇒ B : train the model on C , then fine-tune on ˆ B , finally test on B .
Hypothesis and Expectations
• Hypothesis: the probed model can learn abstract concepts C A from A .
• Expectation 1: ∆( A ⇒ B ) is significantly high, i.e., A ⇒ B brings considerable gain
compared with ⇑ B .
• Expectation 2: ∆( C ⇒ B ) is significantly lower than ∆( A ⇒ B ) (or close to zero), i.e.,Expectation 1 is highly correlated with the learning of abstract concepts rather than otherfactors.
P ROBING A BSTRACTION C APABILITY OF L ANGUAGE M ODELS
The abstract concepts mainly considered in this work is grammar, a set of hidden rules behindconcrete sentences that determine how terminals are combined into sentences that are valid to thesyntax.
To design a grammar probe, we instantiate the framework with formal language translation(FLT) tasks.
We assume that the generative grammar of the source and target languages containthe abstract concepts of FLT tasks , and that the surface patterns (e.g., familiar bigrams) are boundedwith task-specific terminal sets.
We give a more specific definition of abstraction in FLT tasks:
[Table]
Definition : Considering an FLT task T :
L s → L t that translate the source language L s (withgrammar G s and terminals S s ) to the target language L t (with grammar G t and terminals S t ), and aset of concrete pairs T = { ( l is → l it ) } k in which l is and l it are sentences from L s and L t respectively,the abstraction capability is learning a mapping from G s to G t during training on T rather than justsimply memorizing terminal-specific patterns that are bounded with S s and S t .
Our FLT tasks are majorly derived from the synthetic semantic parsing task COGS (Kim & Linzen,2020) and the Probabilistic Context-Free Grammar (PCFG) it used.
We directly take the sourcegrammar G s of COGS which mimics the English natural language grammar, and reconstruct thetarget grammar G t to be chain-structured (detailed in Appendix K.1).
Part of the G s and G t areshown in Table 1.
Terminals in G s (and in G t ) can be divided into three groups respectively: the verbterminals S v (the PREDICATE terminals S P in G t ), the noun terminals S n (the ENTITY terminals
S E ) and the conjunction terminals S c (the CONCAT terminals S C ).
The production rules can alsobe categorized as T-Production rules (i.e., rules only containing terminals at the right side) and
N-Production rules (i.e., rules only involving non-terminals).
We assign task A and B (also C and
B ) with different group of terminals to ensure that there is no overlap between the task-specificcharacteristic spaces generated by the PCFG of task A and B (also C and B ).
We describe theinstantiation of different sets A , ˆ B , B and C as following.
Examples of these sets are contained in
Appendix F.1.
Train set A .
To synthesis A , we first derive the new source and target grammar (denoted as G + s and 186 G + t ) from G s and G t by only one-to-one replacing the terminals with a group of new terminals 2sampled from the Wordlist Corpora in NLTK (Bird et al., 2009).
Additionally, as the original terminalgroup S c (also S C ) only contains a single terminal, we add 31 additional terminals into the new S c(and S C ) to increase the diversity.
The terminal diversity will be further discussed in Section G.1.
Then, we use G + s and G + t to generate examples for train set A .
Note that G + s and G s (also G + tand G t ) share all non-terminals, N-Production rules, and the one-to-one mappings from source
N-Production rules to target N-Production rules, except for the terminal sets.
Transfer set ˆ B and Test set B .
We take the train set in COGS as ˆ B , and take the sententialcomplement ( Com. ) set and subject modification ( Mod. ) set as B for two sub-probes.
The ˆ B onlycontains examples with up to 2 recursions and object modifications, while the B contains up to 12recursions and subject modifications.
It has been proved that training on ˆ B is not enough for a DNNmodel to learn the full grammars of COGS for handling the test cases in B (Kim & Linzen, 2020).
Contrast set C .
Compared with A , C is generated with the same source grammar G + s , but the targetgrammar is totally changed as G − t : for each rule of G + t , its right-side word order is reversed 3 .
Exceptfor the generative grammar, all other factors are kept the same with A during generating C .
E XPERIMENTAL S ETUP AND M AIN R ESULTS
We probe two pre-trained language models: T5-Base (Raffel et al., 2020) and GPT2-Medium (Radfordet al., 2019).
Our experiments are based on the Huggingface Transformer models (Wolf et al., 2020).
For both (continue) pre-training and fine-tuning, we take Adam (Loshchilov & Hutter, 2018) with1e-5 learning rate and 0.01 weight decay.
Batch size is 8 and max training step is 100k.
We generate3 groups of new terminals, repeat the experiments on each group with 2 random seeds, and finally
[Table]
average 6 results.
The default evaluation metric is exact match accuracy.
The early-stopping strategyis applied to avoid catastrophic forgetting.
Detailed settings are listed in Appendix K.
Table 2 shows the main results of our probe.
For both two sub-probes, the performances of twoprobed models are in line with two Expectations set in Section 3.
First, the results of ⇑ B are verylow, and A ⇒ B can bring significant improvement, which is in line with Expectation 1.
Second,the results of C ⇒ B are much lower than A ⇒ B (and are even just comparable with ⇑ B ), whichis in line with Expectation 2.
As two expectations are experimentally examined, we can draw apreliminary conclusion: our probing results provide strong evidence that two probed PLMshave the abstraction capability to learn abstract concepts from concrete instances rather than justmemorize surface patterns, and to transfer the learned abstract concepts beyond specific tasks.
A NALYSIS
Based on our designed probe and results above, we further analyze the abstraction capability of PLMsto answer the RQs mentioned in Section 1.
All experiments below are derived from Com.
sub-probe,and are mainly conducted with T5-Base model except that are explicitly mentioned.
6. L EARNING P ROCESS OF A BSTRACT C ONCEPTS
To investigate the learning process of abstract concepts , we save checkpoints for every 1,000 stepsduring training on A .
Each checkpoint is further fine-tuned on ˆ B and tested on B .
For comparison, wealso investigate the process of memorizing surface patterns by directly examining each checkpointon the held-out dev set in task A .
Figure 3 shows the performance curves of two learning processes.
The training phase exhibits a "memorize-then-abstract" two-stage process.
As shown in
Figure 3, there is an obvious phase difference (48k training steps) between two time points that twolearning processes achieve their 90% relative performance, respectively.
Such a phase differencemeans that when the model has already performed well on task A in an early training phase, thelearning of desired abstract concepts is still on-going.
In other words, the in-task performance in anearly training phase comes mainly from the effects of some task-specific surface patterns ratherthan general abstract concepts.
With extending the training phase, the abstract concepts can be furtherextracted/enhanced.
This phase difference also suggests that the pre-training process should becontinued even if the model has already achieved a good in-pre-training performance .
The learning of abstract concepts is accelerated after in-task examples are well learned.
Afterthe model reaches 90% in-task relative performance (i.e. right side of the red dashed line), thelearning curve of abstract concepts (i.e., the blue curve) rises more rapidly.
The learning of abstract concepts is not stable in the early training phase.
The curve of in-taskperformance is much smoother than the cross-task one.
This suggests that the learning and transfer ofabstract concepts is not stable.
Nevertheless, the large fluctuations occur mainly in the early phasesof training.
With increasing training steps, this instability gradually decreases.
6. A BSTRACT A TTENTION H EADS
To investigate how the learned abstract concepts are distributed in the model, we first conduct preliminary experiments by separately freezing parameters in each layer and sub-layer during fine-tuning,and find that the parameters in attention sub-layers play important roles (detailed in Appendix E).
Tofurther determine the contribution of each attention head, we consider measuring the performance
[Figure]
i n
[Figure]
degradation after excluding the effect of each head.
Specifically, we evaluate the change in perplexity(PPL) of examples in B after pruning the normalized wight of each head as follows,
[Equation]
in which h represents a certain head, θ is the full set of parameters in PLM after fine-tuning on ˆ B , θ − hmeans pruning the h head, and ( l is , l it ) is the input-output pair in B .
Note that a higher PPL means alower performance.
Considering that some heads may store the task-specific knowledge learned fromfine-tuned data ˆ B , pruning these heads may also lead to performance changes.
Therefore, we alsoevaluate a baseline PPL change ∆ θ, ˆ B ( h ) on fine-tuned examples in ˆ B and measure the difference in
PPL changes (DPC) =
∆ θ,B − ∆ θ, ˆ B .
The DPC of each head is shown in Figure 4.
Abstract concepts are largely contained in a few heads, not evenly distributed in all heads.
Note that there are totally 432 attention heads in T5-Base.
Figure 4 shows that among hundreds ofheads, only a dozen of them are highly correlated with storing abstract concepts in our probe.
These abstract attention heads are gathered in middle layers in T5.
A larger index in Figure 4means that the corresponding head is more away from the input side and closer to the output side.
Itshows that the abstract attention heads (i.e., heads with high DPC) are mainly located in the middlelayers of T5-Base model, i.e., the last encoder layers and first decoder layers.
We further explore whether abstract concepts are modularized in the model.
A modular is a partof parameters that can individually perform a specific target functionality (Csordás et al., 2020).
To investigate modularity, we take the method of freezing certain parameters during fine-tuning toexamine whether the update of these parameters can be independent.
We consider the top 36 headswith the highest DPC (which contain some redundant heads) as abstract heads.
For comparison, weseparately experiment with freezing 36 random heads.
Table 3 shows that freezing abstract headstakes effect while freezing random heads does not.
We further explore the modularity in Appendix E.
[Table] [Table]
[Figure]
6. R OBUSTNESS OF A BSTRACTION C APABILITY
We explore the robustness of the probed abstraction capability when the abstract concepts in ourdesigned probes are mutated.
Different from the contrast task C in which the target grammar is totallychanged, here we consider partially injecting mutations into source/target-side grammar.
According tothe formal grammar in Table 1, we consider injecting mutations at different abstract levels: changing
T-Production rules can be regarded as a low-level mutation , since only terminals will be influencedand the whole sentence structure is kept; changing non-iterative N-Production rules can be regardedas a mid-level mutation , since the local structure will be mutated but the whole recursive structure ispreserved; changing iterative N-Production rules can be regarded as a high-level mutation , sincethe whole recursive structure will be reconstructed.
Based on the grammar used in formal languagetask A , we design three derivations G ∗ t by injecting mutations into the original G + t in different levels.
Examples of different derivations are contained in Appendix F.4.
Coarse G ∗ t (low-level mutation): We omit the non-terminals AGENT, THEME, RECIPIENT, andtheir corresponding terminals.
In other words, the second T-Production rule in Table 1 is removed.
Compared with G + t (also G t ), Coarse G ∗ t does not contain detailed arguments of PREDICATE.
Local Reverse (LocalR) G ∗ t (mid-level mutation):
The local word order in a sentence is reversed.
Specifically, we reverse the right-side word orders of the N-Production rules, except for the rule inthe last row of Table 1 which is an iterative one.
It means that the order of CLAUSEs (determined bythe last rule) remains the same, while the terminals in each CLAUSE are locally reversed.
Nested G ∗ t (high-level mutation): It is obtained by changing the iterative rule (i.e, the last rulein Table 1) from the chain-structure to be nested.
The new N-Production rule is "CLAUSE ↠
PREDICATE ( AGENT, CONCAT CLAUSE )".
We can also construct G ∗ s from G + s with the same technique except for the coarse one, as the sourcelanguage must contain enough information to generate targets.
Thus, we design a Redundant G ∗ swhich contains redundant terminals that are not mapped into targets (detailed in Appendix F.3).
Weseparately change the source and target grammars to derivations and show results in Figure 5.
PLMs can exhibit robustness against mutations in abstract concepts.
Results of these derivationswith mutations are higher than the Control Exp (and Contrast Exp), indicating that even though thelearned abstract concepts are only partially matched with that in downstream tasks, the abstractioncapability of PLMs can still leverage the similar parts in two sets of mutated abstract concepts.
Abstraction capability is more robust to low-level mutations.
Among three kinds of derivations,the low-level mutated ones (Coarse G ∗ t and Redundant G ∗ s ) perform best, while the high-level mutatedones (Nested G ∗ t and G ∗ s ) perform worst.
This trend implies that the robustness of the abstractioncapability decreases as the mutation level of abstract concept rises.
This also suggests that matchingof high-level abstract concepts should be prioritized when selecting pre-training tasks.
Abstraction capability is more robust to source-side mutations.
Comparing the results in
Figure 5a and 5b, source-side mutations bring less affects to downstream performance than target-sideones, indicating that PLMs can more robustly reuse source-side abstract concepts.
Redundant information barely affects abstraction.
Surprisingly, the performance of Redundant 309 G ∗ s is nearly the same with that of the original G + s , which means that injecting redundant informationinto inputs would hardly affect the learning of abstract concepts.
It indicates that the abstract capabilityof PLM can naturally exclude the influence of irrelevant information.
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Fuzzy abstract concepts can also be learned and transferred.
Compared with the formal grammar discussed above, which can be concretely defined, fuzzy grammar is more free (such as naturallanguage grammar).
To explore how would abstraction capability perform on fuzzy grammar, wetake natural language sentences for experiments and design different sets by mimicking the Com.sub-probe.
Detailed designs are described in Appendix H. We report BLEU score in Table 4.
It showsthat the performance of PLMs on learning fuzzy grammar is also in line with our two expectations.
G ENERAL F ACTORS & G ENERIC P RE -T RAINING
As explored in previous work, there are some general factors that influence the performance of DNNmodels (Bommasani et al., 2021; Wei et al., 2022; Henighan et al., 2020), such as model size and datascale.
We investigate how these general factors and the generic pre-training influence the learning ofabstract concepts.
More results and analysis can be found in Appendix G.1.
PLMs exhibit better abstraction with larger model sizes.
Figure 6a and 6b show that for both T5and GPT2 architectures, larger pre-trained language models have better abstraction capability thanthe smaller ones, as we can observe that the gains from the Control Exp to the Main Exp becomegreater with the model sizes increasing.
Larger data scale in pre-training helps better abstraction.
Figure 6c shows T5-Base performancewith different scales of the train set A .
It shows that performance increases rapidly from ∼ 300 to∼ 3.4K (with ∼ 50% absolute accuracy improvement) and improves marginally (and unstably) from∼ 3.4K to ∼ 680K (with ∼ 5% absolute accuracy improvement). Overall, the performance trend isgoing up with data scale increasing, indicating that the larger data scales benefit abstraction.
Generic pre-training is critical for the emergence of abstraction.
We probe the abstractioncapability of randomly initialized T5-Base and GPT2-Medium (i.e., without loading pre-trainedcheckpoints) and report the results in Table 5.
The poor performance on A ⇒ B reveals that withoutgeneric pre-training, these deep learning models can hardly extract transferable abstract conceptsfrom task A , even though they can still achieve >98% dev set performance on task A by fitting sometask-specific suffer patterns.
The comparison of the results in Table 2 and Table 5 demonstrate thatthe broader background pre-training is critical for the emergence of abstraction capability.
C ONCLUSIONS
In this paper, we introduce a systematic probing framework from a transferability perspective to guidethe design of probes for abstraction capability.
We instantiate this framework as a grammar probeand show strong evidence that two probed PLMs have the abstraction capability.
We further analyzethis probed capability by investigating several in-depth questions and provide insightful conclusions.
E THICS S TATEMENT
A sufficiently robust abstraction capability that can perfectly extract abstract concepts and exclude concrete information in any situation will help deep learning models avoid many potential risks of ethical issues such as social bias and privacy breaches.
However, as investigated in this work,the abstraction capability of some commonly used deep learning models may be fragile and can beaffected by their training situation.
This suggests that the abstraction capabilities of these models arestill not reliable enough to naturally avoid these potential ethical issues, and calls for future work toexplore ways to strengthen the robustness of the abstraction capabilities of deep learning models.
A D ISCUSSIONS
Below are more discussions about our work.
Potential factors that may hinder our probing.
We consider the main factor that could block theuse of our probing framework is the catastrophic forgetting problem in deep learning (Goodfellowet al., 2013; Kemker et al., 2018).
Since our probing framework relies on the transferability propertyof abstract concepts, if catastrophic forgetting dominates the learning of downstream tasks, suchtransferability will hardly take effect and the probing results will fail to reveal the abstractioncapabilities.
Considering this problem, we utilize the early-stopping strategy (detailed in Appendix)to alleviate catastrophic forgetting.
Moreover, our tested pre-trained models are naturally more robustto catastrophic forgetting (Ramasesh et al., 2021).
Better understanding "why does transfer learning work".
Recent success of transfer learningshows that pre-training (or continue pre-training) with similar source tasks can help better solvedownstream target task (e.g., question answering (Khashabi et al., 2020; Liu et al., 2021b), face verification (Cao et al., 2013), and general NLU tasks (Pruksachatkun et al., 2020)).
Some previous workin cross-lingual transfer learning empirically observed that the model can transfer some knowledgebeyond vocabulary (Artetxe et al., 2020; Ri & Tsuruoka, 2022), but they did not consider to excludethe influence from other potential factors.
Our results can serve as stronger evidence for the reason tothe success of transfer learning, that in addition to transferring some surface patterns, the better targetperformance can also benefit from similar abstract concepts learned from source tasks.
Limitations and future work.
The main limitations in this work are 1) we do not quantify theabstraction capability and 2) we only test two large pre-trained models.
We leave these two points toour future work.
Another future direction is to further explore the mechanisms behind abstractions.
B C OMPARISONS WITH P REVIOUS F INDINGS ABOUT L EARNING D YNAMIC
Comparison with information bottleneck.
Shwartz-Ziv & Tishby (2017) found a two-phase learning process from the view of information flow in deep neural networks: empirical error minimizationphase and representation compression phase.
This process is different from the memorize-thenabstract process since they measure the training dynamics in quite different perspectives.
The formerfocuses on the compression of representation (and reduction of mutual information) while the latterportrays the learning of abstract concepts.
The analogy between the two may lie in that the extraction of abstract concepts from concrete instances is in some way have the effect of informationcompression.
Comparison with Grokking.
Power et al.
(2022) revels that the improvement in generalization(on validation set) can happen well past the point of over-fitting (on train set).
Both ’grokking’and ’memorize-then-abstract’ phenomenon indicate that some general patterns are always learnedin a later training stage.
The difference is that the ’grokking’ focuses on generalization beyondover-fitting training data, while ’memorize-then-abstract’ portrays the transfer of abstract conceptsbeyond task-specific characteristics.
C P RINCIPLES FOR D ESIGNING P ROBING T ASKS
To verify whether the model could learn abstract concepts from task A by assessing ∆( A ⇒ B ) , wepropose the following principles for designing task B :
The space of task-specific characteristics of B should be very different from that of A sothat the memorization of surface patterns in A is helpless to B .
The abstract concepts of B should be the same as that of A so that the abstraction made onA could be reflected with a better performance on B .
The fine-tuning-only approach ⇑ B should be not enough to learn task B perfectly; otherwise,the gain from the abstraction on A would not be noticeable.
Furthermore, to verify that the performance gain on B is from the abstraction on A rather than otherfactors, we consider a contrast task C of A :
The abstract concepts of C should be very different from A (also B ), while other latent factors are kept the same such as data scale and pre-training steps.
The consideration of contrast task is similar to selectivity (Hewitt & Liang, 2019).
D A N O PERATION P ROBE
The operations semantics (e.g., conjunction) in the Boolean algebra can be regarded as transitionfunctions between the given Boolean variables and corresponding outputs.
We want to examinewhether the model can learn the meaning of operations from concrete logical expressions, orjust learn superficial correlations from specific sketches in expressions .
In operation probe, we instantiate the framework with logical expression evaluation (LEE) tasks.
Weconsider operation semantics in logical expressions as abstract concepts , and surface patterns(e.g., local string matching) are bounded with operation sketches.
Figure 7a shows two kind of sketches: chain sketch and tree sketch.
The model trained on chainsketch may learned the meaning of operations (e.g., conjunction of two Boolean variables) or simplymemorize some head or tail patterns of strings (e.g., if the head of input string is "False AND ( ", theoutput is always "False").
Learning operation semantics can more generally help understand otherexpressions with different sketches, but memorizing head or tail patterns in chain sketches is helplessor even harmful to understand tree sketches, since these surface patterns can lead to wrong results indifferent sketches.
We give a more specific definition of abstraction in LEE tasks:
Definition 2 : Considering an LEE task L : E s → B t that the source logical expressions E s (withoperation semantics P s and sketch K s ) are evaluated as Boolean values B t , and a set of inputoutput pairs L = { ( e is → b it ) } k in which e is is an logical expression sampled from E s and b it ∈{ True, False } is the evaluation result of e is , the abstraction capability is learning the meanings ofoperations P s during training on L rather than memorizing sketch-specific patterns that are boundedwith K s .
Our LLE tasks probe the learning of four logical operations: P + s = {Conjunction (Conj.)
, Disconjunction (Disc.), Alternative Denial (Alt.), Joint Denial (Joi.)}.
Figure 7b illustrates the transitionfunctions of these operations..
For generating data, each operations in P + s is constantly aligned withone operator (i.e., a concrete string) in S o .
Examples of these sets are contained in Appendix F.2.
Train set A .
We synthesize the data in A with P + s and chain sketch.
Each expression contains eightoperators which are sampled from S o .
Transfer set ˆ B and Test set B .
We synthesize the data in ˆ B ∪ B with P + s and tree sketch.
Eachexpression contains eight operators which sampled from S o .
When probing a certain operationp s ∈ P + s , we limit that the expression in ˆ B does not contain p s , while each expression in B mustcontain p s .
To make the model familiar with operators in S o or not forget them during further finetuning, we supplement ˆ B with 100 two-operator expressions which cover the full S o .
Empirically,as DNN models can be easily influenced by the statistical bias in label distributions, we balance the’True’ and ’False’ examples during sampling.
Contrast set C .
The operators and sketch in C are kept the same with A , but each operator in S ois aligned with another set of logical operations P − s = {Material Implication, Converse Implication,
Material Non
implication, Converse Non-implication}.
The transition functions of these operationsare listed in Appendix F.2.
Results of our operation probe is shown in Table 6. [Figure]
Conjunction
L e ft V a r . Disjunction Alternative Denial Joint Denial
Right Var
[Figure]
[Figure] [Table] E A BSTRACT C ONCEPTS ARE M ODULARIZED IN PLM S To supplement our analysis on abstract attention heads, here we provide our detailed explorations toidentify the modulars in PLMs that store abstract concepts in our probes.
Our explorations are basedon the following property and assumption.
Property: Forgetting of Abstract Concepts.
Consider the Main Exp A ⇒ B that task A and
B share the abstract concepts but do not share task-specific knowledge.
After fully fine-tuning ontask B , the model’s parameters will somehow over-fit the task-specific knowledge in task B and theabstract concepts stored in these parameters will be partially forgotten.
Assumption: Identifying the Modularization of Abstraction by Freezing Parameters.
If amodular in the model individually store a part of abstract concepts, these parameters can be directlyreused in new tasks without further fine-tuning.
Furthermore, considering the property above, freezingthis abstract modular can avoid the forgetting of abstract concepts, resulting in the performanceimprovement in Main Exp A ⇒ B .
Based on this assumption, to identify whether abstract concepts are modularized in some parameters,we partially freeze the model in a coarse-to-fine manner.
The following experiments are conductedon one of the three pre-training terminal sets.
First, we freeze each layer in the model, showing in
Figure 8.
We find that the last layer in encoder and the first layer in decoder modularize part ofabstract concepts.
Furthermore, we freeze these two layers in our Contrast Exp C ⇒ B and find noimprovements in Figure 9a, indicating that the improvement of freezing these two layers comes fromkeep abstract concepts.
In next step, we try to identify the abstract concepts are stored in attention layers or FF layers.
Weseparately freeze the attention sub-layer and FF sub-layer in the last encoder layer and first decoderlayer.
Figure 9b shows that attention layers take more responsibility for storing abstract concepts.
Then, we analyze whether these abstract concepts are modularized in some attention heads or averagedin the whole attention layers.
Our investigation in Section 6.2 finds that the abstract concepts arecentralized in some middle-layer attention heads.
Based on the results in Figure 4, we freeze the top 36 heads to further verify that they are responsible for storing abstract concepts.
Results in Table 7indicate that abstract concepts are modularized in this small part of attention heads.
[Figure]
[Figure]
[Figure]
[Table]
F E XAMPLES IN O UR D ESIGNED T ASKS
Here, we list some examples in our designed probing tasks.
The full sets are contained in our
Supplementary Material.
[-E D 
-]F.
G RAMMAR P ROBE
Table 8 and 9 shows examples in Com.
and
Mod.
sub-probes in grammar probe, respectively.
F. O PERATION P ROBE
Table 10 lists the operation behind each symbol, and Table 11 shows examples in Conj.
sub-probe inoperation probe.
Figure 10 illustrates the transitions of operations used in contrast task in operationprobe.
F. R
For Redundant G ∗ s , we supplement the second T-Production rule in Table 1 as "sub / direct-obj /indirect-obj ↠ S adj S n ".
Terminals in S adj can re regarded as adjectives for S n .
These terminals [Table] [Table] are only contained in the source side and no terminals in target side are aligned with them.
Table 13shows an example in Redundant G ∗ s .
F. D ERIVATIONS
Here we show examples of different derivations of original grammars.
Table 12 shows examples indifferent G ∗ t .
The terminal ’NONE’ is omitted.
G M ORE E XPERIMENTAL R ESULTS
We present additional experimental results to supplement our probing and analysis.
G. D ATA S CALE AND
[Figure]
Figure 11a shows the effects from data scale for different model sizes. It shows that the performanceimproves marginally and unstably when the data scale increases from 1.7K to 680K instances. Moreover, it seems that the performance gap between models with different sizes is still considerablewhen the data scale is enough large.
Figure 11b shows the effects from data diversity for different model sizes.
Here, we consider theterminal diversity as a perspective of data diversity, i.e., the number of terminals of the grammar. Following Section 4, we only change the number of terminals in S c and S C , increasing from 1 to128. The overall trend is that the performance improves marginally and unstably when the diversityincreases.
Interestingly, we observe that for all three models, their performances achieve the peakbefore rising to 128 terminals and then keep oscillating.
We speculate that their performances arebounded by the limited data scale, as we control the data scale as 34K instances when increasingthe terminal diversity.
To examine our speculation, we conduct another experiment on T5-Base thatpre-training on 680K instances with 128 terminals, achieving an average accuracy rate of 93.5%.
This performance is higher than the result after pre-training on 680K instances with 32 terminals(89.2%) and higher than the best average accuracy of T5-Base in Figure 11b (88.4%), suggesting thathigher data diversity should be equipped with a larger data scale.
[Table]
[Table]
G. M ULTI -G RAMMAR P RE -T RAINING
Before this section, we consider the setup in which we only see one pair of input and output grammarsduring pre-training.
This section explores whether multi-grammar pre-training would influencethe model to exhibit abstraction.
Here, we consider two cases: can or can not access the goldengrammar.
The golden grammar is the grammar used in the downstream task.
For the multi-grammarpre-training, we assemble different target grammars in Section 6.3 while keep the source grammar.
During generating pre-training data with more than one target grammar, for each instance, we add aprefix (chosen from original , coarse , localreverse , nest , and reverse ) at the beginning of the sourcetokens, guiding the model which target grammar it should use.
Table 14 shows the results withdifferent ensemble grammars.
First, consider that we have no access to the golden grammar.
We take the target grammar thatperforms the best in Section 6.3, Coarse G ∗ t , as the single-grammar baseline.
Table 14 shows thataugmenting the Coarse G ∗ t with other target grammars can always perform better than the singlegrammar.
Even augmenting with the Reverse G ∗ t from the contrast task can bring a slight gain (1.4%accuracy).
It indicates that even though the model has no access to the golden abstract concepts,increasing the diversity of abstract concepts can make the model better aware of the existenceof abstract concepts .
This awareness can be regarded as a higher level of abstraction ability.
Then, considering that the model has access to the golden grammar, the downstream task performanceis lower than only pre-training on the golden grammar (accuracy 88.2%).
Therefore, augmentingother similar abstract concepts would confuse the model and make it hard to choose which conceptsshould be used for the downstream task.
G. I NCREASE T OTAL N UMBER OF T ERMINALS
We increase the total number of terminals to ∼ 1,500 in our probing tasks and report the results with
T5-Base in Table 15.
These results are similar to the original results in Table 2 and are still in linewith our two expectations.
H F UZZY G RAMMAR
The abstract concepts discussed in grammar probe in Section 4 can be concretely defined, but inmany application scenarios, abstract concepts can be fuzzy (e.g., natural language grammar).
Here,we want to examine whether models can learn fuzzy grammar or just can recognize the concrete one. [Figure] L e ft [Figure]
[Figure]
[Figure]
ConverseImplication [Table] We take natural language sentences for experiments and design different sets by mimicking the Com.sub-probe in our grammar probe.
Data in our fuzzy grammar probe is taken from Europarl v7 Koehn (2005), a large parallel corpusfor machine translation .
For the probing on natural language data, we can not guarantee to satisfythe requirements in our framework perfectly, as the grammar of the natural language is hard to becontrollable as the formal language.
We describe the instantiation of different sets as following.
Train set A .
We take the German-to-French (De-Fr) sentence paris as A .
Transfer set ˆ B and Test set B .
We take English-to-Romanian (En-Ro) as the probing task B .
Asboth German and English are belong to the West Germanic language branch while both Frenchand Romanian are belong to Italic branch, the abstract grammars used in De-Fr and En-Ro havesome similarities.
To satisfy that the ⇑ B performs poorly, we limit the ˆ B with only short sentences(15.7/13.8 words in En/Ro sentences in average) while B with only long sentences (78.0/74.4 wordsin En/Ro sentences in average).
It means that the model can learn most of the lexicons from ˆ B butcan not be aware of the grammars of long sentences.
Contrast set C .
Mimicking the construction of C in the formal language task, we also reverse theword order in the target language of A .
Table 4 shows the performance of two models on natural language data.
These results indicate thatfuzzy grammar in natural language data can also be learned and transferred by the two PLMs.
Inaddition, as this setting can also be regarded as a length generalization problem, the low ∆( C ⇒ B )further confirm that our probing results benefit from learning abstract concepts rather thansurface patterns (i.e., length distribution) .
I T RY TO M EASURE A BSTRACTION C APABILITY
As mentioned in Section A that one limitation in our probing is the lack of a metric that canquantitatively measure the abstraction ability.
Thus, we can not compare the abstraction ability of twomodels with different architectures.
Here, we try to design such a metric to compare the abstractionability of T5-Base, with ∼ 220M parameters, and GPT2-Medium, with 345M parameters.
In the beginning, we want to clarify that this metric is just for primary exploration, as it is based on astrong assumption that can not be satisfied in all situations.
Assumption: We assume that the performance score of a certain task (such as accuracy and BLEUscore) can linearly reflect the ability of the model to solve this task.
It means that, for instance, improving the accuracy from 90% to 100% is not harder than improvingfrom 40% to 50%.
Apparently, this assumption does not suit all tasks and performance scores (even [Figure] the tasks and scores in our probing).
But it does not influence the comparison between T5-Base and
GPT2-Medium.
The reason will be discussed later.
Intuitively, we consider the contribution of abstract concepts to overall performance as a measure ofabstraction ability, that is,
[Equation]
in which the MoA means the M etric
o f A bstraction, score f means the full performance score on acertain task without limiting the training data, and score a means the part of the score contributed bythe abstract concepts.
Following our probing framework, we consider the score a as the relative gainfrom ⇑ B to A ⇒ B .
Furthermore, considering the influence of other factors which is reflected by
C ⇒ B , we design the score a as:
[Equation]
in which score() represents the performance score of a certain procedure, and max[score( ⇑
B ) , score( C ⇒ B )] means to choose the maximum performance score between ⇑ B and C ⇒ B .
For the full performance score in the denominator in Equation 2, we evaluate the model performanceon B after (only) fine-tuning on the full set ˜ B , which is sampled in the same distribution of B (ratherthan a limited distribution of ˆ B ).
We denote this procedure as ˜ ⇑ B .
Thus, the metric in Equation 2can be formalized as:
[Equation]
Table 16 shows MoA for two models on grammar probe and fuzzy grammar probe, and lists the scoresrequired to calculate MoA .
On each task, MoA of T5-Base is higher than that of GPT2-Medium.
Furthermore, during calculating MoA , the baseline score max[score( ⇑ B ) , score( C ⇒ B )] of T5Base is always higher than that of GPT2-Medium.
As it is harder for the model to improve theaccuracy and BLEU scores on these tasks from a relatively higher baseline, MoA can just underestimate the abstract ability of T5-Base.
Therefore, we can roughly conclude that the abstractionability of T5-Base is higher than GPT2-Medium.
J C OMPARISON W ITH P REVIOUS N EGATIVE R ESULTS
Some previous work demonstrated that neural models could not learn abstract concepts (Liu et al. ,2020; Chen et al., 2020; Liu et al., 2021a; Chollet, 2019; Mitchell, 2021; Zadrozny, 2021).
Ourprobing results shed some light that neural models, especially PLMs, exhibit abstraction ability tosome extent.
Compared with previous work, two points could lead to different conclusions. [Table] [Table]
The first point is the probing methodology.
In all works (including ours), the basic idea of probingabstraction is to separate it with memorization.
To implement this idea, previous work has almostalways involved designing a special probing task in which memorization of the train set is helplessto solve the test set.
However, such an implementation constraints the generation of the train set,which could bring some biases or limitations in training data.
To overcome these biases or limitations,the model should have some other abilities more than abstraction, such as reasoning and systematicgeneralizability.
Therefore, the previous disappointing results may have been caused by the lack ofother abilities rather than abstraction.
The second point is the test model.
Some previous work probed the vanilla Transformer or LSTMwhile we take the pre-trained language models.
We suppose that the model may acquire betterabstraction ability from the pre-training corpus, and can better exhibit this ability with larger modelsizes.
K D ETAILS OF E XPERIMENTS
K. D ATA
We show more details about the sets described in Section 4, including data scales, average inputlengths and average output lengths.
For the target side grammar of our formal language tasks, we mentioned in Section 4 that we changethe original target grammar of COGS to be chain-structured.
In Table 18, we list some exampleswith the original target grammar and the new chain-structured grammar.
First, to distinguish theinput and output tokens, we capitalize all output tokens (e.g., from "rose" to "ROSE").
Second, wereplace the variables (e.g., "x _ 1") in the original grammar with its corresponding terminals (e.g.,"ROSE").
Then, we group the terminals of AGENT (e.g., "DOG"), THEME (e.g., "ROSE") and
RECIPIENT with their corresponding terminal of PREDICATE (e.g., "HELP")
and combine thisgroup of terminals in a function format, i.e., "PREDICATE ( AGENT, THEME, RECIPIENT )".
If thepredicate is not equipped with an agent, theme or recipient in the original grammar, the correspondingnew non-terminals (i.e., AGENT, THEME and RECIPIENT, respectively) in the function formatabove will be filled with the terminal NONE (e.g., "HELP ( DOG, ROSE, NONE )").
For simplicity,we omitted NONE in Table 1, Table 8, and Table 12.
Such a function format is the minimum unit of a
CLAUSE.
Finally, each CLAUSE is concatenated with another CLAUSE by the terminal CCOMP(e.g., "HOPE ( LIAM, NONE, NONE ) CCOMP PREFER ( DOG, NONE, NONE )").
K. P ROCEDURE
Training Each pre-training takes 100,000 steps, and the final-step checkpoint is used for fine-tuning. Each fine -tuning
takes
100,000 steps, and the checkpoints for every ,000 steps are saved
[Table] [Table] Evaluation We take an early-stopping strategy in our evaluation to avoid catastrophic forgetting.
First, each checkpoint saved during fine-tuning is evaluated on the held-out dev set.
We choose thefirst checkpoint that achieves the best dev score for testing.
For formal language tasks, we utilize theconstraint decoding strategy that the model can only generate the words in the vocabulary.
Compute and Resources We majorly use Tesla-V100-16GB GPUs for training and evaluation,except for the experiments on T5-Large or GPT2-Large, which require Tesla-V100-32GB GPUs.
Onaverage, one pre-training takes ∼ 15 GPU hours, one fine-tuning takes ∼ 15 GPU hours (includingsaving checkpoints), and one testing takes ∼ 2 GPU hours (as test cases are very long).
K. H YPERPARAMETERS
Hyperparameters used for training and testing are listed in Table 19.
K. D EFINITION OF P ERPLEXITY (PPL)
Th following equation explain how to calculate PPL in Equation 1,
[Equation]
K. R ESULTS
We list the detailed results that are plotted in the figures (i.e., Figure 5 and Figure 6), includingthe average scores, minimum scores, maximum scores, and standard deviations for all replicateexperiments.
[Table]
[Table] [Table]
[Table] [Table] [Table]