A RE P RE - TRAINED L ANGUAGE M ODELS A WARE OF P HRASES ? S IMPLE BUT S TRONG B ASELINES FOR G RAMMAR I NDUCTION
A BSTRACT
With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings.
In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency.
Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training.
In addition, we report intriguing ﬁndings in the induced trees, including the fact that some pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.
I NTRODUCTION
Grammar induction, which is closely related to unsupervised parsing and latent tree learning, allows one to associate syntactic trees, i.e., constituency and dependency trees, with sentences.
As grammar induction essentially assumes no supervision from gold-standard syntactic trees, the existing approaches for this task mainly rely on unsupervised objectives, such as language modeling (Shen et al., 2018b; 2019; Kim et al., 2019a;b) and cloze-style word prediction (Drozdov et al., 2019) to train their task-oriented models.
On the other hand, there is a trend in the natural language processing (NLP) community of leveraging pre-trained language models (LMs), e.g., ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), as a means of acquiring contextualized word representations.
These representations have proven to be surprisingly effective, playing key roles in recent improvements in various models for diverse NLP tasks.
In this paper, inspired by the fact that the training objectives of both the approaches for grammar induction and for training LMs are identical, namely, (masked) language modeling, we investigate whether pre-trained LMs can also be utilized for grammar induction/unsupervised parsing, especially without training.
Speciﬁcally, we focus on extracting constituency trees from pre-trained LMs without ﬁne-tuning or introducing another task-speciﬁc module, at least one of which is usually required in other cases where representations from pre-trained LMs are employed.
This restriction provides us with some advantages: (1) it enables us to derive strong baselines for grammar induction with reduced time and space complexity, offering a chance to reexamine the current status of existing grammar induction methods, (2) it facilitates an analysis on how much and what kind of syntactic information each pre-trained LM contains in its intermediate representations and attention distributions in terms of phrase-structure grammar, and (3) it allows us to easily inject biases into our framework, for instance, to encourage the right-skewness of the induced trees, resulting in performance gains in English unsupervised parsing.
First, we brieﬂy mention related work ( § 2).
Then, we introduce the intuition behind our proposal in detail ( § 3), which is motivated by our observation that we can cluster words in a sentence according to the similarity of their attention distributions over words in the sentence.
Based on this intuition, we deﬁne a straightforward yet effective method ( § 4) of drawing constituency trees directly from pretrained LMs with no ﬁne-tuning or addition of task-speciﬁc parts, instead resorting to the concept of Syntactic Distance (Shen et al., 2018a;b).
Then, we conduct experiments ( § 5) on the induced constituency trees, discovering some intriguing phenomena.
Moreover, we analyze the pre-trained
LMs and constituency trees from various points of view, including looking into which layer(s) of the LMs is considered to be sensitive to phrase information ( § 6).
To summarize, our contributions in this work are as follows:
• By investigating the attention distributions from Transformer-based pre-trained LMs, we show that there is evidence to suggest that several attention heads of the LMs exhibit syntactic structure akin to constituency grammar.
• Inspired by the above observation, we propose a method that facilitates the derivation of constituency trees from pre-trained LMs without training.
We also demonstrate that the induced trees can serve as a strong baseline for English grammar induction.
• We inspect, in view of our framework, what type of syntactic knowledge the pre-trained LMs capture, discovering interesting facts, e.g., that some pre-trained LMs are more aware of adverb phrases than other approaches.
R ELATED W ORK
Grammar induction is a task whose goal is to infer from sequential data grammars which generalize, and are able to account for unseen data (Lari & Young (1990); Clark (2001); Klein & Manning (2002; 2004), to name a few).
Traditionally, this was done by learning explicit grammar rules (e.g., context free rewrite rules), though more recent methods employ neural networks to learn such rules implicitly, focusing more on the induced grammars’ ability to generate or parse sequences.
Speciﬁcally, Shen et al.
(2018b) proposed Parsing-Reading-Predict Network (PRPN) where the concept of Syntactic Distance is ﬁrst introduced.
They devised a neural model for language modeling where the model is encouraged to recognize syntactic structure.
The authors also probed the possibility of inducing constituency trees without access to gold-standard trees by adopting an algorithm that recursively splits a sequence of words into two parts, the split point being determined according to correlated syntactic distances; the point having the biggest distance becomes the ﬁrst target of division.
Shen et al.
(2019) presented a model called Ordered Neurons (ON), which is a revised version of LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber (1997)) which reﬂects the hierarchical biases of natural language and can be used to compute syntactic distances.
Shen et al.
(2018a) trained a supervised parser relying on the concept of syntactic distance.
Other studies include Drozdov et al.
(2019), who trained deep inside-outside recursive autoencoders (DIORA) to derive syntactic trees in an exhaustive way with the aid of the inside-outside algorithm, and Kim et al. (2019a) who proposed Compound Probabilistic Context-Free Grammars (compound
PCFG), showing that neural PCFG models are capable of producing promising unsupervised parsing results.
Li et al.
(2019) proved that an ensemble of unsupervised parsing models can be beneﬁcial, while Shi et al.
(2019) utilized additional training signals from pictures related with input text.
Dyer et al.
(2016) proposed Recurrent Neural Network Grammars (RNNG) for both language modeling and parsing, and Kim et al.
(2019b) suggested an unsupervised variant of the RNNG.
There also exists another line of research on task-speciﬁc latent tree learning (Yogatama et al., 2017; Choi et al., 2018; Havrylov et al., 2019; Maillard et al., 2019).
The goal here is not to construct linguistically plausible trees, but to induce trees ﬁtted to improving target performance.
Naturally, the induced performance-based trees need not resemble linguistically plausible trees, and some studies (Williams et al., 2018a; Nangia & Bowman, 2018) examined the apparent fact that performance-based and lingusitically plausible trees bear little resemblance to one another.
Concerning pre-trained language models (Peters et al.
; Devlin et al.
; Radford et al.
; Yang et al.
; Liu et al.
(2019b), inter alia )—particularly those employing a Transformer architecture (Vaswani et al., 2017)—these have proven to be helpful for diverse NLP downstream tasks.
In spite of this, there is no vivid picture for explaining what particular factors contribute to performance gains, even though some recent work has attempted to shed light on this question.
In detail, one group of studies (Raganato & Tiedemann (2018); Clark et al.
; Ethayarajh (2019);
Hao et al.
; Voita et al.
(2019), inter alia ) has focused on dissecting the intermediate representations and attention distributions of the pre-trained LMs, while the another group of publications (Mareˇcek & Rosa (2018); Goldberg (2019); Hewitt & Manning (2019);
Liu et al.
(2019a) ; Rosa & Mareˇcek (2019), to name a few) delve into the question of the existence of syntactic knowledge in Transformer-based models.
Particularly, Mareˇcek & Rosa (2019) proposed an algorithm for extracting constituency trees from Transformers trained for machine translation, which is similar to our approach.
M OTIVATION
As pioneers in the literature have pointed out, the multi-head self-attention mechanism (Vaswani et al., 2017) is a key component in Transformer-based language models, and it seems this mechanism empowers the models to capture certain semantic and syntactic information existing in natural language.
Among a diverse set of knowledge they may capture, in this work we concentrate on phrase-structure grammar by seeking to extract constituency trees directly from their attention information and intermediate weights.
In preliminary experiments, where we manually visualize and investigate the intermediate representations and attention distributions of several pre-trained LMs given input, we have found some evidence which suggests that the pre-trained LMs exhibit syntactic structure akin to constituency grammar in some degree.
Speciﬁcally, we have noticed some patterns which are often displayed in self-attention heatmaps as explicit horizontal lines, or groups of rectangles of various sizes.
As an attention distribution of a word in an input sentence corresponds to a row in a heatmap matrix, we can say that the appearance of these patterns indicates the existence of groups of words where the attention distributions of the words in the same group are relatively similar.
Interestingly, we have also discovered the fact that the groups of words we observed are fairly correlated with the constituents of the input sentence, as shown in Figure 1 (above) and Figure 3 (in Appendix A.1).
Even though we have identiﬁed some patterns which match with the constituents of sentences, it is not enough to conclude that the pre-trained LMs are aware of syntactic phrases as found in phrasestructure grammars.
To demonstrate the claim, we attempt to obtain constituency trees in a zero-shot learning fashion, relying only on the knowledge from the pre-trained LMs.
To this end, we suggest the following, inspired from our ﬁnding: two words in a sentence are syntactically close to each other (i.e., the two words belong to the same constituent) if their attention distributions over words in the sentence are also close to each other.
Note that this implicitly presumes that each word is more likely to attend more on the words in the same constituent to enrich its representation in the pre-trained LMs.
Finally, we utilize the assumption to compute syntactic distances between each pair of adjacent words in a sentence, from which the corresponding constituency tree can be built.
P ROPOSED M ETHOD 4. S YNTACTIC D ISTANCE AND T REE C ONSTRUCTION
We leverage the concept of Syntactic Distance proposed by Shen et al.
(2018a;b) to draw constituency trees from raw sentences in an intuitive way.
Formally, given a sequence of words in a sentence, w 1 , w 2 , . . . , w n , we compute d =
[ d 1 , d 2 , . . .
, d n − 1 ] where d i corresponds to the syntactic distance between w i and w i +1 .
Each d i is deﬁned as follows:
[Equation]
where f ( · , · ) and g ( · ) are a distance measure function and representation extractor function, respectively.
The function g converts each word into the corresponding vector representation, while f computes the syntactic distance between the two words given their representations.
Once d is derived, it can be easily converted into the target constituency tree by a simple algorithm following
Shen et al.
(2018a).
For details of the algorithm, we refer the reader to Appendix A.2.
Although previous studies attempted to explicitly train the functions f and g with supervision (with access to gold-standard trees, Shen et al. (2018a)) or to obtain them as a by-product of training particular models that are carefully designed to recognize syntactic information (Shen et al., 2018b; 2019), in this work we stick to simple distance metric functions for f and pre-trained LMs for g , forgoing any training process.
In other words, we focus on investigating the possibility of pre-trained
LMs possessing constituency information in a form that can be readily extracted with straightforward computations.
If the trees induced by the syntactic distances derived from the pre-trained LMs are similar enough to gold-standard syntax trees, we can reasonably claim that the LMs resemble phrase-structure.
4. P RE - TRAINED L ANGUAGE M ODELS
We consider four types of recently proposed language models.
These are: BERT (Devlin et al., 2019), GPT-
(Radford et al., 2019), RoBERTa (Liu et al., 2019b), and XLNet (Yang et al., 2019).
They all have in common that they are based on the Transformer architecture and have been proven to be effective in natural language understanding (Wang et al., 2019) or generation.
We handle two variants for each LM, varying in the number of layers, attention heads, and hidden dimensions, resulting in eight different cases in total.
In particular, each LM has two variants.
(1) base: consists of l =12 layers, a =12 attention heads, and d =768 hidden dimensions, while (2) large: has l =24 layers, a =16 attention heads, and d =1024 hidden dimensions.
2 We deal with a wide range of pre-trained
LMs, unlike previous work which has mostly analyzed a speciﬁc model, particularly BERT.
For details about each LM, we refer readers to the respective original papers.
In terms of our formulation, each LM instance provides two categories of representation extractor functions, G v and G d .
Speciﬁcally, G v refers to a set of functions { g vj | j = 1 , . . .
, l } , each of which simply outputs the intermediate hidden representation of a given word on the j th layer of the LM.
Likewise, G d is a set of functions { g d ( j,k ) | j = 1 , . . .
, l, k = 1 , . . .
, a + 1 } , each of which outputs the attention distribution of an input word by the k th attention head on the j th layer of the LM.
Even though our main motivation comes from the self-attention mechanism, we also deal with the intermediate hidden representations present in the pre-trained LMs by introducing G v , considering that the hidden representations serve as storage of collective information taken from the processing of the pre-trained LMs.
Note that k ranges up to a + 1 , not a , implying that we consider the average of all attention distributions on the same layer in addition to the individual ones.
This averaging function can be regarded as an ensemble of other functions in the layer which are specialized for different aspects of information, and we expect that this technique will provide a better option in some cases as reported in previous work (Li et al., 2019).
One remaining issue is that all the pre-trained LMs we use regard each input sentence as a sequence of subword tokens, while our formulation assumes words cannot be further divided into smaller tokens.
To resolve this difference, we tested certain heuristics that guide how subword tokens for a complete word should be exploited to represent the word, and we have empirically found that the best result comes when each word is represented by an average of the representations of its subwords.
Therefore, we adopt the above heuristic in this work for cases where a word is tokenized into more than two parts.
4. D ISTANCE M EASURE F UNCTIONS
For the distance measure function f , we prepare three options ( F v ) for G v and two options ( F d ) for
G d .
Formally, f ∈ F v ∪ F d , where F v = { COS , L1 , L2 } , F d = { JSD , HEL } .
COS, L1, L2, JSD, and HEL correspond to Cosine, L1, and L2, Jensen-Shannon, and Hellinger distance respectively.
The functions in F v are only compatible with the elements of G v , and the same holds for F d and
G d .
The exact deﬁnition of each function is listed in Appendix A.3.
4. I NJECTING B IAS INTO S YNTACTIC D ISTANCES
One of the main advantages we obtain by leveraging syntactic distances to derive parse trees is that we can easily inject inductive bias into our framework by simply modifying the values of the syntactic distances.
Hence, we investigate whether the extracted trees from our method can be further reﬁned with the aid of additional biases.
To this end, we introduce a well-known bias for English constituency trees—the right-skewness bias—in a simple linear form.
Namely, our intention is to inﬂuence the induced trees such that they are moderately right-skewed following the nature of gold-standard parse trees in English.
Formally, we compute ˆ
d i by appending the following linear bias term to every d i :
[Equation]
where AVG ( · ) outputs an average of all elements in a vector, λ is a hyperparameter, and i ranges from 1 to m = n − 1 .
We write ˆd =
[ ˆ d 1 , ˆ d 2 , . .
, ˆ d m ] in place of d to signify biased syntactic distances.
The main purpose of introducing such a bias is examining what changes are made to the resulting tree structures rather than boosting quantitative performance per se , though it is of note that it serves this purpose as well.
We believe that this additional consideration is necessary based on two points.
First, English is what is known as a head-initial language.
That is, given a selector and argument, the selector has a strong tendency to appear on the left, e.g., ‘eat food’, or ‘to Canada’.
Head-initial languages therefore have an in-built preference for right-branching structures.
By adjusting the bias injected into syntactic distances derived from pre-trained LMs, we can ﬁgure out whether the LMs are capable of inducing the right-branching bias, which is one of the main properties of English syntax; if injecting the bias does not inﬂuence the performance of the LMs on unsupervised parsing, we can conjecture they are inherently capturing the bias to some extent.
Second, as mentioned before, we have witnessed some previous work (Shen et al., 2018b; 2019; Htut et al., 2018; Li et al., 2019; Shi et al., 2019) where the right-skewness bias is implicitly exploited, although it could be regarded as not ideal.
What we intend to focus on is the question about which beneﬁts the bias provides for such parsing models, leading to overall performance improvements.
In other words, we look for what the exact contribution of the bias is when it is injected into grammar induction models, by explicitly controlling the bias using our framework.
E XPERIMENTS 5. G ENERAL S ETTINGS 5.1. D ATASETS
In this section, we conduct unsupervised constituency parsing on two datasets.
The ﬁrst dataset is
WSJ Penn Treebank (PTB, Marcus et al.
(1993)), in which human-annotated gold-standard trees are available.
We use the standard split of the dataset—2-21 for training, 22 for validation, and 23 for test.
The second one is MNLI (Williams et al., 2018b), which is originally designed to test natural language inference but often utilized as a means of evaluating parsers.
It contains constituency trees produced by an external parser (Klein & Manning, 2003).
We leverage the union of two different versions of the MNLI development set as test data following convention (Htut et al., 2018; Drozdov et al., 2019), and we call it the MNLI test set in this paper.
Moreover, we randomly sample 40K sentences from the training set of the MNLI to utilize them as a validation set.
To preprocess the datasets, we follow the setting of Kim et al. (2019a) with the minor exceptions that words are not lower-cased and number characters are preserved instead of being substituted by a special character.
5.1. I MPLEMENTATION D ETAILS
For implementation, to compare pre-trained LMs in an uniﬁed manner, we resort to an integrated
PyTorch codebase that supports all the models we consider.
For each LM, we tune the best combination of f and g functions using the validation set.
Then, we derive a set of d for sentences in the test set using the chosen functions, followed by the resulting constituency trees converted from each d by the tree construction algorithm in Section 4.1.
In addition to sentence-level F1 (S-F1) score, we report label recall scores for six main categories: SBAR, NP, VP, PP, ADJP, and ADVP.
We also present the results of utilizing ˆ d instead of d , empirically setting the bias hyperparameter λ as 1.5.
We do not ﬁne-tune the LMs on domain-speciﬁc data, as we here focus on ﬁnding their universal characteristics.
We take four na ¨ ıve baselines into account, random (averaged over 5 trials), balanced, left-branching, and right-branching binary trees.
In addition, we present two more baselines which are identical to our models except that their g functions are based on a randomly initialized XLNet-base rather than pre-trained ones.
To be concrete, We provide ‘Random XLNet-base ( F v )’ which applies the functions in F v on random hidden representations and ‘Random XLNet-base ( F d )’ that utilizes the functions in F d and random attention distributions, respectively.
Considering the randomness of initialization and possible choices for f , the ﬁnal score for each of the baselines is calculated as an average over 5 trials of each possible f , i.e., an average over 5 × 3 runs in case of F v and 5 ×runs for F d .
These baselines enable us to estimate the exact advantage we obtain by pre-training
LMs, effectively removing additional unexpected gains that may exist.
Furthermore, we compare our parse trees against ones from existing grammar induction models.
All scripts used in our experiments will be publicly available for reproduction and further analysis.
5. E XPERIMENTAL R ESULTS ON PTB
In Table 1, we report the results of the various models on the PTB test set.
First of all, our method combined with pre-trained LMs shows competitive or comparable results in terms of S-F1 even without the right-skewness bias.
This result implies that the extracted trees from our method can be regarded as a baseline for English grammar induction.
Moreover, pre-trained LMs show substantial improvements over Random Transformers (XLNet-base), demonstrating that training language models on large corpora, in fact, enables the LMs to be more aware of syntactic information.
When the right-skewness bias is applied to syntactic distances derived from pre-trained LMs, the
S-F1 scores of the LMs increase by up to ten percentage points.
This improvement indicates that the pre-trained LMs do not properly capture the largely right-branching nature of English syntax, at least when observed through the lens of our framework.
By explicitly controlling the bias through our framework and observing the performance gap between our models with and without the bias,
L
[Table]
layer number,
[Table]
: attention head number (AVG: the average of all attentions).
† : Results reported by Kim et al. (2019a).
Approaches in which COO parser is utilized
[Table]
we conﬁrm that the main contribution of the bias comes from its capability to capture subordinate clauses (SBAR) and verb phrases (VP).
This observation provides a hint for what some previous work on unsupervised parsing desired to obtain by introducing the bias to their models.
It is intriguing to see that all of the existing grammar induction models are inferior to the right-branching baseline in recognizing SBAR and VP (although some of them already utilized the right-skewness bias), implying that the same problem—models do not properly capture the right-branching nature— may also exist in current grammar induction models.
One possible assumption is that the models do not need the bias to perform well in language modeling, although future work should provide a rigorous analysis about the phenomenon.
On the other hand, the existing models show exceptionally high recall scores on noun phrases (NP), even though our pre-trained LMs also have success to some extent in capturing noun phrases compared to na¨ıve baselines.
From this, we conjecture that neural models trained with a language modeling objective become largely equipped with the ability to understand the concept of NP.
In contrast, the pre-trained LMs record the best recall scores on adjective and adverb phrases (ADJP and ADVP), suggesting that the LMs and existing models capture disparate aspects of English syntax to differing degrees.
To further explain why some pre-trained LMs are good at capturing ADJPs and ADVPs, we manually investigated the attention heatmaps of the sentences that contain ADJPs or ADVPs.
From the inspection, we empirically found that there are some keywords—including ‘two’, ‘ago’, ‘too’, and ‘far’—which have different patterns of attention distributions compared to those of their neighbors and that these keywords can be a clue for our framework to recognize the existence of ADJPs or ADJPs.
It is also worth mentioning that ADJPs and ADVPs consist of a relatively smaller number of words than those of SBAR and VP, indicating that the LMs combined with our method have strength in correctly ﬁnding small chunks of words, i.e., low-level phrases.
Meanwhile, in comparison with other LM models, GPT-2 and XLNet based models demonstrate their effectiveness and robustness in unsupervised parsing.
Particularly, the XLNet-base model serves as a robust baseline achieving the top performance among LM candidates.
One plausible explanation for this outcome is that the training objective of XLNet, which considers both autoencoding (AE) and autoregressive (AR) features, might encourage the model to be better aware of
[Figure]
phrase structure than other LMs.
Another possible hypothesis is that AR objective functions (e.g., typical language modeling) are more effective in training syntax-aware neural models than AE objectives (e.g., masked language modeling), as both GPT-2 and XLNet are pre-trained on AR variants.
However, it is hard to conclude what factors contribute to their high performance at this stage.
Interestingly, there is an obvious trend that the functions in F d —the distance measure functions for attention distributions—lead most of the LM instances to the best parsing results, indicating that deriving parse trees from attention information can be more compact and efﬁcient than extracting them from the LMs’ intermediate representations, which should contain linguistic knowledge beyond phrase structure.
In addition, the results in Table 1 show that large parameterizations of the
LMs generally increase their parsing performance, but this improvement is not always guaranteed.
Meanwhile, as we expected in Section 4.2 and as seen in the ‘A’ (attention head number) column of Table 1, the average of attention distributions in the same layer often provides better results than individual attention distributions.
5. E XPERIMENTAL R ESULTS ON MNLI
We present the results of various models on the MNLI test set in Table 3 of Appendix A.5.
We observe trends in the results which mainly coincide with those of the PTB dataset.
Particularly, (1) right-branching trees are strong baselines for the task, especially showing their strengths in capturing SBAR and VP clauses/phrases, (2) our method resorting to the LM instances is also comparable to the right-branching trees, demonstrating its superiority in recognizing different aspects of phrase categories including prepositional phrases (PP) and adverb phrases (ADVP), and (3) attention distributions seem more effective for distilling the phrase structures of sentences than intermediate representations.
However, there are some issues worth mentioning.
First, the right-branching baseline seems to be even stronger in the case of MNLI, recording a score of over 50 in sentence-level F1.
We conjecture that this result comes principally from two reasons: (1) the average length of sentences in MNLI is much shorter than in PTB, giving a disproportionate advantage to na¨ıve baselines, and (2) our data preprocessing, which follows Kim et al. (2019a), removes all punctuation marks, unlike previous work (Htut et al., 2018; Drozdov et al., 2019), leading to an unexpected advantage for the rightbranching scheme.
Moreover, it deserves to consider the fact that the gold-standard parse trees in
MNLI are not human-annotated, rather automatically generated.
Second, in terms of consistency in identifying the best choice of f and g for each LM, we observe that most of the best combinations of f and g tuned for PTB do not correspond well to the best ones for MNLI.
Does this observation imply that a speciﬁc combination of these functions and the resulting performance do not generalize well across different data domains?
To clarify, we manually investigated the performance of some combinations of f and g , which are tuned on PTB but tested on MNLI instead.
As a result, we discover that particular combinations of f and g which are good at PTB are also competitive on MNLI, even though they fail to record the best scores on MNLI.
[Table]
Concretely
[Table]
the union of f d (JSD) and g d
[Table]
13) —the best duo for the XLNet-base on PTB—achieves 39.2 in sentence-level F1 on MNLI, which is very close to the top performance (39.3) we can obtain when leveraging the XLNet-base.
It is also worth noting that GPT-2 and XLNet are efﬁcient in capturing PP and ADVP respectively, regardless of the data domain and the choice of f and g
[Table]
F URTHER A NALYSIS 6. PERFORMANCE COMPARISON BY LAYER
To take a closer look at how different the layers of the pre-trained LMs are in terms of parsing performance, we retrieve the best sentence-level F1 scores from the l th layer of an LM from all combinations of f and g l , with regard to the PTB and MNLI respectively.
Then we plot the scores as graphs in Figure 2 for the PTB and Figure 4 in Appendix A.4 for the MNLI.
Each score is from the models to which the bias is not applied.
From the graphs, we observe several patterns.
First, XLNet-based models outperform other competitors across most of the layers.
Second, the best outcomes are largely shown in the middle layers of the LMs akin to the observation from Shen et al.
(2019), except for some cases where the ﬁrst layers (especially in case of MNLI) record the best.
Interestingly, GPT-2 shows a decreasing trend in its output values as the layer becomes high, while other models generally exhibit the opposite pattern.
Moreover, we discover from raw statistics that regardless of the choice of f and g l , the parsing performance reported as S-F1 is moderately correlated with the layer number l .
In other words, it seems that there are some particular layers in the LMs which are more sensitive to syntactic information.
6. E STIMATING THE UPPER LIMIT OF DISTANCE MEASURE FUNCTIONS
Although we have introduced effective candidates for f , we explore the potential of extracting more sophisticated trees from pre-trained LMs, supposing we are equipped with a pseudo-optimum f , call it f ideal .
To obtain f ideal , we train a simple linear layer on each layer of the pre-trained LMs with supervision from the gold-standard trees of the PTB training set, while g remains unchanged—the pre-trained LMs are frozen during training.
We choose the XLNet-base model as a representative for the pre-trained LMs.
For more details about experimental settings, refer to Appendix A.6.
In Table 2, we present three new results using f ideal .
As a baseline, we report the performance of f ideal with a randomly initialized XLNet-base.
Then, we list the worst and best result of f ideal according to g , when it is combined with the pre-trained LM.
We here mention some ﬁndings from the experiment.
First, comparing the results with the pre-trained LM against one with the random
LM, we reconﬁrm that pre-training an LM apparently enables the model to capture some aspects of grammar.
Speciﬁcally, our method is comparable to the linear model trained on the gold-standard trees.
Second, we ﬁnd that there is a tendency for the performance of f ideal relying on different LM layers to follow one we already observed in Section 6.1—the best result comes from the middle layers of the LM while the worst from the ﬁrst and last layer.
Third, we identify that the LM has a potential to show improved performance on grammar induction by adopting a more sophisticated f .
However, we emphasize that our method equipped with a simple f without gold-standard trees is remarkably reasonable in recognizing constituency grammar, being especially good at catching
ADJP and ADVP.
6. C ONSTITUENCY T REE E XAMPLES
We visualize several gold-standard trees from PTB and the corresponding tree predictions for comparison.
For more details, we refer readers to Appendix A.7.
C ONCLUSIONS AND F UTURE W ORK
In this paper, we propose a simple but effective method of inducing constituency trees from pretrained language models in a zero-shot learning fashion.
Furthermore, we report a set of intuitive ﬁndings observed from the extracted trees, demonstrating that the pre-trained LMs exhibit some properties similar to constituency grammar.
In addition, we show that our method can serve as a strong baseline for English grammar induction when combined with (or even without) appropriate linguistic biases.
On the other hand, there are still remaining issues that can be good starting points for future work.
First, although we analyzed our method based on two popular datasets, we focused only on English grammar induction.
As each language has its own properties (and correspondingly would need individualized biases), it is desirable to expand this work to other languages.
Second, it would also be desirable to investigate whether further improvements can be achieved by directly grafting the pre-trained LMs onto existing grammar induction models.
Lastly, by verifying the usefulness of the knowledge from the pre-trained LMs and linguistic biases for grammar induction, we want to point out that there is still much room for improvement in the existing grammar induction models.
A CKNOWLEDGMENTS
We would like to thank Reinald Kim Amplayo and the anonymous reviewers for their thoughtful and valuable comments.
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF2016M3C4A7952587).
A. T REE C ONSTRUCTION A LGORITHM WITH S YNTACTIC D ISTANCES
[Table]
A. D ISTANCE M EASURE F UNCTIONS
Function ( f ) Deﬁnition
Functions for intermediate representations ( F v )
[Equation]
L
[Equation] [Equation]
Functions for attention distributions ( F d )
JSD ( P (cid:107) Q ) (( D KL ( P (cid:107) M ) + D KL ( Q (cid:107) M )) / 2)where M
= ( P + Q ) /and
D KL ( A (cid:107) B ) =
(cid:80) w ∈ S
A ( w ) log( A ( w ) /B
( w ))
HEL ( P, Q )
[Figure]
A. P ERFORMANCE C OMPARISON BY LAYER ON MNLI
[Figure]
A. E XPERIMENTAL R ESULTS ON MNLI
[Table]
A. E XPERIMENTAL DETAILS FOR TRAINING IDEAL DISTANCE MEASURE FUNCTION
In this part, we present the detailed speciﬁcations of the experiments introduced in Section 6.2.
We assume f ideal is only compatible with the functions in G v , as the functions in G d are not suitable for training as the sizes of the representations provided by G d are variable according to the length of an input sentence.
To train the pseudo-optimal function f ideal , we minimize a pair-wise learning-to-rank loss following previous work (Burges et al., 2005;
Shen et al., 2018a):
[Equation]
where d gold and d pred are computed from the gold tree and our predicted one, respectively.
[ x ] + is deﬁned as max (0 , x ) .
We train the f ideal with the PTB training set for 5 epochs.
Each batch of the training set contains 16 sentences.
We use an ADAM optimizer (Kingma & Ba, 2014) with the learning rate 5e-4.
We train the variations of f ideal differentiated by the choice of g in G v and report the best result in the Table 2.
Each f ideal is chosen based on its performance on the PTB validation set.
Considering the randomness of training, every result for f ideal is averaged over 3 different trials.
A. C ONSTITUENCY T REE E XAMPLES
We randomly select six sentences from PTB and visualize their trees, where the resulting group of trees for each sentence consists of a gold constituency tree and two induced trees (one without the right-skewness bias and the other with the bias) from our best model—XLNet-base.
The ‘T’ character in the induced trees indicates a dummy tag.
S
NP-SBJ
DT
These
VP
VBP include
NP NP
DT a
NN child-care
NN initiative
CC and
NP NP
NNS extensions
PP
IN of
NP NP
JJ soon-to-expire
NN tax
NNS breaks
PP
IN for
NP NP
JJ low-income
NN housing
CC and
NP
NN research-and-development
NNS expenditures
T
T T
T T
These
T include
T a
T
T child-care
T initiative
T
T T
T and
T T
T
T extensions
T of
T soon-to-expire
T
T tax
T breaks
T
T for
T
T low-income
T housing
T
T and
T
T research-and-development
T expenditures
T
T T
These
T include
T T
T a
T
T child-care
T initiative
T
T and
T T
T extensions
T of
T
T soon-to-expire
T
T
T tax
T breaks
T T
T for
T
T low-income
T housing
T
T and
T
T research-and-development
T expenditures
S
CC
But
NP-SBJ
NP
NNP
HOFI
POS 's
JJ first
NN offer
VP
MD
would
VP
VB have
VP
VBN given
NP NP
NNP
Ideal
POS 's
JJ other
NNS shareholders
NP NP
QP
RB about
CD
NN %
PP
IN of
NP
DT the
JJ combined
NN company
T
T T
But
T T
T
HOFI
T
T 's
T
T first
T offer
T
T T
would
T have
T given
T T
T
Ideal
T
T 's
T
T other
T shareholders
T T
T about
T
T T
T %
T of
T
T the
T
T combined
T company
T T
T
But
T T
HOFI
T T
T 's
T
T first
T offer
T
T T
would
T have
T given
T T
T
Ideal
T
T 's
T
T other
T shareholders
T
T about
T
T T
T %
T
T of
T
T the
T
T combined
T company
S
S
NP-SBJ
PRP
It
VP
VBD was
NP-PRD
NNP
Friday
DT the
NNP 13th
CC and
S
NP-SBJ
DT the
NN stock
NN market
VP
VBD plummeted
NP-EXT
QP
RB nearly
CD
NNS points
T T
T T
It
T was
T T
Friday
T
T the
T 13th
T T
T and
T
T the
T
T stock
T market
T
T plummeted
T
T nearly
T T
T points
T
T T
It
T
T was
T T
Friday
T
T the
T 13th
T
T and
T T
T the
T
T stock
T market
T
T plummeted
T
T nearly
T T
T points
S
PP-TMP
IN
Until
ADVP
RB recently
NP-SBJ
NP
JJ national
NNS governments
PP-LOC
IN in
NP
NNP
Europe
VP VP
VBD controlled
NP NP
JJS most
PP
IN of
NP
DT the
NN air
NN time
CC and
VP
VBD allowed
NP
ADJP
JJ little
CC or
DT no
NN advertising
T
T T
T
Until
T recently
T
T national
T
T governments
T
T in
T
Europe
T T
T controlled
T T
T
T most
T
T of T the
T air
T time
T T
T and
T allowed
T T
T little
T
T or
T no
T advertising
T T
T
Until
T recently
T
T T
T national
T governments
T
T in
T
Europe
T
T controlled
T T
T most
T
T of
T
T the
T
T air
T time
T T
T and
T allowed
T
T little
T
T or
T
T no
T advertising
S
ADVP
RB
Nevertheless
NP-SBJ
NNP
Ms.
NNP
Garzarelli
VP
VBD said
SBAR
S
NP-SBJPRP she
VP
VBD was
VP
VBN swamped
PP-CLR
IN with
NP NP
NN phone
NNS calls
PP-TMP
IN over
NP
DT the
NN weekend
PPIN from
NP
JJ nervous
NNS shareholders
T
T T
Nevertheless
T T
Ms.
T
Garzarelli
T T
T said
T she
T T
T T
T
T was
T swamped
T with
T
T phone
T calls
T T
T over
T the
T weekend
T
T from
T
T nervous
T shareholders
T T
Nevertheless
T
T T
Ms.
T
Garzarelli
T
T said
T
T she
T T
T was
T swamped
T
T with
T T
T phone
T calls
T
T T
T over
T the
T weekend
T
T from
T
T nervous
T shareholders
S
NP-SBJ
NNS
Analysts
CC and
NNS competitors
ADVP
RB however
VP
VBP doubt
SBAR
S
NP-SBJ
DT the
NNS numbers
VP
VBD were
ADJP-PRD
RB that
JJ high
T
T T
Analysts
T
T and
T competitors
T T
T however
T doubt
T
T T
T the
T numbers
T were
T
T that
T high
T T
T
Analysts
T
T and
T competitors
T T
T however
T doubt
T T
T the
T numbers
T
T were
T
T that
T high