Original: T RACKING THE PROGRESS OF LANGUAGE MODELS BY EXTRACTING THEIR UNDERLYING KNOWLEDGE GRAPHS
Revised: T RACKING THE PROGRESS OF LANGUAGE MODELS BY EXTRACTING THEIR UNDERLYING KNOWLEDGE GRAPHS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The state of the art, previously dominated by pre-trained word embeddings, is now being pushed forward by large pre-trained contextual representation models.
Revised: The state of the art of language models, previously dominated by pre-trained word embeddings, is now being pushed forward by large pre-trained contextual representations.

Original: This success has driven growing interest to understand what these models encode inside their inner workings.
Revised: This success has driven growing interest to understand what these models encode inside their inner workings.

Original: Despite this, understanding their semantic skills has been elusive, often leading to unsuccessful, non-conclusive, or contradictory results among different works.
Revised: Despite this, understanding their semantic skills has been elusive, often leading to unsuccessful, non-conclusive, or contradictory results among different works.

Original: In this work, we deﬁne a probing classiﬁer that wethen use to extract the underlying knowledge graph of nine of the  most inﬂuential models of the last years, including word embeddings, context encoders, and text generators.
Revised: In this work, we deﬁne a probing classiﬁer that we use to extract the underlying knowledge graph of nine of the currently most inﬂuential language models, including word embeddings, context encoders, and text generators.

Original: This probe is based on concept relatedness, grounded on WordNet.
Revised: This probe is based on concept relatedness, grounded on WordNet.

Original: Our results show that this knowledge is present in all the models, but has several inaccuracies.
Revised: Our results show that this knowledge is present in all the models, but has several inaccuracies.

Original: Furthermore, we show that the different pre-training strategies and architectures lead to different model biases.
Revised: Furthermore, we show that the different pre-training strategies and architectures lead to different model biases.

Original: We conduct a systematic evaluation to discover speciﬁc factors that explain why some concepts are challenging for the different families of models.
Revised: We conduct a systematic evaluation to discover speciﬁc factors that explain why some concepts are challenging for the different families of models.

Original: We hope our insights will motivate the future development of models that capture concepts more precisely.
Revised: We hope our insights will motivate the future development of models that capture concepts more precisely.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural language processing (NLP) encompasses a wide variety of applications such as summarization (Kovaleva et al., 2019), information retrieval (Zhan et al., 2020), and machine translation (Tang et al., 2018), among others.
Revised: Natural language processing (NLP) encompasses a wide variety of applications such as summarization (Kovaleva et al., 2019), information retrieval (Zhan et al., 2020), and machine translation (Tang et al., 2018), among others.

Original: Currently, the use of pre-trained language models has become the de facto starting point to tackle most of these applications.
Revised: Currently, the use of pre-trained language models has become the de facto starting point to tackle most of these applications.

Original: The usual pipeline consists of ﬁnetuning a pre-trained language model by using a discriminative learning objective to adapt the model to the requirements of each speciﬁc task.
Revised: The usual pipeline consists of ﬁnetuning a pre-trained language model by using a discriminative learning objective to adapt the model to the requirements of each speciﬁc task.

Original: Asa key ingredient, these models are pre-trained using massive amounts of unlabeled data that can include millions of documents. Furthermore, current models have a large size that can include billions of parameters.
Revised: As key ingredients, these models are pre-trained using massive amounts of unlabeled data that can include millions of documents, and may include billions of parameters.

Original: Massive data and parameters are supplemented with a suitable learning architecture, resulting in a highly powerful but also complex model, whose internal operation is hard to analyze.
Revised: Massive data and parameters are supplemented with a suitable learning architecture, resulting in a highly powerful but also complex model, whose internal operation is hard to analyze.

Original: The success of pre-trained language model has driven the interest to understand the mechanisms that they use to solve NLP tasks.
Revised: The success of pre-trained language models has driven the interest to understand how they manage to solve NLP tasks.

Original: As an example, in the case of BERT (Devlin et al., 2019), one of the most popular pre-trained models based on a Transformer architecture (Vaswani et al., 2017), several studies have attempted to access the knowledge encoded in its layers and attention heads (Tenney et al., 2019b;
Revised: As an example, in the case of BERT (Devlin et al., 2019), one of the most popular pre-trained models based on a Transformer architecture (Vaswani et al., 2017), several studies have attempted to access the knowledge encoded in its layers and attention heads (Tenney et al., 2019b;

Original: Devlin et al., 2019; Hewitt & Manning, 2019).
Revised: Devlin et al., 2019; Hewitt & Manning, 2019).

Original: In particular, (Jawahar et al., 2019) show that BERT can solve tasks at a syntactic level by using Transformer blocks to encode a soft hierarchy of features at different levels of abstraction.
Revised: In particular, (Jawahar et al., 2019) shows that BERT can solve tasks at a syntactic level by using Transformer blocks to encode a soft hierarchy of features at different levels of abstraction.

Original: Similarly, (Hewitt & Manning, 2019) show that BERT is capable of encoding structural information from text.
Revised: Similarly, (Hewitt & Manning, 2019) shows that BERT is capable of encoding structural information from text.

Original: In particular, using a structural probe, they show that syntax trees are embedded in a linear transformation of the encodings provided by BERT.
Revised: In particular, using a structural probe, they show that syntax trees are embedded in a linear transformation of the encodings provided by BERT.

Original: In general, previous efforts have provided strong evidence indicating that current pre-trained language models encode complex syntactic rules, however, relevant evidence about their abilities to capture semantic information remains still elusive.
Revised: In general, previous efforts have provided strong evidence indicating that current pre-trained language models encode complex syntactic rules, however, relevant evidence about their abilities to capture semantic information remains still elusive.

Original: As an example, a recent study (Si et al., 2019) attempts to locate the encoding of semantic information as part of the top layers of Transformer architectures, however, results provide contradictory evidence.
Revised: As an example, a recent study (Si et al., 2019) attempts to locate the encoding of semantic information as part of the top layers of Transformer architectures, however, results provide contradictory evidence.

Original: Similarly, (Kovaleva et al., 2019) focus on studying knowledge encoded by self-attention weights, however, results provide evidence for over-parameterization but not about language understanding capabilities.
Revised: Similarly, (Kovaleva et al., 2019) focuses on studying knowledge encoded by self-attention weights, however, results provide evidence for over-parameterization but not about language understanding capabilities.

Original: In this work, we address the challenging question about to what extent pre-trained language models encode semantic information.
Revised: In this work, we study to which extent pre-trained language models encode semantic information.

Original: As a key source of semantic knowledge, we focus on studying to what extend pre-trained language modelscan encode the semantic embedded in the conceptual taxonomy of WordNet  (Miller, 1995).
Revised: As a key source of semantic knowledge, we focus on studying how precisely pre-trained language models encode the concept relations embedded in the conceptual taxonomy of WordNet 1 (Miller, 1995).

Original: The ability to understand, organize, and correctly use concepts is one of the most remarkable capacities of human intelligence (Lake et al., 2017).
Revised: The ability to understand, organize, and correctly use concepts is one of the most remarkable capabilities of human intelligence (Lake et al., 2017).

Original: Therefore, a quantiﬁcation of the ability that a pre-trained language model can exhibit to encode the conceptual organization behind WordNet is highly valuable.
Revised: Therefore, a quantiﬁcation of the ability that a pre-trained language model can exhibit to encode the conceptual organization behind WordNet is highly valuable.

Original: In particular, it can provide useful insights about the inner mechanisms that these models use to encode semantic information.
Revised: In particular, it can provide useful insights about the inner mechanisms that these models use to encode semantic information.

Original: Furthermore, an analysis of concepts and associations that result difﬁcult to these models can provide relevant insights about how to improve them.
Revised: Furthermore, an analysis of concepts and associations that result difﬁcult to these models can provide relevant insights about how to improve them.

Original: In contrast to most previous works, we do not focus on a particular model, but we embrace our research in general terms by considering a large list of the most popular pre-trained language and text-embedding models.
Revised: In contrast to most previous works, we do not focus on a particular model, but target a large list of the most popular pre-trained language and text-embedding models.

Original: In this sense, one of our goals is to provide a comparative analysis of the capacities of different types of approaches.
Revised: In this sense, one of our goals is to provide a comparative analysis of the capacities of different types of approaches.

Original: Following Hewitt & Manning (2019), we study semantic performance by deﬁning a probing classiﬁer based on concept relatedness according to WordNet.
Revised: Following Hewitt & Manning (2019), we study semantic performance by deﬁning a probing classiﬁer based on concept relatedness according to WordNet.

Original: Using this tool, we analyze the different models, enlightening how and where semantic knowledge is encoded.
Revised: Using this tool, we analyze the different models, enlightening how and where semantic knowledge is encoded.

Original: Furthermore, we analyze how these models encode suitable information to recreate the structure of WordNet.
Revised: Furthermore, we explore how these models encode suitable information to recreate the structure of WordNet.

Original: Among our main results, we show that the different pre-training strategies and architectures lead to different model biases.
Revised: Among our main results, we show that the different pre-training strategies and architectures lead to different model biases.

Original: In particular, we show that contextualized word embeddings, such as BERT, encode high-level concepts and hierarchical relationships among them, creating a taxonomy.
Revised: In particular, we show that contextualized word embeddings, such as BERT, encode high-level concepts and hierarchical relationships among them, creating a taxonomy.

Original: This ﬁnding corroboratesresults presented in previous work  (Reif et al., 2019) that claim that BERT vectors are stored inlinear sub-spaces that have correspondence with semantic knowledge.
Revised: This ﬁnding corroborates previous work results (Reif et al., 2019) that claim that BERT vectors are stored in sub-spaces that have correspondence with semantic knowledge.

Original: On the other hand, our results show that all the analyzed models have a high correlation with each other in terms of the concepts that are hard or easy to capture.

Original: Our study also shows evidence about the limitations of current pre-trained language models, demonstrating that  allof them have difﬁculties to encode speciﬁc concepts.
Revised: Our study also shows evidence about the limitations of current pre-trained language models, demonstrating that they all have difﬁculties encoding speciﬁc concepts.

Original: As an example,across all the models of the family of BERT, performance degrades as the predicted concept or relation is closer to the leaves of the WordNet taxonomy.
Revised: As an example, all the models struggle with concepts related to “taxonomical groups”, performing worse than chance in some cases.

Original: Our results also reveal that models have very distinctive patterns in terms of where they encode most of the semantic information.
Revised: Our results also reveal that models have very distinctive patterns in terms of where they encode most of the semantic information.

Original: These patterns are dependant on architecture and not on model sizes.
Revised: These patterns are dependant on architecture and not on model sizes.

Original: This work is organized as follows.

Original: Section 3 introduces the methodology used to conduct this study, introducing the probing classiﬁer, and specifying the method used to reconstruct the WordNet taxonomy.

Original: Section 4 shows the results obtained using the probing classiﬁer, discussing how accurate the semantic knowledge of each model is and showing the results of the reconstruction of WordNet.

Original: Section 5, uncovers which instances of our tests are more difﬁcult to solve for each of these models, revealing their main limitations.

Original: Section 6 reveals which blocks of these architectures have a correspondence to the semantic tasks of the study.

Original: Finally, in Section 7 we discuss the implications of our ﬁndings along with concluding remarks.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The success of deep learning architectures in various NLP tasks has fueled a growing interest from the community to improve understanding of what these models encode.
Revised: The success of deep learning architectures in various NLP tasks has fueled a growing interest from the community to improve understanding of what these models encode.

Original: Several works have studied these models’ impact on downstream tasks at the syntactic(Goldberg, 2019; Hewitt & Manning, 2019) or semantic level (Ettinger, 2020).
Revised: Several works have studied these models’ impact on downstream tasks at the syntactic or semantic level.

Original: Some studies (Tenney et al., 2019b) claim that success in a speciﬁc task helps understand what type of information the model encodes.
Revised: Some studies (Tenney et al., 2019b) claim that success in a speciﬁc task helps understand what type of information the model encodes.

Original: Other studies have improved the understanding of how and why these models work by analyzing correlations between input-targets and speciﬁc architecture blocks, such as layers (Jawahar et al., 2019), encoded hidden states (Tang et al., 2018), and attention heads (Michel et al., 2019).
Revised: Other studies have improved the understanding of what and where these models encode information, by analyzing correlations between input-targets and speciﬁc architecture blocks, such as layers (Jawahar et al., 2019), encoded hidden states (Tang et al., 2018; Saphra & Lopez, 2019), and attention heads (Michel et al., 2019).

Original:  Using probing classiﬁers, Clark et al.
Revised: Evidence of syntactic information : Using probing classiﬁers, Clark et al.

Original: (2019) claims that some speciﬁc BERT’s attention heads show correspondence with syntactic tasks.
Revised: (2019) claims that some speciﬁc BERT’s attention heads show correspondence with syntactic tasks.

Original: They showed that some heads on the same layer have similar behavior.

Original: Goldberg (2019) illustrates the capabilities that BERT has to solve syntactic tasks, such as subject-verb agreement.
Revised: Goldberg (2019) illustrates the capabilities that BERT has to solve syntactic tasks, such as subject-verb agreement.

Original: BERT’s success in these tasks fuels the belief that BERT can code the syntax of a language.
Revised: BERT’s success in these tasks fuels the belief that BERT can code the syntax of a language.

Original: Hewitt & Manning (2019) propose a structural probe that evaluates whether syntax trees are embedded in BERT embeddings’ linear transformation.
Revised: Hewitt & Manning (2019) proposes a structural probe that evaluates whether syntax trees are encoded in a linear transformation of BERT embeddings.

Original: The study shows that such transformation exists in BERT, providing evidence that syntax trees are implicitly embedded in BERT’s vector geometry.
Revised: The study shows that such transformation exists in BERT, providing evidence that syntax trees are implicitly embedded in BERT’s vector geometry.

Original: Reif et al.
Revised: Reif et al.

Original: (2019) have found evidence of syntactic representation in BERT’s attention matrices, with certain directions in space representing particular dependency relations.
Revised: (2019) has found evidence of syntactic representation in BERT’s attention matrices, with speciﬁc directions in space representing particular dependency relations.

Original: The study shows that just as there are speciﬁc syntactic subspaces, there is evidence for subspaces representing semantic information.
Revised: Evidence of semantic information : Reif et al.

Original: The authors suggest that BERT’s internal geometry may be broken into multiple linear subspaces, with separate spaces for different syntactic and semantic information.
Revised: (2019) suggests that BERT’s internal geometry may be broken into multiple linear subspaces, with separate spaces for different syntactic and semantic information.

Original: The ability of Transformer-based models to solve tasks at the semantic level is more debatable.
Revised: Despite this, previous work has not yet reached consensus about this topic.

Original: While some studies show satisfactory results in tasks such as entity types (Tenney et al., 2019a), semantic roles (Rogers et al., 2020), and sentence completion (Ettinger, 2020), other studies show less favorable results in coreference (Tenney et al., 2019b) and Multiple-Choice Reading Comprehension (Si et al., 2019), claiming that BERT’s performance may not reﬂect the model’s true ability of language understanding and reasoning.
Revised: While some studies show satisfactory results in tasks such as entity types (Tenney et al., 2019a), semantic roles (Rogers et al., 2020), and sentence completion (Ettinger, 2020), other studies show less favorable results in coreference (Tenney et al., 2019b) and Multiple-Choice Reading Comprehension (Si et al., 2019), claiming that BERT’s performance may not reﬂect the model’s true ability of language understanding and reasoning.

Original: Some works have studied which blocks of BERT are used to solve tasks at the semantic level.
Revised: Some works have studied which blocks of BERT are used to solve tasks at the semantic level.

Original: Tenney et al.
Revised: Tenney et al.

Original: (2019b) proposed a set of edge probing tasks to test the encoded sentential structure of contextualized word embeddings.
Revised: (2019b) proposes a set of edge probing tasks to test the encoded sentential structure of contextualized word embeddings.

Original: The study shows evidence that the improvements that BERT and GPT-2 offer over non contextualized embeddings as GloVe is only signiﬁcant in syntactic-level tasks.
Revised: The study shows evidence that the improvements that BERT and GPT-2 offer over non contextualized embeddings as GloVe is only signiﬁcant in syntactic-level tasks.

Original: Tenney et al. (2019a) combine information from the many layers of a BERT encoder to solve probing tasks.
Revised: Regarding static word embeddings, Yaghoobzadeh et al. (2019) shows that senses are well represented in single-vector embeddings if they are frequent, and that this does not have a negative impact on NLP tasks whose performance depends on frequent senses. Layer-wise or head-wise information : Tenney et al.

Original: The study shows that the ﬁrst layers of the encoder better solve tasks that require short dependency relationships at the syntactic level (e.g., subject-verb agreement) and that tasks that require long-range dependencies (e.g., subject-object dependencies) require information encoded in the top layers.
Revised: (2019a) shows that the ﬁrst layers of BERT focus on encoding short dependency relationships at the syntactic level (e.g., subject-verb agreement) while top layers focus on encoding long-range dependencies (e.g., subject-object dependencies). Peters et al. (2018a) supports similar declarations for Convolutional, LSTM, and self-attention architectures.

Original: While these studies also claim that the top layers appear to encode semantic information, the evidence to support this claim is not conclusive.
Revised: While these studies also support that the top layers appear to encode semantic information, the evidence to support this claim is not conclusive or contradictory with other works.

Original: For example, Jawahar et al.
Revised: As an example, Jawahar et al.

Original: (2019)  only manages to locate one SentEval semantic task  topped at the last layer.
Revised: (2019) could only identify one SentEval semantic task that topped at the last layer.

Original: Some studies have focused on understanding how BERT layers encode information. Voita et al.
Revised: In terms of information ﬂow, Voita et al.

Original: (2019a) found that information about the past in left-to-right language models gets vanished as the information ﬂows from bottom to top BERT’s layers.
Revised: (2019a) reports that information about the past in left-to-right language models gets vanished as the information ﬂows from bottom to top BERT’s layers.

Original: Press et al.
Revised: Hao et al.

Original: (2020) evaluated different patterns to combine the  BERT layers, ﬁnding that  more self-attention at the bottom and more feedforward sub-layers at the top produced better results in language model tasks.
Revised: (2019) shows that the lower layers of BERT change less during ﬁnetuning, suggesting that layers close to inputs learn more transferable language representations.

Original: Hao et al.
Revised: In terms of architecture design, Press et al.

Original: (2019) showed that  the lower layers of BERT are more invariant during ﬁnetuning, suggesting that the layers close to input learn more transferable language representations.
Revised: (2020) provides evidence that increasing self-attention at the bottom and increasing feed-forward sub-layers at the top improves results in language modeling tasks using BERT.

Original: Other studies have focused on understandingand visualizing howhelpful self-attention heads are to solve some speciﬁc tasks (Vig, 2019).
Revised: Other studies have focused on understanding how self-attention heads contribute to solving speciﬁc tasks (Vig, 2019).

Original: Kovaleva et al.
Revised: Kovaleva et al.

Original: (2019) study attention weights based on results obtained from tasks deﬁned in GLUE (Wang et al., 2018), showing a set of attention patterns that is repeated across different heads.
Revised: (2019) shows a set of attention patterns that is repeated across different heads when trying to solve GLUE tasks (Wang et al., 2018).

Original:  Michel et al.
Revised: Furthermore, Michel et al.

Original: (2019)  show that several heads can be removed from trained Transformer models without degradation in machine translation tasks.
Revised: (2019) and Voita et al. (2019b) show that several heads can be removed from trained Transformer models without degradation in downstream tasks.

Original: This result coincides with those found by Voita et al. (2019b). In summary, on the one hand,the related work shows results that provide strong evidence concluding that BERT and other  modelsbased on the Transformer architecture can encode information at the syntactic level.
Revised: In summary, on the one hand, related work shows results that provide strong evidence concluding that BERT and other Transformer-based models can encode information at the syntactic level.

Original: Furthermore, BERT uses this information to solve various benchmark NLP tasks.
Revised: Furthermore, BERT uses this information to solve various benchmark NLP tasks.

Original: Working with ambiguous words has allowed BERT to perform well on MT tasks, outperforming other architectures in WSD.
Revised: Working with ambiguous words has allowed BERT to perform well on Machine Translation tasks (MT), and outperforming other architectures in word sense disambiguation (WSD).

Original: On the other hand, while some studies claim that top layers are helpful to solve semantic tasks, the results are not conclusive.
Revised: On the other hand, while some studies claim that top layers are helpful to solve semantic tasks, the results are not conclusive.

Original: Efforts to locate which blocks of the Transformer mechanisms operate at the semantic level have been unsuccessful.
Revised: Efforts to locate which blocks of the Transformer architecture operate at the semantic level have been unsuccessful.

Original: In this context, our work ﬁlls the gap between the better understanding that we have  about  word embeddings and language models capacities to work with syntax, and the  shallow comprehension of their abilities to encode and work with semantics.  
Revised: In this context, our work helps to ﬁll the gap between the deeper understanding that we have now about how word embeddings and language models encode and work with syntax, and the still shallow comprehension of their abilities to encode and work with semantics.  S TUDY METHODOLOGY

Revised: Probing methods consist of using the representation of a frozen pre-trained model to train a classiﬁer to achieve a particular task.

Revised: If the probing classiﬁer succeeds in this setting but fails using an alternative model, it means that the source model encodes the knowledge needed to solve the task.

Revised: Furthermore, the performance of the classiﬁer can be used to measure how well the model captures this knowledge (Conneau et al., 2018).

Original: S TUDY METHODOLOGY 3. W ORD EMBEDDINGS MODELS
Revised: Following Hewitt & Manning (2019), we use a similar probing method at the semantic level and apply it to the nine models presented in Section 3.1. Probing techniques have shown useful for NLP tasks (Saphra & Lopez, 2019), however, they have also been questioned (Ravichander et al., 2020), as probe success does not guarantee that the model relies on that information to solve a target task. Consequently, our study limits to shed light on whether the models under evaluation encode relevant knowledge to solve the task of predicting concept relatedness in a semantic ontology such as Wordnet. To study how precisely the models encode semantic information, we measure correctness in predicted relations among concepts at two levels: (a) pair-wise-level by studying performance across sampled pairs of related or unrelated concepts, and (b) graph-level by using pair-wise predictions to reconstruct the actual graph.

Revised: We describe these two approaches in Sections 3.2 and 3.3, respectively.

Revised: They essentially study the same phenomenon as both share the exact same model predictions.

Revised: They only differ in the way they use these predictions.

Revised: 3. W ORD EMBEDDINGS MODELS

Original: This study considers the most inﬂuential Language Models from recent years.
Revised: This study considers the most inﬂuential language models from recent years.

Original: We considered the essential approaches of three model families: non contextualized word embeddings (NCE), contextualized word embeddings (CE), and generative language models (GLM).
Revised: We consider the essential approaches of three model families: non contextualized word embeddings (NCE), contextualized word embeddings (CE), and generative language models (GLM).

Original: We consider Word2Vec
Revised: We consider Word2Vec

Original: (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) for the ﬁrst family of approaches.
Revised: (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) for the ﬁrst family of approaches.

Original: For the CE family, we consider ELMo (Peters et al., 2018), which is implemented on a bidirectional 
Revised: For the CE family, we consider ELMo (Peters et al., 2018b), which is implemented on a bidirectional LSTM architecture, XLNet

Original: LSTM architecture, XLNet (Yang et al., 2019), and BERT (Devlin et al., 2019) and its extensions ALBERT (Lan et al., 2020) and RoBERTa (Liu et al., 2019), all of them based on the Transformer architecture.
Revised: (Yang et al., 2019), and BERT (Devlin et al., 2019) and its extensions ALBERT (Lan et al., 2020) and RoBERTa (Liu et al., 2019), all of them based on the Transformer architecture.

Original: GPT-2 (Radford et al., 2018) and T5 (Raffel et al., 2019) were included in the study to incorporate approaches based on generative language models.
Revised: GPT-2 (Radford et al., 2018) and T5 (Raffel et al., 2019) are included in the study to incorporate approaches based on generative language models.

Original: 3. S EMANTIC PROBING CLASSIFIER
Revised: 3. S EMANTIC PROBING CLASSIFIER

Original: We deﬁne an edge probing classiﬁer that learns to identify if two words are semantically related.
Revised: We deﬁne an edge probing classiﬁer that learns to identify if two concepts are semantically related.

Original: To create the probing classiﬁer, we retrieve all the glosses from the Princeton WordNet Gloss Corpus 1 .
Revised: To create the probing classiﬁer, we retrieve all the glosses from the Princeton WordNet Gloss Corpus 2 .

Original: This dataset provides WordNet’s synsets gloss sentences with annotations identifying occurrences of concepts.
Revised: This dataset provides WordNet’s synsets gloss sentences with annotations identifying occurrences of concepts within different sentence contexts.

Original: The annotations provide a mapping of the used words to their corresponding WordNet node.
Revised: The annotations provide a mapping of the used words to their corresponding WordNet node.

Original: We sample hypernym pairs A,
Revised: We sample hypernym pairs A,

Original: B. We sample a third synset C, at random from an unrelated section of the taxonomy,  taking care that C is not related to either A or B.
Revised: B. Then, from an unrelated section of the taxonomy, we randomly sample a third synset C, taking care that C is not related to either A or B.

Original: Then, (cid:104) A, B, C (cid:105) form a triplet that allows us to create six testing edges for our classiﬁer.
Revised: Then, (cid:104) A, B, C (cid:105) forms a triplet that allows us to create six testing edges for our classiﬁer.

Original: To train the probing classiﬁer, we deﬁne a labeled edge { x, y, L } , with x and y synsets in { A, B, C } , x (cid:54) =
Revised: To train the probing classiﬁer, we deﬁne a labeled edge { x, y, L } , with x and y synsets in { A, B, C } , x (cid:54) =

Original: y .
Revised: y .

Original: L  is the target of the edge, L in {− 1 , +1 } .
Revised: L ∈ { 0 , 1 } is the target of the edge.

Original: If y is direct or indirect parent of x , L = 1 , while L = − 1 in other case.
Revised: If y is direct or indirect parent of x , L = 1 , while L = 0 in other case.

Original: For each synset x , y , we sample one of its glosses S ( x ) , S ( y ) from the dataset.
Revised: For each synset x , y , we sample one of its sentences S ( x ) , S ( y ) from the dataset.

Original: Let M be a word embedding model.
Revised: Let M be a word embedding model.

Original: If M is a non contextualized word embedding model, x and y are encoded by M ( x ) and M ( y ) , respectively.
Revised: If M belongs to the NCE family, x and y are encoded by M ( x ) and M ( y ) , respectively.

Original: If M is a contextualized word embedding or a generative language model, x and y are encoded by the corresponding token of M ( S ( x )) and M ( S ( y )) , respectively.
Revised: If M belongs to the CE or GLM families, then x and y are encoded by the corresponding token of M ( S ( x )) and M ( S ( y )) , respectively.

Original: Accordingly, the gloss of each word provides a context for the creation of its word embedding.
Revised: Accordingly, the context of each concept provides additional information for the creation of its word embedding.

Original: To accommodate for the evaluation of word embeddings of different dimensionalities, we ﬁrst project the embeddings of each word x and y into a lower dimensionality space using a linear layer (see Fig 1).
Revised: To accommodate for the evaluation of concept embeddings of different dimensionalities, we ﬁrst project the embeddings of each concept x and y into a low dimensionality space using a linear layer (see Figure 1).

Revised: Choosing a low target dimensionality for this linear layer was also essential to avoid overﬁtting the classiﬁcation module.

Original: Then, these vectors denoted as x (cid:48) and y (cid:48) , are concatenated and fed into a Multi-Layer Perceptron (MLP) classiﬁer.
Revised: Then, these vectors, denoted as x (cid:48) and y (cid:48) , are concatenated and fed into a Multi-Layer Perceptron (MLP) classiﬁer.

Original: The linear layer and the MLP are the only trainable parameters of our setting, as we use the source model weights without any ﬁnetuning.
Revised: The linear layer and the MLP are the only trainable parameters of our setting, as we use the source model weights without any ﬁnetuning.

Original: Similar structural probes have been proposed to evaluate syntax encoding in BERT.
Revised: Similar structural probes have been proposed to evaluate syntax encoding in BERT.

Original: However, we use an MLP to learn the structural relation between word pairs, providing the test with a mechanism that allows the embeddings to be combined in a non-linear way. 
Revised: However, we use a MLP to learn the structural relation between concept pairs, providing the test with a mechanism that allows the embeddings to be combined in a non-linear way. Tests based on linear transforma structure.

Original: Tests based on linear transformations such as that proposed by Hewitt & Manning (2019) did not allow us to recover the WordNet structure, which indicates that the subspaces in which the word embeddings models encode the semantics are not linear. 
Revised: This indicates that the sub-spaces where the language models encode this semantic information are not linear. The fact that syntactic information is linearly available with high accuracy, makes us believe that syntax trees might be a critical intermediate result needed for the language modeling task; whereas semantic information emerges as an indirect consequence of accurate language modeling and syntactic identiﬁcation, but might not constitute information that the model relies on for the language modeling task, as postulated by Ravichander et al.

Original: 3. R ECONSTRUCTING THE STRUCTURE OF THE KNOWLEDGE GRAPHS
Revised: 3. R ECONSTRUCTING THE STRUCTURE OF A KNOWLEDGE GRAPH

Original: Given a graph G with nodes N and unknown edges E , we deﬁne an auxiliary graph G (cid:48) with nodes N and edges E (cid:48) , comprised of all possible directed edges. For each edge e ∈ E (cid:48) , we obtain a prediction h e that estimates the probability of that edge representing a valid hypernymy relation, and a distance d e that estimates the hypernym distance 2 between the nodes in G .
Revised: The proposed probe classiﬁer predicts if a pair of concepts (cid:104) u, v (cid:105) form a valid (cid:104) parent , child (cid:105) relation according to WordNet, where h (cid:104) u,v (cid:105) ∈ [0 , 1] denotes the corresponding classiﬁer output.

Original: We deﬁne δ ( v ) to be the set of edges {(cid:104) u, v (cid:105) : u ∈ N, u (cid:54) = v } where the edge (cid:104) u, v (cid:105) deﬁnes a (cid:104) parent , child (cid:105) relation. We also deﬁne γ ( S ) as the set of edges {(cid:104) u, v (cid:105) ∈ E (cid:48) : u / ∈ S, v ∈ S } . We estimate the graph topology of G deﬁned by E ⊂ E (cid:48) , by solving this optimization problem: [Equation]
Revised: It is important to note that valid (cid:104) parent , child (cid:105) relations include direct relations, as in (cid:104) dog , poodle (cid:105) , and transitive relations, as in (cid:104) animal , poodle (cid:105) , and that the order of the items in the relation do matter.

Original: where the objective function (1) is used to ﬁnd the best root node r and the nested optimization problem (2) is the minimum spanning arborescence problem applied to the dense graph G (cid:48) . The ﬁnal binary values of x e estimate E by indicating if every possible edge e exist in the graph or not. To solve this optimization problem, we needestimates of h e and d e for each edge e . We use the probing classiﬁer’s output probability as an estimate of h e , and propose the following two alternatives to estimate d e .
Revised: To reconstruct the underlying knowledge graph, for each valid (cid:104) parent , child (cid:105) relation given by h (cid:104) u,v (cid:105) > threshold , we need an estimation of how close are the corresponding nodes in the graph.

Original: Model Conﬁdence Metric (MCM) : As we will discuss in Section 5, all nine models in this study tend to capture close concept relations more precisely than distant links. In practice, that originates that aconcept like chihuahua will be matched with its parent node dog with higher conﬁdence than its parent node entity . Thus the score 1 − h e is a reasonable metric for d e  .
Revised: To do this, we introduce the concept of “parent closeness” between a parent node u and a child node v , that we deﬁne as d e ( u, v ) .

Original:  Transitive Intersections Metric (TIM) :As the before mentioned pattern is only a fortunate side effect that may not hold true for future models, we explored another metric grounded directly in the tree structure of the underlying knowledge graph.
Revised: We propose two alternative scores to estimate d e : i) Transitive Intersections Metric (TIM) : we explore a metric grounded directly in the tree structure of a knowledge graph.

Original: The key observations are that nodes u and v that form a parent-child relation have some transitive connections in common.
Revised: The key observations are that nodes u and v that form a parent-child relation have some transitive connections in common.

Original: Speciﬁcally, all descendants of v are also descendants of u , and all the ancestors of u are also ancestors of v .
Revised: Speciﬁcally, all descendants of v are also descendants of u , and all the ancestors of u are also ancestors of v .

Original: Furthermore, the closer the link between u and v in the graph, the bigger the intersection.
Revised: Furthermore, the closer the link between u and v in the graph, the bigger the intersection.

Original: With that in mind, for each candidate edge e = (cid:104) u, v (cid:105) we deﬁne the followingalternative metric for d e :
Revised: With that in mind, for each candidate edge e = (cid:104) u, v (cid:105) we deﬁne the following metric for d e :

Original: [Equation]
Revised: [Equation]

Revised: where the ﬁrst sum accounts for the similarity in the descendants of nodes u and v , and the second sum accounts for the similarity in the ancestors of nodes u and v .

Original: The reason why we included h (cid:104) u,v (cid:105) at the end is  to consider the strength of the edge direction.
Revised: The term h (cid:104) u,v (cid:105) at the right-hand side accounts for the direction of the edge. Constant C = max u i ,v j d ( u i , v j ) is added to force d e to be positive. N denotes the set of all nodes (concepts).

Original: Constant C was added to force d e to be positive.
Revised: ii) Model Conﬁdence Metric (MCM) : All nine models in this study capture close concept relations more precisely than distant relations (supporting evidence can be found in Appendix C). In practice, this means that a concept like poodle will be matched with its direct parent node dog with higher conﬁdence than with a more distant parent node such as animal . Thus it is reasonable to use d e ( u, v ) =

Original: H OW ACCURATE IS THEIR SEMANTIC KNOWLEDGE ? 4. S EMANTIC PROBING CLASSIFIER RESULTS
Revised: 1 − h (cid:104) u,v (cid:105) . Based on d e , a natural choice to ﬁnd the tree that comprises only the closest parent of each node is to obtain the minimum-spanning-arborescence (MSA) of the graph deﬁned by d e . The MSA is analogous to the minimum-spanning-tree (MST) objective used in Hewitt & Manning (2019), but for the speciﬁc case of directed graphs. For a mathematical formulation of the MSA optimization problem applied to our setting, we refer readers to Appendix A.3.

Revised:  H OW ACCURATE IS THE SEMANTIC KNOWLEDGE ? 4. S EMANTIC EDGE PROBING CLASSIFIER RESULTS

Original: Table 1 shows the results obtained using the edge probing classiﬁer in many variants of the models studied.
Revised: Table 1 shows the results obtained using the edge probing classiﬁer in several model variants.

Original: Results show that regardless of model sizes, performance is homogeneous within each family of models.
Revised: Results show that regardless of model sizes, performance is homogeneous within each family of models.

Original: Additionally, results show that  NCE and GLM methods obtain a worse performance than those achieved by CE methods.
Revised: Additionally, results show that when all the layers are used, NCE and GLM methods obtain a worse performance than those achieved by CE methods.

Original: In particular, CE family’s differences over NCE are signiﬁcant, reaching around 5 points of improvement inF1. When using the base model, some models improve their performance while others decrease it, with ELMo being the only one with a considerable improvement in performance when using the large version of the model (3.7%).
Revised: When single layers are used, GLM show improved performance, suggesting that the generative objective forces the model to capture semantics earlier in the architecture in order to keep the last layers for generative-speciﬁc purposes that do not have a direct relationship with this kind of knowledge. In contrast, CE models degrade or maintain their performance when single layers are used.

Original: Notice, though, that all the metrics in Table 1 are pair-wise metrics, not graph metrics.
Revised: Notice that results in Table 1 show pair-wise metrics not graph metrics.

Original: As we are dealing with graphs, predicted edges are built upon related predicted edges, thus drifts in small regions of the  graph may cause large drifts in downstream connections.
Revised: As we are dealing with graphs, predicted edges are built upon related predicted edges, thus drifts in small regions of the [Table] graph may cause large drifts in downstream connections.

Original: Furthermore, our setup balances positive (links) and negative samples (not links), butin a real reconstruction scenario, the proportion of negative samples will be considerably larger.
Revised: Furthermore, our setup balances positive (links) and negative samples (not links), but the proportion of negative samples will be considerably larger in a real reconstruction scenario.

Original: As a consequence, we emphasize that these numbers must not be considered in isolation, but together with the results reported in sections 4.2 and 4.3. [Table]
Revised: As a consequence, we emphasize that these numbers must not be considered in isolation, but together with the results reported in sections 4.2 and 5.

Original: 4. E XTRACTING THE UNDERLYING K NOWLEDGE G RAPH
Revised: 4. E XTRACTING THE UNDERLYING K NOWLEDGE G RAPH

Original: Predicting a knowledge graph has a complexity of at least O ( N 2 ) in the number of analyzed concepts.
Revised: Predicting a knowledge graph has a complexity of at least O ( N 2 ) in the number of analyzed concepts.

Original: In our case, this imposes a highly demanding computational complexity because WordNet has over 82000 noun synsets.
Revised: In our case, this imposes a highly demanding computational obstacle because WordNet has over 82000 noun synsets.

Original: To accelerate experimentation and  facilitate our analysis and visualizations, we focus on extracting a WordNet sub-graph that comprises only 46 nodes.
Revised: To accelerate experimentation and to facilitate our analysis and visualizations, we focus on extracting a WordNet sub-graph that comprises only 46 nodes.

Original: These nodes were picked to create a target graph with easily recognizable relations.
Revised: These nodes are picked to include easily recognizable relations.

Original: We use the tree-edit-distance to evaluate how close are the reconstructed graphs to the target graph extracted from WordNet.
Revised: We use the tree-edit-distance to evaluate how close are the reconstructed graphs to the target graph extracted from WordNet.

Original: Table 2 shows our results.
Revised: Table 2 shows our results.

Original: q u a li t y
Revised: q u a li t y

Original: o n e n e s s s i m il a r i t
Revised: o n e n e s s s i m il a r i t

Original: y s a m e n e s s i d e n t
Revised: y s a m e n e s s i d e n t

Original: i t y li
Revised: i t y li

Original: k e n e s s e q u a li t
Revised: k e n e s s e q u a li t

Original: y h e r b c h
Revised: y h e r b c h

Original: i c k e n w o o d y _ p l a n t r
Revised: i c k e n w o o d y _ p l a n t r

Original: o
Revised: o

Original: c k c r y s t a l li v i n g _
Revised: c k c r y s t a l li v i n g _

Original: t h i n g a n i m a l c a t r e p t il e m a m m a l p l a n t b i r d e n t
Revised: t h i n g a n i m a l c a t r e p t il e m a m m a l p l a n t b i r d e n t

Original: i t y p
Revised: i t y p

Original: a l m p i n e s p i n a c
Revised: a l m p i n e s p i n a c

Original: h t i g e r p a rr o t n a t u r a l _ o b j e c t o b j e c t a b s t r a c t i o n a p p r o x i m a t i o n a tt r i b
Revised: h t i g e r p a rr o t n a t u r a l _ o b j e c t o b j e c t a b s t r a c t i o n a p p r o x i m a t i o n a tt r i b

Original: u t e c a l c u l u s s a l a m a n d e r p l a t
Revised: u t e c a l c u l u s s a l a m a n d e r p l a t

Original: y p u
Revised: y p u

Original: s m o n k e
Revised: s m o n k e

Original: y c r o c o d il e a lli g a t
Revised: y c r o c o d il e a lli g a t

Original: o r p r
Revised: o r p r

Original: i m a t e h o m o _ s a p i e n s f e li n e s n a k e
Revised: i m a t e h o m o _ s a p i e n s f e li n e s n a k e

Original: a m p
Revised: a m p

Original: h
Revised: h

Original: i b i a n m o n
Revised: i b i a n m o n

Original: o t r e m e f r o g t u r t l e l e tt u c e c a n a r y
Revised: o t r e m e f r o g t u r t l e l e tt u c e c a n a r y

Original: Family
Revised: Family

Original: Model
Revised: Model

Original: Tree Edition Dist.
Revised: Tree Edit Dist.

Original: TIM
Revised: TIM

Original: MCM
Revised: MCM

Original: Avg.
Revised: Avg.

Original: NCE
Revised: NCE

Original: Word2Vec
Revised: Word2Vec

Original: GloVe-42B
Revised: GloVe-42B

Original: GLM
Revised: [Table]

Original: GPTT
Revised: GPTT

Original: CE
Revised: CE

Original: ELMo
Revised: ELMo

Original: BERT
Revised: BERT

Original: RoBERTa
Revised: RoBERTa

Original: XLNet
Revised: XLNet

Original: ALBERT
Revised: ALBERT

Original: Table 2 shows that the knowledge graphs retrieved usingthe CE family of methods are closer to the target than the other graphs.
Revised: Table 2 shows that the knowledge graphs retrieved using CE models are closer to the target than graphs provided by NCE and GLM models.

Original: In particular, the best results are achieved by BERT,ALBERT, and
Revised: In particular, the best results are achieved by BERT,

Original:  XLNet, showing that these models have better capabilities for encoding semantic information than their competitors.
Revised: ALBERT, and XLNet, indicating that these models encode more accurate semantic information than the alternative models.

Original: These results are consistent with the results obtained by the semantic probing classiﬁer.
Revised: These results are consistent with those obtained in Section 4.1.

Original: Visual inspection of the reconstructed graphs reveal that models capture some important categories, but fail to map some ﬁne-grained relations.4. A RE ALL THESE MODELS CORRELATED ? In this section [Figure]
Revised: Visual inspection of the reconstructed graphs as the one found in Figure 2 reveals that models capture some key categories but fail to map some ﬁne-grained relations.

Original: we study if concepts that are easy/hard to associate for one model are also easy/hard to associate for other models.

Original: We do this by analyzing the correlation between outputs using the semantic probing classiﬁer.

Original: Graphs in Figure 3 show the correlation among the models by plotting the F1-score of one model against the rest.

Original: We picked one model of each family: GloVe (Fig 3-left)

Original: RoBERTa-large (Fig 3-middle), and T5-large (Fig 3-right).

Original: [Figure]

Original: The graphs in Figure 3 reveal that correlations within each model family are strong, meaning that they tend to acquire similar biases.

Original: Figure 3-middle shows a strong correlation between RoBERTa and other methods in its family, but weaker with models of the NCE family.

Original: Fig 3-right shows that

Original: T5 has the same difﬁculties as the rest of the methods, even showing some correlation with the NCE family of methods.

Original:  W HAT IS EASY OR HARD ? W HAT ARE THESE MODELS LEARNING ?
Revised:  W HAT IS EASY OR HARD ? W HAT ARE THESE MODELS LEARNING ?

Original: The previous section shows thatmodels from different  families differ in their errors, but that models from the same families have similar biases.
Revised: Section 4 shows that different model families differ in their errors. Furthermore, it shows that within the same family, models have similar biases.

Original: In this section, we elucidate which semantic factors have a higher impact on the performance of these models.
Revised: In this section, we elucidate which semantic factors have an impact on the performance of these models and which ones do not affect their F1-score.

Original: [Figure] Relative depth in the WordNet graph : For each synset, we compared F1 with depth score (0 % for the root and 100 % for leaves) measuring differences between higher/lower level concepts. As seen in Figure 4-left, NCE-based methods are signiﬁcantly impacted by this variable.
Revised: Appendix C shows additional factors details

Original: Graph distance between concepts : We measured the impact of the number of “hops” that separate two tested concepts on the pair-wise F1 score. Figure 4-middle reveals a strong correlation of all the models in this aspect.
Revised: Figure 3-a shows that most models decrease their F1-score as concepts get more speciﬁc (increased depth). We hypothesize that this is because higher-level concepts (e.g. Animal ) appear more frequently and in more diverse contexts, as they are also seen as instances of their subclasses (e.g.

Original: As an example of this phenomenon, closer relations such as (cid:104) chihuahua , dog (cid:105) are, in general, considerably easier to capture than distant relations such as (cid:104) chihuahua , entity (cid:105) . For details on how we implement the distance in WordNet, check Appendix A.2.
Revised: Dog , Cat , Chihuahua ), allowing the models to learn more precise representations for them. In contrast, lower-level concepts will only be present in their speciﬁc contexts (e.g. texts about AppleHead-Chihuahua ).

Original: Concept frequency : In Figure 4-right, we evaluate if frequent concepts are easier or harder to capture for these models. The frequency was computed by counting occurrences in the 38 GB of
Revised: This hypothesis is conﬁrmed by Figure 3-b, as concepts with higher number of sub-classes have higher F1-scores. Furthermore, Figure 3-c also shows that models tend to capture concepts more accurately as the model is exposed to them more frequently in the pre-training corpus.

Original: OpenWebText Corpus 3 . The graph shows that uncommon and highly frequent concepts are harder to be modeled.
Revised: Additionally, Figure 3-c indicates that the models degrade their F1-score when concepts are too frequent.

Original: In particular, NCE-based models and T5 are more sensitive to this factor.
Revised: In particular, NCE and GLM models are more sensitive to this factor.

Original: While the behavior related to frequent concepts seems to be counter-intuitive, we hypothesize that this corresponds to a phenomenon similar to what stop-words exhibit in language.
Revised: We hypothesize that this is because high-frequency concepts appear in contexts with too much variability.

Original: Their unspeciﬁc use confused the models.
Revised: Their unspeciﬁc use makes more difﬁcult to distinguish accurate meaning for NCE and GLM models.

Original: Number of child nodes : We discovered that all models have minimum F1-score for concepts with one or two child nodes (considering both direct and indirect children). After that point, they all increase their F1 as  the number of child nodes increase.
Revised: Another surprising ﬁnding is that CE and GLM models are almost unaffected by the number of senses that a certain word has, neither to their sense ranking or their number of sibling concepts, displaying almost ﬂat charts (Figure 3-d-e-f). This ﬁnding suggests that these models pay as much attention to the context as to the target word, and are probably biased in favor of contextual information.

Original: The same phenomenon shows up when we analyze only direct children.
Revised: This is intuitive for Masked-Language-Models such as BERT, but not for others, such as GPT-2. Furthermore, this behavior is opposed to what NCE models exhibit according to Yaghoobzadeh et al. (2019) and our experiments, as NCE models tend to focus more on the most frequent senses. Analysis of speciﬁc semantic categories indicates that in most cases, the models of the same family have similar behaviors, especially within the NCE or CE families.

Original: We hypothesize that concepts with a greater number of sub-categories appear in more diverse contexts in  the models’ original training corpus, thus models learn more accurate representations for them.
Revised: More importantly, different families show different patterns. Table 3 shows some salient examples. Surprisingly, all models struggle with the category “taxonomic groups”. Manual inspection of sentences and analysis of related concepts make us believe that in these cases the use of the context confuses CE and GLM models, as in

Original: The graph of this variable is in Appendix C.
Revised: [Table] many sentences the corresponding concept could be nicely replaced by another, conveying a modiﬁed but still valid message. This phenomenon does not take place in other categories such as “social group” or “attribute”, even though these concepts are closely related to “taxonomic groups”.

Original:  W HERE IS THIS KNOWLEDGE LOCATED ?
Revised:  W HERE IS THIS KNOWLEDGE LOCATED ?

Original: As mentioned in Section 2, prior work has not shown consensus on where isthe semantic information located inside these architectures.
Revised: As mentioned in Section 2, prior work has not shown consensus about where is semantic information primarily encoded inside these architectures.

Original: Our experiments shed light on this subject.
Revised: Our experiments shed light on this subject.

Original: In Figure 5 we show how each layer contributes to the F1 score. [Figure]
Revised: Figureshows how each layer contributes to the F1-score.

Original: Figures 5-left and 5-middle show the performance across layers for the CE-based models.
Revised: a and 4-b show the performance across layers for the CE-based models.

Original: They reveal that while BERT and RoBERTa use their top-layers to encode semantic information, XLNet and ALBERT use the ﬁrst layers.
Revised: They reveal that while BERT and RoBERTa use their top-layers to encode semantic information, XLNet and ALBERT use the ﬁrst layers.

Original: Figure 5(right) shows that GLM-based models have very distinct patterns.
Revised: Figure 4-c shows that GLM models have very distinct patterns.

Original: While GPT-2 encodes semantic information using all the layers, T5 shows an M shape because its architecture considers an encoder and decoder modules.
Revised: While GPT-2 encodes semantic information in all the layers, T5 shows an M shape because its conﬁguration considers an encoder-decoder architecture.

Original: The graph shows that T5 uses the encoder to hold most  semantic information.
Revised: The chart shows that T5 uses its encoder to hold most of the semantic information.

Original: We also note that small models show similar patterns as their larger counterparts, suggesting that these patterns depend on the architecture and not on model sizes.
Revised: We also note that small models show similar patterns as their larger counterparts, suggesting that these patterns depend on the architecture and not on model size.  F URTHER DISCUSSION AND IMPLICATIONS

Revised: Table 4 summarizes our main ﬁndings.

Revised: Findings (1), (2) and (3) indicate that, to a different extent, all models encode relevant knowledge about the hierarchical semantic relations included in WordNet.

Revised: However, some of this knowledge is incorrect.

Revised: In particular, as we mention in Section 5, we observe that the ability to learn about a concept depends on its frequency of appearances in the training corpus and the level of speciﬁcity of its meaning.

Original: C ONCLUSIONS
Revised: Furthermore, some concept categories result difﬁcult for all model families, while some categories are particularly difﬁcult for contextual models such as CE. We hypothesize that stronger inductive biases are required to capture low-frequency concepts. Furthermore, we believe that new learning approaches are needed to discriminate accurate meaning for high-frequency concepts. As expected, our ﬁndings indicate that model families have different biases leading to different behaviors.

Revised: Thus, our results can illuminate further research to improve semantic capabilities by combining the strengths of each family of models.

Revised: As an example,

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: one could combine them as ensembles or by incorporating more than a single model head, each one equipped with a different loss function (i

Revised: [Table]

Revised: e., one generative approach resembling GLM-based methods and another discriminative resembling CE-based methods).

Revised: Findings (4), (5) and (6) suggest that instead of a standard ﬁnetuning of all layers of BERT according to a given downstream task, to improve semantic capabilities one could perform a task proﬁling to decide the best architecture for the task and also how to take advantage of it.

Revised: By using only a limited number of layers or choosing a different learning rate for each layer, one could exploit the semantic knowledge that the pre-trained model carries, avoiding the degradation of this information present at the top layers, especially when using T5, XLNet, or ALBERT-large.

Revised: Accordingly, recent work on adaptive strategies to output predictions using a limited number of layers (Xin et al., 2020; Liu et al., 2020; Hou et al., 2020; Schwartz et al., 2020; Fan et al., 2020; Bapna et al., 2020) would beneﬁt from using architectures that encode knowledge in the ﬁrst layers.

Revised: To the best of our knowledge, these works have only used BERT and RoBERTa, achieving a good trade-off between accuracy and efﬁciency.

Revised: Only Zhou et al.

Original: In this work, we exploit the semantic conceptual taxonomy behind Wordnet totest the abilities of the current main families of pre-trained language models  to learn semantic knowledge from massive sources of unlabeled data.
Revised: (2020) has explored ALBERT, reporting improved accuracy by stopping earlier. Our ﬁndings explain this behavior and suggest that T5 or XLNet may boot their results even further as these architectures have sharper and higher information peaks in their ﬁrst layers. Findings (7) and (8) suggest that recent success in semantic NLP tasks might be due more to the use of larger models than large corpora for pretraining. This also suggests that to improve model performance in semantic tasks, one could train larger models even without increasing the corpus size.

Original: Our main conclusion is that, indeed, to a signiﬁcant extent, these models can learn relevant knowledge about the organization of concepts in Wordnet.
Revised: A similar claim has been proposed by Li et al. leading to empirical performance improvements. Finally, ﬁnding (9) is important because it suggests that contextual models pay as much attention to the context as to the target word, and are probably biased in favor of contextual information, even if they are not based on the Masked-Language-Model setting. We believe that this inductive bias could be exploited even further in the design of the underlying architecture, thus this ﬁnding might elucidate a design direction to encourage more effective learning of semantic knowledge.

Original: In particular, CE models, which are based on a Transformer architecture and a contextual masking training scheme, such as BERT, appear as the more powerful family of models.
Revised: In this work, we exploit the semantic conceptual taxonomy behind WordNet to test the abilities of current families of pre-trained language models to learn semantic knowledge from massive sources of unlabeled data. Our main conclusion is that, indeed, to a signiﬁcant extent, these models learn relevant knowledge about the organization of concepts in WordNet, but also contain several imprecisions.

Original: We also notice that different families of models present dissimilar behavior, suggesting the encoding of different biases.
Revised: We also notice that different families of models present dissimilar behavior, suggesting the encoding of different biases.

Original: Our ﬁndings also suggest that models such as XLNet and T5 encode information earlier in the pipeline.

Original: As a result, these models will greatly beneﬁt from recent adaptive strategies to output prediction using only a limited number of blocks from the Transformer architecture.

Original: As part of future work, we hope our study helps to inspire new ideas to improve the semantic learning abilities of current pre-trained language models.
Revised: As part of future work, we hope our study helps to inspire new ideas to improve the semantic learning abilities of current pre-trained language models.

Original: A I MPLEMENTATION DETAILS A. E DGE PROBING CLASSIFIER DETAILS
Revised: A I MPLEMENTATION DETAILS A. E DGE PROBING CLASSIFIER DETAILS

Original: To study the extent to which these Language Models deal with semantic knowledge, we extend the methodology introduced by Tenney et al. (2019b).
Revised: To study the extent to which these Language Models deal with semantic knowledge, we extend the methodology introduced by Tenney et al. (2019b).

Original: In that study, the authors deﬁned a probing classiﬁer at the sentence level, training a supervised classiﬁer with a task-speciﬁc label.
Revised: In that study, the authors deﬁned a probing classiﬁer at the sentence level, training a supervised classiﬁer with a task-speciﬁc label.

Original: The probing classiﬁer’s motivation consists of verifying when the sentence’s encoding help to solve a speciﬁc task, quantifying these results for different word embeddings models.
Revised: The probing classiﬁer’s motivation consists of verifying when the sentence’s encoding help to solve a speciﬁc task, quantifying these results for different word embeddings models.

Original: We cast this methodology to deal with semantic knowledge extracted from WordNet.
Revised: We cast this methodology to deal with semantic knowledge extracted from WordNet.

Original: Rather than working at the sentence level, we deﬁne an edge probing classiﬁer that learns to identify if two words are semantically related.
Revised: Rather than working at the sentence level, we deﬁne an edge probing classiﬁer that learns to identify if two concepts are semantically related.

Original: To create the probing classiﬁer, we retrieve all the glosses from the Princeton WordNet Gloss Corpus 4 .
Revised: To create the probing classiﬁer, we retrieve all the glosses from the Princeton WordNet Gloss Corpus 3 .

Original: The dataset provides WordNet’s synsets gloss with manually matched words identifying the context-appropriate sense (see Fig 6(a)).
Revised: The dataset provides WordNet’s synsets gloss with manually matched words identifying the context-appropriate sense (see Figure 5).

Original: In WordNet, each sense is coded as one of the synsets related to the word (e.g., sense tendency.n.03 for the word tendency).
Revised: In WordNet, each sense is coded as one of the synsets related to the concept (e.g., sense tendency.n.03 for the word tendency).

Original: Using a synet A and its speciﬁc sense provided by the tagged gloss, we retrieve from WordNet one of its direct or indirect hypernyms, denoted as B (see Fig 6(b)).
Revised: Using a synset A and its speciﬁc sense provided by the tagged gloss, we retrieve from WordNet one of its direct or indirect hypernyms, denoted as B (see Figure 6).

Original: If WordNet deﬁnes two or more hypernyms for A, we choose one of them at random.
Revised: If WordNet deﬁnes two or more hypernyms for A, we choose one of them at random.

Original: We sample a third synset C, at random from an unrelated section of the taxonomy, taking care that C is not related to either A or B (e.g., animal.n.01 ).
Revised: We sample a third synset C, at random from an unrelated section of the taxonomy, taking care that C is not related to either A or B (e.g., animal.n.01 ).

Original: Then, (cid:104) A, B, C (cid:105) form a triplet that allows us to create six testing edges for our classiﬁer: (cid:104) A, B (cid:105) , which is compounded by a pair of related words through the semantic relation hypernym of , and ﬁve pairs of unrelated words ( (cid:104) A, C (cid:105) , (cid:104) B, C (cid:105) , (cid:104) B, A (cid:105) , (cid:104) C, A (cid:105) , (cid:104) C, B (cid:105) ).
Revised: Then, (cid:104) A, B, C (cid:105) form a triplet that allows us to create six testing edges for our classiﬁer: (cid:104) A, B (cid:105) , which is compounded by a pair of related words through the semantic relation hypernym of , and ﬁve pairs of unrelated words ( (cid:104) A, C (cid:105) , (cid:104) B, C (cid:105) , (cid:104) B, A (cid:105) , (cid:104) C, A (cid:105) , (cid:104) C, B (cid:105) ).

Original: We associate a label to each of these pairs that show whether the pair is related or not (see Fig 6(c)).
Revised: We associate a label to each of these pairs that show whether the pair is related or not (see Figure 6).

Original: Note that we deﬁne directed edges, meaning that the pair (cid:104) A, B (cid:105) is related, but (cid:104) B, A (cid:105) is unrelated to the relationship hypernym of .
Revised: Note that we deﬁne directed edges, meaning that the pair (cid:104) A, B (cid:105) is related, but (cid:104) B, A (cid:105) is unrelated to the relationship hypernym of .

Original: Accordingly, the edge probing classiﬁer will need to identify the pair’s components and the order in which the words were declared in the pair. 
Revised: Accordingly, the edge probing classiﬁer will need to identify the pair’s components and the order in which the concepts were declared in the pair. [Figure] < , > → < > < , > → < > < , > → < > < , > → < > < , > →

Original:  We create training and testing partitions ensuring that each partition has the same proportion of leaves versus internal nodes.
Revised: [Figure] > < , > → [Figure] We create training and testing partitions ensuring that each partition has the same proportion of leaves versus internal nodes.

Original: The latter is essential to identify related pairs.
Revised: The latter is essential to identify related pairs.

Original: During training, we guarantee that each training synset is seen at least once by the probing classiﬁer.
Revised: During training, we guarantee that each training synset is seen at least once by the probing classiﬁer.

Original: To guarantee the above, we sample each synset in the training set and sample some of its hypernyms at random.
Revised: To guarantee the above, we sample each synset in the training set and sample some of its hypernyms at random.

Original: Then.
Revised: Then.

Original: we randomly sample some unrelated synset for each related pair that has no relation to any of the words in the related pair.
Revised: we randomly sample some unrelated synset for each related pair that has no relation to any of the words in the related pair.

Original: We create three partitions from this data on 70/15/15 for training, development, and testing foldings, respectively.
Revised: We create three partitions from this data on 70/15/15 for training, development, and testing foldings, respectively.

Original: We train the MLP classiﬁer using a weighted binary cross-entropy loss function.
Revised: We train the MLP classiﬁer using a weighted binary cross-entropy loss function.

Original: Since we have one positive and ﬁve negative examples per triplet, we use a weighted loss function with weights 5 and 1 for the positive and negative class, respectively.
Revised: Since we have one positive and ﬁve negative examples per triplet, we use a weighted loss function with weights 5 and 1 for the positive and negative class, respectively.

Original: Accordingly, positive and negative examples have the same relevance during training.
Revised: Accordingly, positive and negative examples have the same relevance during training.

Original: We implemented the linear layer and the MLP classiﬁer using FFNN with 384 hidden units.
Revised: We implemented the linear layer and the MLP classiﬁer using a feed forward network with 384 hidden units.

Original: The MLP was trained using dropout at 0.425 and a L 2 regularizer to avoid overﬁtting.
Revised: The MLP was trained using dropout at 0.425 and a Lregularizer to avoid overﬁtting.

Original: To create the vector representations for each of the word embeddings models considered in this study, we concatenate the hidden state vectors of all the layers for each tagged synset.
Revised: To create the vector representations for each of the word embeddings models considered in this study, we concatenate the hidden state vectors of all the layers for each tagged synset.

Original: For both
Revised: For both

Original: CE and GLM-based models, each gloss was used as a context to build speciﬁc contextual word embeddings.
Revised: CE and GLM-based models, each gloss was used as a context to build speciﬁc contextual word embeddings.

Original: If the gloss has more than one tagged token, we take only the ﬁrst of them for the analysis.
Revised: If the gloss has more than one tagged token, we take only the ﬁrst of them for the analysis.

Original: A. W ORD N ET METRICS : DISTANCE
Revised: A. W ORD N ET METRICS : DISTANCE

Original: Formally, let d W ( x, y ) be the Wordnet distance between two synsets x , y , deﬁned by:
Revised: Formally, let d W ( x, y ) be the Wordnet distance between two synsets x , y , deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: if yis ancestor of x,
Revised: y

Original: [Equation]
Revised: [Equation]

Original:  z 
Revised: if y is ancestor of x, d path ( x, z ) + d path ( y, z ) otherwise , where d path ( x, y ) is the length of the shortest path between x and y in WordNet, measured in number of hops, and z is the closest common ancestor of x and y in the case that y is not an ancestor of x . A. M INIMUM -S PANNING -A RBORESCENCE OPTIMIZATION PROBLEM APPLIED TO K NOWLEDGE G RAPH R ECONSTRUCTION Given a graph G with nodes N and unknown edges E , we deﬁne an auxiliary graph G (cid:48) with nodes N and edges E (cid:48) , comprised of all possible directed edges.

Original: [Equation]
Revised: For each edge e ∈ E (cid:48) , we obtain a prediction h e that estimates the probability of that edge representing a valid hypernymy relation, and a distance d e that estimates the “parent closeness” 4 between the nodes in G . We deﬁne δ ( v ) to be the set of edges {(cid:104) u, v (cid:105) : u ∈ N, u (cid:54) = v } where edge (cid:104) u, v (cid:105) represents a (cid:104) parent , child (cid:105) relation. We also deﬁne γ ( S ) to be the set of edges {(cid:104) u, v (cid:105) ∈ E (cid:48) : u / ∈ S, v ∈ S } . We estimate the graph topology of G deﬁned by E ⊂ E (cid:48) by solving the following optimization problem:

Revised: [Equation]

Original: otherwise , where d path ( x, y ) is  the length of the shortest path between x and y in WordNet, measured in number of hops, and z is the closest common ancestor ofx and y in the case that y is not an ancestor of x .
Revised: Objective function (2) is used to ﬁnd the best root node r ; and the nested optimization problem (3) is the minimum spanning arborescence problem applied to the dense graph G (cid:48) . The ﬁnal binary values of x e estimate E by indicating if every possible edge e exist in the graph or not. To solve this optimization problem, we need estimates of h e and d e for each edge e . We use the output of the probing classiﬁer as an estimate of the probability of h e , and use TIM and MCM scores as estimates for d e (See Section 3.3).

Original: B R ECONSTRUCTED G RAPHS
Revised: B R ECONSTRUCTED G RAPHS

Original: c a n a r y c h i c
Revised: c a n a r y c h i c

Original: k e n p a
Revised: k e n p a

Original: rr o t e n t
Revised: rr o t e n t

Original: i t y li
Revised: i t y li

Original: k e n e s s a p p r
Revised: k e n e s s a p p r

Original: o x
Revised: o x

Original: i m a t i o n o n e n e s s a b s t r a c t
Revised: i m a t i o n o n e n e s s a b s t r a c t

Original: i o n o b j e c t q u a li t y a lli g a t o r c r o c o d il e t u r t l e s n a k e p
Revised: i o n o b j e c t q u a li t y a lli g a t o r c r o c o d il e t u r t l e s n a k e p

Original: i n e p a l m l e tt u c e s p i n a c
Revised: i n e p a l m l e tt u c e s p i n a c

Original: h r o c k c a l c u l u s c r y s t a l s a m e n e s s p l a t y p u
Revised: h r o c k c a l c u l u s c r y s t a l s a m e n e s s p l a t y p u

Original: s a n i m a l p l a n t s a l a m a n d e r f r o g a tt r i b u t e b i r d r e p t il e m a m m a l a m
Revised: s a n i m a l p l a n t s a l a m a n d e r f r o g a tt r i b u t e b i r d r e p t il e m a m m a l a m

Original: p h
Revised: p h

Original: i b i a n w o o d y _ p l a n t h e r b
Revised: i b i a n w o o d y _ p l a n t h e r b

Original: h
Revised: h

Original: o m o _ s a p i e n s m o n k e
Revised: o m o _ s a p i e n s m o n k e

Original: y m o n o t
Revised: y m o n o t

Original: r
Revised: r

Original: e m e p r
Revised: e m e p r

Original: i m a t e f e li
Revised: i m a t e f e li

Original: n e s i m il a r i t
Revised: n e s i m il a r i t

Original: y i d
Revised: y i d

Original: e n t i t
Revised: e n t i t

Original: y e q u a li t y t
Revised: y e q u a li t y t

Original: i g e r c a t n a t u r a l _ o b
Revised: i g e r c a t n a t u r a l _ o b

Original: j e c t li v i n g _
Revised: j e c t li v i n g _

Original: t h i n g
Revised: t h i n g

Original: [Figure]
Revised: [Figure]

Original: o _
Revised: o _

Original: [Figure]
Revised: [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: e
Revised: e

Original: [Figure]
Revised: [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: a t
Revised: a t

Original: [Figure]
Revised: [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: a n i m a
Revised: a n i m a

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: r i b u t e q u a li t y s a l a m a n d e r s n a k e m a m m a l f e li n e s p
Revised: r i b u t e q u a li t y s a l a m a n d e r s n a k e m a m m a l f e li n e s p

Original: i n a c h b i r d r e p t il e c a l c u l u s a b s t r a c t
Revised: i n a c h b i r d r e p t il e c a l c u l u s a b s t r a c t

Original: i
Revised: i

Original: o
Revised: o

Original: n t u r t l e c h
Revised: n t u r t l e c h

Original: i c k e n e q u a li t y m o n o t r e m
Revised: i c k e n e q u a li t y m o n o t r e m

Original: e s i m il a r i t
Revised: e s i m il a r i t

Original: y w o o d
Revised: y w o o d

Original: y _ p l a n t li v i n g _
Revised: y _ p l a n t li v i n g _

Original: t h i n g t
Revised: t h i n g t

Original: i g e r li k e n e s s o n e n e s s p r
Revised: i g e r li k e n e s s o n e n e s s p r

Original: i m a t e l e tt u
Revised: i m a t e l e tt u

Original: c e o b
Revised: c e o b

Original: j e c t h e r b p l a n t a p p r o x
Revised: j e c t h e r b p l a n t a p p r o x

Original: i m a t i o n s a m e n e s
Revised: i m a t i o n s a m e n e s

Original: s c r o c
Revised: s c r o c

Original: o d il e p a rr o t m o n k e y
Revised: o d il e p a rr o t m o n k e y

Original: (a
Revised: (a

Original: Word2Vec f r o g c r y s t a l a m p h
Revised: Word2Vec f r o g c r y s t a l a m p h

Original: i b i a n h e r b p a l m r o c k c a n a r y c r o c o d il e e n t
Revised: i b i a n h e r b p a l m r o c k c a n a r y c r o c o d il e e n t

Original: i t y l e tt u
Revised: i t y l e tt u

Original: c
Revised: c

Original: e m o n o t r e m e p
Revised: e m o n o t r e m e p

Original: i n e c a t m o n k e y p a rr o t s a m e n e s s s p
Revised: i n e c a t m o n k e y p a rr o t s a m e n e s s s p

Original: i n a c h h o m o _ s a p i e n s c
Revised: i n a c h h o m o _ s a p i e n s c

Original: h
Revised: h

Original: i c k e n o n e n e s s f e li n e a p p r
Revised: i c k e n o n e n e s s f e li n e a p p r

Original: o x
Revised: o x

Original: i m a t i o n li v i n g _
Revised: i m a t i o n li v i n g _

Original: t h i n g a lli g a t
Revised: t h i n g a lli g a t

Original: o r s a l a m a n d e r p l a n t i d e n t
Revised: o r s a l a m a n d e r p l a n t i d e n t

Original: i t
Revised: i t

Original: y p
Revised: y p

Original: r
Revised: r

Original: i m a t e a n i m a l c a l c u l u s m a m m a l p l a t y p u
Revised: i m a t e a n i m a l c a l c u l u s m a m m a l p l a t y p u

Original: s o b
Revised: s o b

Original: j e c t s i m il a r i t
Revised: j e c t s i m il a r i t

Original: y t i g
Revised: y t i g

Original: e r q u a li t y a b s t r a c t i o n t u r t l e li k e n e s s n a t
Revised: e r q u a li t y a b s t r a c t i o n t u r t l e li k e n e s s n a t

Original: u
Revised: u

Original: r a l _ o b
Revised: r a l _ o b

Original: j e c t e q u a li t y r e p t il e s n a k e a tt r i b u t e b i r d w o o d y _ p l a n t e q u a li t y s a l a m a n d e r li v i n g _
Revised: j e c t e q u a li t y r e p t il e s n a k e a tt r i b u t e b i r d w o o d y _ p l a n t e q u a li t y s a l a m a n d e r li v i n g _

Original: t h i n g f r o g t
Revised: t h i n g f r o g t

Original: i g e r o b
Revised: i g e r o b

Original: j e c t s a m e n e s s p l a n t a tt r i b u t e a m p h
Revised: j e c t s a m e n e s s p l a n t a tt r i b u t e a m p h

Original: i b i a n r o c k p
Revised: i b i a n r o c k p

Original: r
Revised: r

Original: i m a t e a n i m a l m a m m a l
Revised: i m a t e a n i m a l m a m m a l

Original: h e r b s i m il a r i t y c a t t u r t l e r e p t il e li k e
Revised: h e r b s i m il a r i t y c a t t u r t l e r e p t il e li k e

Original: n e s s o n e n e s s n a t u r a l _ o b j e c t c a l c u l u s c r y s t a l m o n o t
Revised: n e s s o n e n e s s n a t u r a l _ o b j e c t c a l c u l u s c r y s t a l m o n o t

Original: r
Revised: r

Original: e m e s n a k
Revised: e m e s n a k

Original: e
Revised: e

Original: m o n k e y a p p r
Revised: m o n k e y a p p r

Original: o x
Revised: o x

Original: i m a t i o
Revised: i m a t i o

Original: n a b s t r a c t i o n c a n a r y q u a li t y i d e n t
Revised: n a b s t r a c t i o n c a n a r y q u a li t y i d e n t

Original: i t y p
Revised: i t y p

Original: i n e p a r r o t s p i n a c h f e li n e w o o d y _ p l a n t b i r d c h
Revised: i n e p a r r o t s p i n a c h f e li n e w o o d y _ p l a n t b i r d c h

Original: i c k e n h
Revised: i c k e n h

Original: o m o _ s a p i e n s a lli g a t
Revised: o m o _ s a p i e n s a lli g a t

Original: o r c r o c
Revised: o r c r o c

Original: o d il e p a l m l e tt u c e p l a t
Revised: o d il e p a l m l e tt u c e p l a t

Original: y p u s e n t i t
Revised: y p u s e n t i t

Original: y m o n
Revised: y m o n

Original: o t r e m e o n e n e s s l i k e n e s s h
Revised: o t r e m e o n e n e s s l i k e n e s s h

Original: o m o _ s a p i e n s p
Revised: o m o _ s a p i e n s p

Original: r
Revised: r

Original: i m a t e f e l i n e p i n e c a l c u l u
Revised: i m a t e f e l i n e p i n e c a l c u l u

Original: s e n t i t y a n i m a l a b s t r a c t
Revised: s e n t i t y a n i m a l a b s t r a c t

Original: i
Revised: i

Original: o
Revised: o

Original: n r e p t i l e p l a n t s i m i
Revised: n r e p t i l e p l a n t s i m i

Original: l a r i t
Revised: l a r i t

Original: y n a t u r a l _ o b
Revised: y n a t u r a l _ o b

Original: j e c t h e r b e q u a l i t y a m p h
Revised: j e c t h e r b e q u a l i t y a m p h

Original: i b i a n q u a l i t y o
Revised: i b i a n q u a l i t y o

Original: b
Revised: b

Original: j e c t l i v i n g _
Revised: j e c t l i v i n g _

Original: t h i n g a p p r o x
Revised: t h i n g a p p r o x

Original: i m a t i o n m a m m a l b i r d a tt r i b
Revised: i m a t i o n m a m m a l b i r d a tt r i b

Original: u t e i d e n t
Revised: u t e i d e n t

Original: i t y p
Revised: i t y p

Original: a l m t
Revised: a l m t

Original: i g e r c a n a r y c r o c o d i
Revised: i g e r c a n a r y c r o c o d i

Original: l e
Revised: l e

Original: s p
Revised: s p

Original: i n a c h l e tt u c e c
Revised: i n a c h l e tt u c e c

Original: h
Revised: h

Original: i c k e n a l l i g a t
Revised: i c k e n a l l i g a t

Original: o r t u r t l e p l a t y p u s s n a k e c a t f r o g m o n k e y r
Revised: o r t u r t l e p l a t y p u s s n a k e c a t f r o g m o n k e y r

Original: o
Revised: o

Original: c
Revised: c

Original: k s a l a m a n d e r w o o d y _
Revised: k s a l a m a n d e r w o o d y _

Original: p l a n t c
Revised: p l a n t c

Original: r y s t a l s a m e n e s s p a rr o t
Revised: r y s t a l s a m e n e s s p a rr o t

Original: [Figure]
Revised: [Figure]

Original: c a n a r y p l a n t a m p h i b i a n b i r d l e tt u c e w
Revised: c a n a r y p l a n t a m p h i b i a n b i r d l e tt u c e w

Original: o o
Revised: o o

Original: d y _ p l a n t r e p t il e p l a t y p u s c a t m o n k e y h e r b t
Revised: d y _ p l a n t r e p t il e p l a t y p u s c a t m o n k e y h e r b t

Original: i g e r n a t u r a l _ o b
Revised: i g e r n a t u r a l _ o b

Original: j e c t a b s t r a c t i o n a n i m a l o
Revised: j e c t a b s t r a c t i o n a n i m a l o

Original: b j e c t a tt r i b
Revised: b j e c t a tt r i b

Original: u t e e n t i t y s a m e n e s s a p p r
Revised: u t e e n t i t y s a m e n e s s a p p r

Original: o x
Revised: o x

Original: i m a t i o n o n e n e s s t u r t l e f e li n e h o m o _ s a p i e n s c
Revised: i m a t i o n o n e n e s s t u r t l e f e li n e h o m o _ s a p i e n s c

Original: h
Revised: h

Original: i c k e n m o n o t r e m e s a l a m a n d e r p a rr o t p r
Revised: i c k e n m o n o t r e m e s a l a m a n d e r p a rr o t p r

Original: i m a t e c r o c
Revised: i m a t e c r o c

Original: o d il e s p i n a c h p a l m p i n e a lli g a t o r li v i n g _
Revised: o d il e s p i n a c h p a l m p i n e a lli g a t o r li v i n g _

Original: t
Revised: t

Original: h
Revised: h

Original: i n g f r o g m a m m a l s n a k e c r y s t a l r o c k c a l c u l u s li k e n e s s s i m il a r i t
Revised: i n g f r o g m a m m a l s n a k e c r y s t a l r o c k c a l c u l u s li k e n e s s s i m il a r i t

Original: y i d
Revised: y i d

Original: e n t i t
Revised: e n t i t

Original: y e q u a li t y q u a li t y t
Revised: y e q u a li t y q u a li t y t

Original: u r t l e f e li n e i d e n t i t
Revised: u r t l e f e li n e i d e n t i t

Original: y a p p r o x
Revised: y a p p r o x

Original: i m a t i o n t
Revised: i m a t i o n t

Original: i g e r s a l a m a n d e r p a l m p r
Revised: i g e r s a l a m a n d e r p a l m p r

Original: i m a t
Revised: i m a t

Original: e s a m e n e s s o n e n e s s s i m il a r i t
Revised: e s a m e n e s s o n e n e s s s i m il a r i t

Original: y e q u a li t y e n t
Revised: y e q u a li t y e n t

Original: i t y li
Revised: i t y li

Original: k e n e s s l e tt u c e s p
Revised: k e n e s s l e tt u c e s p

Original: i n a c h w o o d y _ p l a n t h e r b r o c k c a l c u l u s c r y s t a l q u a li t
Revised: i n a c h w o o d y _ p l a n t h e r b r o c k c a l c u l u s c r y s t a l q u a li t

Original: y o b j e c t a b s t r a c t i o n n a t u r a l _ o b
Revised: y o b j e c t a b s t r a c t i o n n a t u r a l _ o b

Original: j e c t a n i m a l a tt r i b
Revised: j e c t a n i m a l a tt r i b

Original: u t e p l a n t b i r d r e p t il e c r o c o d il e c a n a r y c a t a lli g a t o r p a rr o t p i n e c h
Revised: u t e p l a n t b i r d r e p t il e c r o c o d il e c a n a r y c a t a lli g a t o r p a rr o t p i n e c h

Original: i c k e n m o n o t r e m e p l a t y p u s a m p
Revised: i c k e n m o n o t r e m e p l a t y p u s a m p

Original: h
Revised: h

Original: i b i a n s n a k e m o n k e y f r o g h
Revised: i b i a n s n a k e m o n k e y f r o g h

Original: o m o _ s a p i e n s m a m m a l li v i n g _
Revised: o m o _ s a p i e n s m a m m a l li v i n g _

Original: t h i n g t
Revised: t h i n g t

Original: i g e r f e li n e c r y s t
Revised: i g e r f e li n e c r y s t

Original: a l p l a n t n a t
Revised: a l p l a n t n a t

Original: u
Revised: u

Original: r a l _ o b
Revised: r a l _ o b

Original: j e c t s a m e n e s s a lli g a t
Revised: j e c t s a m e n e s s a lli g a t

Original: o r p r
Revised: o r p r

Original: i m a t e w o o d y _
Revised: i m a t e w o o d y _

Original: p l a n t s i m il a r i t
Revised: p l a n t s i m il a r i t

Original: y a n i m a l m o n k e y p a rr
Revised: y a n i m a l m o n k e y p a rr

Original: o t
Revised: o t

Original: r
Revised: r

Original: o
Revised: o

Original: c
Revised: c

Original: k c r o c o d il e s n a k
Revised: k c r o c o d il e s n a k

Original: e
Revised: e

Original: f r o g c a n a r y c a t r e p t il e c h
Revised: f r o g c a n a r y c a t r e p t il e c h

Original: i c k e n p l a t y p u
Revised: i c k e n p l a t y p u

Original: s s p i n a c h b i r d o n e n e s s m o n o t r e m e h e r b c a l c u l u s q u a li t y li k e n e s s a p p r
Revised: s s p i n a c h b i r d o n e n e s s m o n o t r e m e h e r b c a l c u l u s q u a li t y li k e n e s s a p p r

Original: o x
Revised: o x

Original: i m a t i o
Revised: i m a t i o

Original: n a b s t r a c t i o n a tt r i b
Revised: n a b s t r a c t i o n a tt r i b

Original: u t e l e tt u c e li v i n g _ t
Revised: u t e l e tt u c e li v i n g _ t

Original: h
Revised: h

Original: i n g o b j e c t e n t i t y p a l m e q u a li t
Revised: i n g o b j e c t e n t i t y p a l m e q u a li t

Original: y i d e n t i t y a m p
Revised: y i d e n t i t y a m p

Original: h
Revised: h

Original: i b i a n h o m o
Revised: i b i a n h o m o

Original: s a p i e n s t u r t l e p
Revised: s a p i e n s t u r t l e p

Original: i n e s a l a m a n d e r m a m m a l (
Revised: i n e s a l a m a n d e r m a m m a l (

Original: [Figure]
Revised: [Figure]

Original: ) T m o n o t r e m e f r o g c h
Revised: ) T m o n o t r e m e f r o g c h

Original: i c k e n s n a k e t u r t l e i d e n t
Revised: i c k e n s n a k e t u r t l e i d e n t

Original: i t
Revised: i t

Original: y e q u a li t y l e tt u c e c a t f e li n e p i n e m o n k e y a m p h
Revised: y e q u a li t y l e tt u c e c a t f e li n e p i n e m o n k e y a m p h

Original: i b i a n q u a li t y a b s t r a c t
Revised: i b i a n q u a li t y a b s t r a c t

Original: i
Revised: i

Original: o
Revised: o

Original: n a tt r i b u t e li k e n e s s li v i n g _
Revised: n a tt r i b u t e li k e n e s s li v i n g _

Original: t h
Revised: t h

Original: i n g o b j e c t a n i m a l s i m il a r i t y w o o d y _ p l a n t p l a n t s a m e n e s s b i r d c r y s t a l a p p r o x
Revised: i n g o b j e c t a n i m a l s i m il a r i t y w o o d y _ p l a n t p l a n t s a m e n e s s b i r d c r y s t a l a p p r o x

Original: i m a t i o n
Revised: i m a t i o n

Original: h e r b r e p t il e r
Revised: h e r b r e p t il e r

Original: o
Revised: o

Original: c
Revised: c

Original: k
Revised: k

Original: o n e n e s s m a m m a l n a t u r a l _ o b
Revised: o n e n e s s m a m m a l n a t u r a l _ o b

Original: j e c t p r i m a t e a l l i g a t
Revised: j e c t p r i m a t e a l l i g a t

Original: o r p a r r o t t i g
Revised: o r p a r r o t t i g

Original: e r c a n a r y c r o c o d i
Revised: e r c a n a r y c r o c o d i

Original: l e p l a t y p u
Revised: l e p l a t y p u

Original: s s a l a m a n d e r c a l c u l u s p a l m h
Revised: s s a l a m a n d e r c a l c u l u s p a l m h

Original: o m o _ s a p i e n s s p i n a c h e n t i t y
Revised: o m o _ s a p i e n s s p i n a c h e n t i t y

Original: [Figure]
Revised: [Figure]

Original: a m p h
Revised: a m p h

Original: i b i a n s n a k e m o n k e y a lli g a t
Revised: i b i a n s n a k e m o n k e y a lli g a t

Original: o r f e li n e f r o g m o n o t r e m e p l a t y p u s p
Revised: o r f e li n e f r o g m o n o t r e m e p l a t y p u s p

Original: r
Revised: r

Original: i m a t e p
Revised: i m a t e p

Original: a rr o t b i r d h o m o _ s a p i e n s r e p t il e s p i n a c h p a l m l e
Revised: a rr o t b i r d h o m o _ s a p i e n s r e p t il e s p i n a c h p a l m l e

Original: tt u c e c r o c o d il e p i n e r
Revised: tt u c e c r o c o d il e p i n e r

Original: o
Revised: o

Original: c
Revised: c

Original: k c r y s t a l a n i m a l s a m e n e s s o n e n e s s e q u a li t y a b s t r a c t
Revised: k c r y s t a l a n i m a l s a m e n e s s o n e n e s s e q u a li t y a b s t r a c t

Original: i o n i d e n t i t
Revised: i o n i d e n t i t

Original: y s i m il a r i t
Revised: y s i m il a r i t

Original: y li
Revised: y li

Original: k e n e s s a p p r
Revised: k e n e s s a p p r

Original: o x
Revised: o x

Original: i m a t i o n h e r b c h
Revised: i m a t i o n h e r b c h

Original: i c k e n c a n a r y w o o d y _
Revised: i c k e n c a n a r y w o o d y _

Original: p l a n t n a t u r a l _ o b j e c t t i g
Revised: p l a n t n a t u r a l _ o b j e c t t i g

Original: e r t u r t
Revised: e r t u r t

Original: l e s a l a m a n d e r e n t
Revised: l e s a l a m a n d e r e n t

Original: i t y a tt r i b u t e li v i n g _
Revised: i t y a tt r i b u t e li v i n g _

Original: t h
Revised: t h

Original: i n g o b j e c t c a t p l a n t m a m m a l q u a li t y c a l c u l u s
Revised: i n g o b j e c t c a t p l a n t m a m m a l q u a li t y c a l c u l u s

Original: [Table]
Revised: [Table]

Original: a tt r i b u t e c a n a r y c a l c u l u s q u a li t y t
Revised: a tt r i b u t e c a n a r y c a l c u l u s q u a li t y t

Original: u r t l e f e li n e s n a k e r e p t il e p
Revised: u r t l e f e li n e s n a k e r e p t il e p

Original: r
Revised: r

Original: i m a t e a m p
Revised: i m a t e a m p

Original: h
Revised: h

Original: i b i a n m o n
Revised: i b i a n m o n

Original: o t r e m e b i r d w o o d y _ p l a n t h o m o _ s a p i e n s p a rr o t m o n k e y s i m il a r i t
Revised: o t r e m e b i r d w o o d y _ p l a n t h o m o _ s a p i e n s p a rr o t m o n k e y s i m il a r i t

Original: y i d
Revised: y i d

Original: e n t i t
Revised: e n t i t

Original: y e q u a li t y li
Revised: y e q u a li t y li

Original: k e n e s s a b s t r a c t
Revised: k e n e s s a b s t r a c t

Original: i
Revised: i

Original: o
Revised: o

Original: n a p p r
Revised: n a p p r

Original: o
Revised: o

Original: x
Revised: x

Original: i m a t i o n o n e n e s s t
Revised: i m a t i o n o n e n e s s t

Original: i g e r p l a t y p u
Revised: i g e r p l a t y p u

Original: s s p i n a c h s a l a m a n d e r li v i n g _ t h i n g r o c k c r y s t a l a n i m a l o b j e c t s a m e n e s s f r o g c h
Revised: s s p i n a c h s a l a m a n d e r li v i n g _ t h i n g r o c k c r y s t a l a n i m a l o b j e c t s a m e n e s s f r o g c h

Original: i c k e n p a l m h e r b p i n e l e tt u c e e n t
Revised: i c k e n p a l m h e r b p i n e l e tt u c e e n t

Original: i t y a lli g a t
Revised: i t y a lli g a t

Original: o r c r o c
Revised: o r c r o c

Original: o d il e m a m m a l p l a n t n a t u r a l _ o b j e c t c a t (b) ELMo-large reconstruction using TIM q u a li t
Revised: o d il e m a m m a l p l a n t n a t u r a l _ o b j e c t c a t (b) ELMo-large reconstruction using TIM q u a li t

Original: y o n e n e s s s i m il a r i t
Revised: y o n e n e s s s i m il a r i t

Original: y s a m e n e s s i d e n t
Revised: y s a m e n e s s i d e n t

Original: i t y li
Revised: i t y li

Original: k e n e s s e q u a li t
Revised: k e n e s s e q u a li t

Original: y h e r b c h
Revised: y h e r b c h

Original: i c k e n w o o d y _ p l a n t r
Revised: i c k e n w o o d y _ p l a n t r

Original: o
Revised: o

Original: c k c r y s t a l li v i n g _
Revised: c k c r y s t a l li v i n g _

Original: t h i n g a n i m a l c a t r e p t il e m a m m a l p l a n t b i r d e n t
Revised: t h i n g a n i m a l c a t r e p t il e m a m m a l p l a n t b i r d e n t

Original: i t y p
Revised: i t y p

Original: a l m p i n e s p i n a c
Revised: a l m p i n e s p i n a c

Original: h t i g e r p a rr o t n a t u r a l _ o b j e c t o b j e c t a b s t r a c t i o n a p p r o x i m a t i o n a tt r i b
Revised: h t i g e r p a rr o t n a t u r a l _ o b j e c t o b j e c t a b s t r a c t i o n a p p r o x i m a t i o n a tt r i b

Original: u t e c a l c u l u s s a l a m a n d e r p l a t
Revised: u t e c a l c u l u s s a l a m a n d e r p l a t

Original: y p u
Revised: y p u

Original: s m o n k e
Revised: s m o n k e

Original: y c r o c o d il e a lli g a t
Revised: y c r o c o d il e a lli g a t

Original: o r p r
Revised: o r p r

Original: i m a t e h o m o _ s a p i e n s f e li n e s n a k e
Revised: i m a t e h o m o _ s a p i e n s f e li n e s n a k e

Original: a m p
Revised: a m p

Original: h
Revised: h

Original: i b i a n m o n
Revised: i b i a n m o n

Original: o t r e m e f r o g t u r t l e l e tt u c e c a n a r y a b s t r a c t i o n a tt r i b u t e n a t u r a l _ o b j e c t o b j e c t w o o d y _ p l a n t h e r b m o n k e y s n a k e p
Revised: o t r e m e f r o g t u r t l e l e tt u c e c a n a r y a b s t r a c t i o n a tt r i b u t e n a t u r a l _ o b j e c t o b j e c t w o o d y _ p l a n t h e r b m o n k e y s n a k e p

Original: r
Revised: r

Original: i m a t e c h
Revised: i m a t e c h

Original: i c k e n s i m il a r i t
Revised: i c k e n s i m il a r i t

Original: y c r y s t a l li v i n g _
Revised: y c r y s t a l li v i n g _

Original: t h i n g e n t
Revised: t h i n g e n t

Original: i t y c a l c u l u s h o m o _ s a p i e n s a m p h
Revised: i t y c a l c u l u s h o m o _ s a p i e n s a m p h

Original: i b i a n l e tt u c e p l a t y p
Revised: i b i a n l e tt u c e p l a t y p

Original: u s f r o g f e li n e p a l m p i n e t u r t
Revised: u s f r o g f e li n e p a l m p i n e t u r t

Original: l e li k e n e s s e q u a li t y s a m e n e s s o n e n e s s a p p r
Revised: l e li k e n e s s e q u a li t y s a m e n e s s o n e n e s s a p p r

Original: o x
Revised: o x

Original: i m a t i o n a lli g a t
Revised: i m a t i o n a lli g a t

Original: o r c a n a r y s p
Revised: o r c a n a r y s p

Original: i n a c h i d e n t
Revised: i n a c h i d e n t

Original: i t
Revised: i t

Original: y a n i m a l r o c k b i r d m a m m a l p l a n t r e p t
Revised: y a n i m a l r o c k b i r d m a m m a l p l a n t r e p t

Original: i l e q u a li t
Revised: i l e q u a li t

Original: y s a l a m a n d e r t i g e r c a t p a rr o t m o n o t r e m e c
Revised: y s a l a m a n d e r t i g e r c a t p a rr o t m o n o t r e m e c

Original: r o c o d il e q u a li t y a b s t r a c t i o n li k e n e s s e n t i t
Revised: r o c o d il e q u a li t y a b s t r a c t i o n li k e n e s s e n t i t

Original: y m o n
Revised: y m o n

Original: o t r e m e a m p h
Revised: o t r e m e a m p h

Original: i b i a n m o n k e
Revised: i b i a n m o n k e

Original: y
Revised: y

Original: h
Revised: h

Original: o m o _ s a p i e n s t u r t l e f r o g f e li n e a p p r
Revised: o m o _ s a p i e n s t u r t l e f r o g f e li n e a p p r

Original: o
Revised: o

Original: x
Revised: x

Original: i m a t i o n e q u a li t
Revised: i m a t i o n e q u a li t

Original: y o n e n e s s i d e n t
Revised: y o n e n e s s i d e n t

Original: i t
Revised: i t

Original: y s i m il a r i t
Revised: y s i m il a r i t

Original: y s a m e n e s s c r o c o d il e t
Revised: y s a m e n e s s c r o c o d il e t

Original: i g e r p a rr o t s a l a m a n d e r a lli g a t
Revised: i g e r p a rr o t s a l a m a n d e r a lli g a t

Original: o r s p i n
Revised: o r s p i n

Original: a c h s n a k e c h
Revised: a c h s n a k e c h

Original: i c k e n m a m m a l h e r b c a t r e p t il e p r
Revised: i c k e n m a m m a l h e r b c a t r e p t il e p r

Original: i m a t e p l a n t w o o d y _
Revised: i m a t e p l a n t w o o d y _

Original: p l a n t b i r d c r y s t a l c a l c u l u s r o c k li v i n g _ t h i n g o b j e c t n a t u r a l _ o b j e c t a tt r i b u t e l e tt u c e c a n a r y p i n e p a l m a n i m a l p l a t y p u s t i g e r e n t
Revised: p l a n t b i r d c r y s t a l c a l c u l u s r o c k li v i n g _ t h i n g o b j e c t n a t u r a l _ o b j e c t a tt r i b u t e l e tt u c e c a n a r y p i n e p a l m a n i m a l p l a t y p u s t i g e r e n t

Original: i t y a b s t r a c t
Revised: i t y a b s t r a c t

Original: i o n q u a li t y s i m il a r i t
Revised: i o n q u a li t y s i m il a r i t

Original: y m a
Revised: y m a

Original: m
Revised: m

Original: m
Revised: m

Original: a l r e p t il e b i r d w o o d y _ p l a n t p l a n t h e r bp a l m l e tt u c e n a t u r a l _ o b j e c t p l a t y p u s e q u a li t y i d e n t i t y li k e n e s s a n i m a l c a n a r y s p
Revised: a l r e p t il e b i r d w o o d y _ p l a n t p l a n t h e r bp a l m l e tt u c e n a t u r a l _ o b j e c t p l a t y p u s e q u a li t y i d e n t i t y li k e n e s s a n i m a l c a n a r y s p

Original: i n a c h a tt r i b u t e r o c k c r y s t a l o b j e c t c a l c u l u s li v i n g _ t h i n g a m p h
Revised: i n a c h a tt r i b u t e r o c k c r y s t a l o b j e c t c a l c u l u s li v i n g _ t h i n g a m p h

Original: i b i a n a p p r o x
Revised: i b i a n a p p r o x

Original: i m a t i o n t u r t l e c h
Revised: i m a t i o n t u r t l e c h

Original: i c k e n p i n e f r o g f e li n e c a t m o n k e
Revised: i c k e n p i n e f r o g f e li n e c a t m o n k e

Original: y m o n o t
Revised: y m o n o t

Original: r
Revised: r

Original: e m e p r
Revised: e m e p r

Original: i m a t
Revised: i m a t

Original: e s n a k e p a rr o t a lli g a t
Revised: e s n a k e p a rr o t a lli g a t

Original: o r s a l a m a n d e r c r o c o d il e h o m o _ s a p i e n s o n e n e s s s a m e n e s s
Revised: o r s a l a m a n d e r c r o c o d il e h o m o _ s a p i e n s o n e n e s s s a m e n e s s

Original: [Figure]
Revised: [Figure]

Original: c a n a r y f e li n e s a l a m a n d e r a lli g a t o r m o n k e y t
Revised: c a n a r y f e li n e s a l a m a n d e r a lli g a t o r m o n k e y t

Original: i g e r p a rr o t p l a t y p u s c a t f r
Revised: i g e r p a rr o t p l a t y p u s c a t f r

Original: o g c r
Revised: o g c r

Original: o c o d il e t u r t l e c h
Revised: o c o d il e t u r t l e c h

Original: i c k e n e n t
Revised: i c k e n e n t

Original: i t y a m p
Revised: i t y a m p

Original: h
Revised: h

Original: i b i a n p r
Revised: i b i a n p r

Original: i m a t e b i r d r e p t il e p l a n t m a m m a l s n a k e m o n
Revised: i m a t e b i r d r e p t il e p l a n t m a m m a l s n a k e m o n

Original: o t r e m e a b s t r a c t i o n q u a li t y a p p r o x i m a t i o n s p i n a c h p a l m
Revised: o t r e m e a b s t r a c t i o n q u a li t y a p p r o x i m a t i o n s p i n a c h p a l m

Original: h e r b l e tt u c e p
Revised: h e r b l e tt u c e p

Original: i n e w o o d y _ p l a n t o b
Revised: i n e w o o d y _ p l a n t o b

Original: j e c t n a t u r a l _ o b
Revised: j e c t n a t u r a l _ o b

Original: j e c t a tt r i b u t e li v i n g _
Revised: j e c t a tt r i b u t e li v i n g _

Original: t
Revised: t

Original: h
Revised: h

Original: i n g h
Revised: i n g h

Original: o m o _ s a p i e n s s i m il a r i t
Revised: o m o _ s a p i e n s s i m il a r i t

Original: y li
Revised: y li

Original: k e n e s s i d e n t
Revised: k e n e s s i d e n t

Original: i t y s a m e n e s s o n e n e s s e q u a li t y a n i m a l r o c k c
Revised: i t y s a m e n e s s o n e n e s s e q u a li t y a n i m a l r o c k c

Original: a l c u l u s c r y s t a l
Revised: a l c u l u s c r y s t a l

Original: [Table]
Revised: [Table]

Original: h e r b w o o d y _
Revised: h e r b w o o d y _

Original: p l a n t c a t m o n o t
Revised: p l a n t c a t m o n o t

Original: r
Revised: r

Original: e m e f e li n e c a n a r y s a l a m a n d e r a lli g a t
Revised: e m e f e li n e c a n a r y s a l a m a n d e r a lli g a t

Original: o r s p i n
Revised: o r s p i n

Original: a c h c r o c o d il e p l a t y p u
Revised: a c h c r o c o d il e p l a t y p u

Original: s c a l c u l u s c r y s t a l li v i n g _
Revised: s c a l c u l u s c r y s t a l li v i n g _

Original: t h i n g a n i m a l n a t u r a l _ o b j e c t a tt r i b
Revised: t h i n g a n i m a l n a t u r a l _ o b j e c t a tt r i b

Original: u t e
Revised: u t e

Original: o
Revised: o

Original: b
Revised: b

Original: j e c t t i g
Revised: j e c t t i g

Original: e r h o m o _ s a p i e n s a b s t r a c t i o n q u a li t y i d e n t i t
Revised: e r h o m o _ s a p i e n s a b s t r a c t i o n q u a li t y i d e n t i t

Original: y a p p r o x
Revised: y a p p r o x

Original: i m a t i o n s i m il a r
Revised: i m a t i o n s i m il a r

Original: i t
Revised: i t

Original: y li
Revised: y li

Original: k e n e s s f r o g p a l m p i n e t u r t
Revised: k e n e s s f r o g p a l m p i n e t u r t

Original: l e c h
Revised: l e c h

Original: i c k e n l e tt u c e e n t i t y b i r d r o c k r e p t il e m a m m a l p l a n t p r i m a t e a m p
Revised: i c k e n l e tt u c e e n t i t y b i r d r o c k r e p t il e m a m m a l p l a n t p r i m a t e a m p

Original: h
Revised: h

Original: i b i a n p a rr o t m o n k e y s n a k e
Revised: i b i a n p a rr o t m o n k e y s n a k e

Original: s a m e n e s s o n e n e s s e q u a li t y
Revised: s a m e n e s s o n e n e s s e q u a li t y

Original: [Table]
Revised: [Table]

Original: e n t i t y c h
Revised: e n t i t y c h

Original: i c k e n t u r t
Revised: i c k e n t u r t

Original: l e f r o g p a l m c
Revised: l e f r o g p a l m c

Original: r
Revised: r

Original: o c o d i l e t i g e r m o n o t r e m e q u a li t y w o o d y _ p l a n t h e r b p
Revised: o c o d i l e t i g e r m o n o t r e m e q u a li t y w o o d y _ p l a n t h e r b p

Original: r
Revised: r

Original: i m a t e r e p t il e p l a t y p u
Revised: i m a t e r e p t il e p l a t y p u

Original: s a m p
Revised: s a m p

Original: h
Revised: h

Original: i b i a n s n a k e s a l a m a n d e r c a t c a l c u l u s r o c k l i v i n g _
Revised: i b i a n s n a k e s a l a m a n d e r c a t c a l c u l u s r o c k l i v i n g _

Original: t
Revised: t

Original: h
Revised: h

Original: i n g p l a n t c r y s t a l a n i m a l e q u a l i t y o n e n e s s a b s t r a c t
Revised: i n g p l a n t c r y s t a l a n i m a l e q u a l i t y o n e n e s s a b s t r a c t

Original: i
Revised: i

Original: o n o b
Revised: o n o b

Original: j e c t n a t u r a l _ o b
Revised: j e c t n a t u r a l _ o b

Original: j e c t a tt r i b u t e c a n a r y a l l i g a t
Revised: j e c t a tt r i b u t e c a n a r y a l l i g a t

Original: o r p
Revised: o r p

Original: i n e l e tt u c e l i k e n e s s i d e n t
Revised: i n e l e tt u c e l i k e n e s s i d e n t

Original: i t
Revised: i t

Original: y a p p r o x
Revised: y a p p r o x

Original: i m a t i o n s a m e n e s s
Revised: i m a t i o n s a m e n e s s

Original: s i m i l a r i t
Revised: s i m i l a r i t

Original: y s p
Revised: y s p

Original: i n a c h h o m o _ s a p i e n s p a rr o t m o n k e y b i r d m a m m a l f e l i n e
Revised: i n a c h h o m o _ s a p i e n s p a rr o t m o n k e y b i r d m a m m a l f e l i n e

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: s i m il a r i t
Revised: s i m il a r i t

Original: y a tt r i b u t e q u a li t y c h
Revised: y a tt r i b u t e q u a li t y c h

Original: i c k e n p a l m s a m e n e s s i d e n t i t y c r o c o d
Revised: i c k e n p a l m s a m e n e s s i d e n t i t y c r o c o d

Original: i l e p i n e t u r t l e h o m o
Revised: i l e p i n e t u r t l e h o m o

Original: s a p i e n s li k e n e s s e q u a li t y c a l c u l u s s n a k e p a rr o t m o n k e y c a t w o o d y _
Revised: s a p i e n s li k e n e s s e q u a li t y c a l c u l u s s n a k e p a rr o t m o n k e y c a t w o o d y _

Original: p l a n t f e li n e r e p t il e p r i m a t e a m p h
Revised: p l a n t f e li n e r e p t il e p r i m a t e a m p h

Original: i b i a n p l a n t h e r b a lli g a t
Revised: i b i a n p l a n t h e r b a lli g a t

Original: o r l e tt u c e p l a t y p u s f r o g t
Revised: o r l e tt u c e p l a t y p u s f r o g t

Original: i g e r s a l a m a n d e r n a t u r a l _
Revised: i g e r s a l a m a n d e r n a t u r a l _

Original: o
Revised: o

Original: b j e c t a b s t r a c t i o n c a n a r y a p p r o x i m a t i o n o n e n e s s m o n o t r e m e b i r d m a m m a l e n t i t
Revised: b j e c t a b s t r a c t i o n c a n a r y a p p r o x i m a t i o n o n e n e s s m o n o t r e m e b i r d m a m m a l e n t i t

Original: y s p
Revised: y s p

Original: i n a c h o b j e c t a n i m a l r o c k c r y s t a l li v i n g _
Revised: i n a c h o b j e c t a n i m a l r o c k c r y s t a l li v i n g _

Original: t h i n g
Revised: t h i n g

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: C F URTHER ANALYSIS OF FACTORS ’ IMPACT IN MODELS PERFORMANCE
Revised: C F URTHER ANALYSIS OF FACTORS ’ IMPACT IN MODELS PERFORMANCE

Revised: [Figure] [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: , and by their sense ranking (how frequent or rare those senses are).

Revised: Surprisingly contextualized models, and specially CE models have no signiﬁcant impact by this factor, suggesting that these models are very effective at deducing the correct sense based on their context.

Revised: These charts also suggest that these models may be considering context even more than the words themselves.

Revised: This is intuitive for Masked-Language-Models such as BERT, but not for others, such as GPT-2.

Revised: Non-contextualized models are impacted by this factor, as expected.

Revised: [Figure]

Revised: D A CCURACY COMPARISON

Revised: In Table 1 we analyzed F1 scores for each model and best layer.

Revised: Here we provide an analogous table but reporting accuracies instead of F1 scores.

Revised: [Table]

Revised: E P RE -T RAINING CORPUS COMPARISON

Revised: [Table]

Revised: F A RE ALL THESE MODELS CORRELATED ?

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: GPT- .

Revised: T .

Revised: ELMo .

Revised: [Table]

Revised: In this section, we study if concepts that are easy/hard to associate for one model are also easy/hard to associate for other models.

Revised: We do this by analyzing the correlation between outputs using the semantic probing classiﬁer.

Revised: Charts in Figure 18 show the correlation among the models by plotting the F1-score of one model against the rest.

Revised: We picked one model of each family: GloVe (Fig 18-left)

Revised: RoBERTa

Revised: Fig

Revised: Figure 18-left reveals a strong semantic correlation between Word2Vec and GloVe, but weaker with the rest.

Revised: Figure 18-middle shows a strong correlation between RoBERTa and other methods in its family, but weaker with models of the NCE family.

Revised: Fig 18-right shows that T5 has the same difﬁculties as the rest of the methods, even showing some correlation with the NCE family of methods.

Revised: The charts in Figure 18 reveal that models are strongly correlated, specially within the same model family, suggesting that similar approaches lead to similar biases, regardless of their size.

Revised: In other words, the concepts that were difﬁcult in the past are, to great extent, still difﬁcult for state-of-the-art models.

Revised: Thus, novel inductive biases are needed to get closer to human semantic capabilities.

Revised: [Table]