Original: Joint Learning of Hierarchical Word Embeddings from a Corpus and a Taxonomy
Revised: Joint Learning of Hierarchical Word Embeddings from a Corpus and a Taxonomy

Original: Abstract
Revised: Abstract

Original: Identifying the hypernym relations that hold between words is a fundamental task in NLP.
Revised: Identifying the hypernym relations that hold between words is a fundamental task in NLP.

Original: Word embedding methods have recently shown some capability to encode hypernymy.
Revised: Word embedding methods have recently shown some capability to encode hypernymy.

Original: However, such methods tend not to explicitly encode the hypernym hierarchy that exists between words.
Revised: However, such methods tend not to explicitly encode the hypernym hierarchy that exists between words.

Original: In this paper, we propose a method to learn a hierarchical word embedding in a speciﬁc order to capture the hypernymy.
Revised: In this paper, we propose a method to learn a hierarchical word embedding in a speciﬁc order to capture the hypernymy.

Original: To learn the word embeddings, the proposed method considers not only the hypernym relations that exists between words on a taxonomy, but also their contextual information in a large text corpus.
Revised: To learn the word embeddings, the proposed method considers not only the hypernym relations that exists between words on a taxonomy, but also their contextual information in a large text corpus.

Original: The experimental results on a supervised hypernymy detection and a newlyproposed hierarchical path completion tasks show the ability of the proposed method to encode the hierarchy.
Revised: The experimental results on a supervised hypernymy detection and a newlyproposed hierarchical path completion tasks show the ability of the proposed method to encode the hierarchy.

Original: Moreover, the proposed method outperforms previously proposed methods for learning word and hypernym-speciﬁc word embeddings on multiple benchmarks.
Revised: Moreover, the proposed method outperforms previously proposed methods for learning word and hypernym-speciﬁc word embeddings on multiple benchmarks.

Original: 1. Introduction
Revised: 1. Introduction

Original: Hypernymy relation is an essential component for many Natural Language Processing (NLP) tasks.
Revised: Hypernymy relation is an essential component for many Natural Language Processing (NLP) tasks.

Original: It represents an asymmetric relation between name of a class (hypernym) and a particular instance of it (hyponym) .
Revised: It represents an asymmetric relation between name of a class (hypernym) and a particular instance of it (hyponym) .

Original: For example, given the hypernymy pair (bird, vertebrate) , the hyponym word bird is a particular instance of the hypernym word vertebrate , and one can simply say “a bird is a vertebrate” .
Revised: For example, given the hypernymy pair (bird, vertebrate) , the hyponym word bird is a particular instance of the hypernym word vertebrate , and one can simply say “a bird is a vertebrate” .

Original: Hypernymy identiﬁcation plays a major role in various NLP tasks such as question answering [Huang et al., 2008], taxonomy construction [Navigli et al., 2011], textual entailment [Dagan et al., 2013] and text generation [Biran and McKeown, 2013], to name a few.
Revised: Hypernymy identiﬁcation plays a major role in various NLP tasks such as question answering [Huang et al., 2008], taxonomy construction [Navigli et al., 2011], textual entailment [Dagan et al., 2013] and text generation [Biran and McKeown, 2013], to name a few.

Original: The task of identifying hypernymy has been long addressed by relying on either the lexicalsyntactic patterns where a particular textual pattern suggests the existence of a hypernymy relation, or the distributional representation of a given pair of words.
Revised: The task of identifying hypernymy has been long addressed by relying on either the lexicalsyntactic patterns where a particular textual pattern suggests the existence of a hypernymy relation, or the distributional representation of a given pair of words.

Original: In a pattern-based approach, from a sentence like “a bird such as a falcon” we can identify a hypernymy relation between bird and falcon .
Revised: In a pattern-based approach, from a sentence like “a bird such as a falcon” we can identify a hypernymy relation between bird and falcon .

Original: Despite its simplicity and efﬁciency, pattern-based approaches suffer from a low precision and coverage issue [Yu et al., 2015].
Revised: Despite its simplicity and efﬁciency, pattern-based approaches suffer from a low precision and coverage issue [Yu et al., 2015].

Original: For example, let us consider the sentence “some birds recorded in
Revised: For example, let us consider the sentence “some birds recorded in

Original: Africa such as Gadwall” , a typical lexical-pattern based approach may incorrectly detect (Gadwall,
Revised: Africa such as Gadwall” , a typical lexical-pattern based approach may incorrectly detect (Gadwall,

Original: Africa) as hypernymy.
Revised: Africa) as hypernymy.

Original: In contrast to the requirement of the pattern-based approaches to have the hypernym and hyponym words occurring in the same sentence, the distributional family of approaches rely on the words co-occurrence statistics in a large text corpus to represent the words and then deduce the hold of hypernymy.
Revised: In contrast to the requirement of the pattern-based approaches to have the hypernym and hyponym words occurring in the same sentence, the distributional family of approaches rely on the words co-occurrence statistics in a large text corpus to represent the words and then deduce the hold of hypernymy.

Original: It works on the assumption that taxonomically related words tend to occur in a similar context.
Revised: It works on the assumption that taxonomically related words tend to occur in a similar context.

Original: In particular, a hypernym word has a broader context than its hypernym, and therefore the contextual features of the hyponym word are usually a subset of that of its hypernym.
Revised: In particular, a hypernym word has a broader context than its hypernym, and therefore the contextual features of the hyponym word are usually a subset of that of its hypernym.

Original: However, such approaches commonly struggle from discriminating the hypernym relations from other lexico-semantic relations [Shwartz et al., 2016b].
Revised: However, such approaches commonly struggle from discriminating the hypernym relations from other lexico-semantic relations [Shwartz et al., 2016b].

Original: Lately, distributed word representations (a.k.a word embeddings)
Revised: Lately, distributed word representations (a.k.a word embeddings)

Original: [Mikolov et al., 2013, Pennington et al., 2014] have shown some capability to encode hypernymy.
Revised: [Mikolov et al., 2013, Pennington et al., 2014] have shown some capability to encode hypernymy.

Original: Such methods typically embed the words into dense, low-dimensional, real-valued vectors, using the co-occurrence statistics obtained from a text corpora, and then used in a supervised settings to detect hypernymy.
Revised: Such methods typically embed the words into dense, low-dimensional, real-valued vectors, using the co-occurrence statistics obtained from a text corpora, and then used in a supervised settings to detect hypernymy.

Original: However, typical word embeddings models rely only on co-occurrence based similarity, and therefore are insufﬁcient to encode taxonomic relations in the learnt word embeddings
Revised: However, typical word embeddings models rely only on co-occurrence based similarity, and therefore are insufﬁcient to encode taxonomic relations in the learnt word embeddings

Original: [Anh et al., 2016].
Revised: [Anh et al., 2016].

Original: Several recent studies have proposed methods for learning hypernymy-speciﬁc word embeddings [Yu et al., 2015, Anh et al., 2016, Nguyen et al., 2017].
Revised: Several recent studies have proposed methods for learning hypernymy-speciﬁc word embeddings [Yu et al., 2015, Anh et al., 2016, Nguyen et al., 2017].

Original: [Yu et al., 2015] design a neural network to learn word embeddings that perceive hypernymy, purely relying on extracted pairwise training data from a web corpus.
Revised: Yu et al. [2015] design a neural network to learn word embeddings that perceive hypernymy, purely relying on extracted pairwise training data from a web corpus.

Original: Moreover, [Anh et al., 2016] proposed a similar approach, but further utilise the contextual information of the pre-extracted pairwise hypernymy.
Revised: Moreover, Anh et al. [2016] proposed a similar approach, but further utilise the contextual information of the pre-extracted pairwise hypernymy.

Original: Similarly, [Nguyen et al., 2017] introduce the HyperVec model that use the Skip-gram with Negative Sampling (SGNS)
Revised: Similarly, Nguyen et al. [2017] introduce the HyperVec model that use the Skip-gram with Negative Sampling (SGNS)

Original: [Mikolov et al., 2013] objective to learn the word embeddings from a corpus subject to pairwise hypernymy constraints extracted from a taxonomy.
Revised: [Mikolov et al., 2013] objective to learn the word embeddings from a corpus subject to pairwise hypernymy constraints extracted from a taxonomy.

Revised: Moreover, Nickel and Kiela [2017] proposed a new model that embed symbolic data into hyperbolic space, particularly into Poincar´e ball, to learn hierarchical embeddings.

Revised: Another model that is recently proposed is the LEAR model of Vuli´c and Mrkˇsi´c [2018].

Revised: LEAR is a post-processing model that takes any word vectors and then adjust the input vectors to emphasis the hypernym relations.

Original: The aforementioned methods merely consider pairwise hypernymy relations, ignoring the hierarchical hypernymy path connecting a word to the root in a taxonomy.
Revised: The aforementioned methods merely consider pairwise hypernymy relations rather than the hierarchical hypernymy path connecting a word to the root in a taxonomy.

Original: In this paper, we propose a method that jointly learns hierarchical word embeddings (HWE) from a corpus and a taxonomy.
Revised: In this paper, we propose a method that jointly learns hierarchical word embeddings (HWE) from a corpus and a taxonomy.

Original: The proposed method begins by embedding the words into random low-dimensional real-valued vectors, and subsequently updates the embeddings to encode the hierarchical structure available in the taxonomy.
Revised: The proposed method begins by embedding the words into random low-dimensional real-valued vectors, and subsequently updates the embeddings to encode the hierarchical structure available in the taxonomy.

Original: To train the proposed method, we use a taxonomy to extract the hierarchical hypernymy paths for the hyponym words, and use the global vector [Pennington et al., 2014] as a context-aware objective between the hypernym and hyponym words.
Revised: To train the proposed method, we use a taxonomy to extract the hierarchical hypernymy paths for the hyponym words, and use the global vector [Pennington et al., 2014] as a context-aware objective between the hypernym and hyponym words.

Original: As such, the proposed model beneﬁts from both the contextual information as well as the taxonomic relations to learn the embeddings.
Revised: As such, the proposed model beneﬁts from both the contextual information as well as the taxonomic relations to learn the embeddings.

Original: In our experiments, we evaluate our hierarchical word embeddings on the standard supervised hypernymy task on various benchmarks,  and a newly-proposed hierarchical path completion task.
Revised: In our experiments, we evaluate our hierarchical word embeddings on the standard supervised hypernymy task on various benchmarks, graded lexical entailment prediction and a newly-proposed hierarchical path completion task.

Original: On the supervised hypernymy identiﬁcation, the proposed method reports an improvement over several previously proposed methods, showing the beneﬁt of considering the full hierarchical hypernymy paths instead of only pairwise relations.
Revised: On the supervised hypernymy identiﬁcation, the proposed method reports an improvement over several previously proposed methods, showing the beneﬁt of considering the full hierarchical hypernymy paths instead of only pairwise relations.

Original: Moreover, a quantitative and qualitative analysis on the newly-proposed hierarchical path completion task further illustrate the ability of the proposed method to encode hypernymy in the learn embeddings.
Revised: Moreover, a quantitative and qualitative analysis on the newly-proposed hierarchical path completion task further illustrate the ability of the proposed method to encode hypernymy in the learn embeddings.

Original: 2. Related Work
Revised: 2. Related Work

Original: Over the years, two lines of approaches, pattern-based and distributional, have been predominantly followed in the NLP community for identifying the hypernymy relation between words.
Revised: Over the years, two lines of approaches, pattern-based and distributional, have been predominantly followed in the NLP community for identifying the hypernymy relation between words.

Original: The patternbased approach uses large text corpora to capture lexical-syntactic patterns with a high probability of representing a hypernymy relationship [Liu and Singh, 2004, Suchanek et al., 2007, Wu et al., 2012].
Revised: The patternbased approach uses large text corpora to capture lexical-syntactic patterns with a high probability of representing a hypernymy relationship [Liu and Singh, 2004, Suchanek et al., 2007, Wu et al., 2012].

Original: Popularly known as the Hearst’s patterns [Hearst, 1992] are an example of such patterns, for instance, the sentence “vertebrate such as birds” indicates that vertebrate is a hypernym of birds .
Revised: Popularly known as the Hearst’s patterns [Hearst, 1992] are an example of such patterns, for instance, the sentence “vertebrate such as birds” indicates that vertebrate is a hypernym of birds .

Original: Extracting such patterns paves the way to automatically build a taxonomy of hypernymy relations, where the task of identifying the existence of hypernymy relation between a given pair of words is then transformed into a problem of checking their existence in such taxonomies.
Revised: Extracting such patterns paves the way to automatically build a taxonomy of hypernymy relations, where the task of identifying the existence of hypernymy relation between a given pair of words is then transformed into a problem of checking their existence in such taxonomies.

Original: Unfortunately, pattern-based approaches are prone to data sparseness issues because two words must co-occur in the same context in order for a pattern to be extracted [Yu et al., 2015, Shwartz et al., 2016b].
Revised: Unfortunately, pattern-based approaches are prone to data sparseness issues because two words must co-occur in the same context in order for a pattern to be extracted [Yu et al., 2015, Shwartz et al., 2016b].

Original: Distributional methods, on the other hand, focus on the word distribution in text corpora to represent the words in a high dimensional vector space [Turney and Pantel, 2010], and rely on the
Revised: Distributional methods, on the other hand, focus on the word distribution in text corpora to represent the words in a high dimensional vector space [Turney and Pantel, 2010], and rely on the

Original: Distributional Inclusion Hypothesis (DIH)
Revised: Distributional Inclusion Hypothesis (DIH)

Original: [Geffet and Dagan, 2005] to identify the hypernymy relationships.
Revised: [Geffet and Dagan, 2005] to identify the hypernymy relationships.

Original: DIH works on the assumption that a signiﬁcant number of the distributional context features of the hyponym word are expected to be a subset of the hypernym word features.
Revised: DIH works on the assumption that a signiﬁcant number of the distributional context features of the hyponym word are expected to be a subset of the hypernym word features.

Original: For instance, given the pair (bird, vertebrate) , DIH assumes that most of the distributional features of bird are features of vertebrate as well, but does not necessarily true for the reverse.
Revised: For instance, given the pair (bird, vertebrate) , DIH assumes that most of the distributional features of bird are features of vertebrate as well, but does not necessarily true for the reverse.

Original: Such unsupervised distributional methods for hypernymy identiﬁcation include [Weeds et al., 2004, Clarke, 2009,
Revised: Such unsupervised distributional methods for hypernymy identiﬁcation include [Weeds et al., 2004, Clarke, 2009,

Original: Kotlerman et al., 2010, Lenci and Benotto, 2012].
Revised: Kotlerman et al., 2010, Lenci and Benotto, 2012].

Original: Pattern-based and distribution-based approaches have complementary strengths, hence some hybrid approaches have been proposed combining the strengths of the two approaches [Mirkin et al., 2006, Shwartz et al., 2016a].
Revised: Pattern-based and distribution-based approaches have complementary strengths, hence some hybrid approaches have been proposed combining the strengths of the two approaches [Mirkin et al., 2006, Shwartz et al., 2016a].

Original: Several methods have been proposed that use pre-trained word embeddings as the input and learn a classiﬁer that can predict whether a hypernym relation exist between a given pair of words.
Revised: Several methods have been proposed that use pre-trained word embeddings as the input and learn a classiﬁer that can predict whether a hypernym relation exist between a given pair of words.

Original: These methods can be seen as a combination between unsupervised and supervised methods in the sense that the pre-trained word emebddings used as the input can be learnt using unsupervised word embedding learning methods [Mikolov et al., 2013, Pennington et al., 2014], and the classiﬁer training part can be seen as a supervised (binary) classiﬁcation task, where the labelled instances for training can be retrieved from a taxonomy containing hypernyms-hyponyms.
Revised: These methods can be seen as a combination between unsupervised and supervised methods in the sense that the pre-trained word emebddings used as the input can be learnt using unsupervised word embedding learning methods [Mikolov et al., 2013, Pennington et al., 2014], and the classiﬁer training part can be seen as a supervised (binary) classiﬁcation task, where the labelled instances for training can be retrieved from a taxonomy containing hypernyms-hyponyms.

Original: An important research question in such approaches is how to represent the relationship between two words using their word embeddings?
Revised: An important research question in such approaches is how to represent the relationship between two words using their word embeddings?

Original: To address this point, several unsupervised operators that take in two vectors (corresponding to the embeddings of the two input words) and return a vector (corresponding to the relation between the two input words) have been studied in prior work such as concatenation [Baroni and Lenci, 2011], difference [Roller et al., 2014], addition [Weeds et al., 2014] and hybrid combinations [Turney and Mohammad, 2015].
Revised: To address this point, several unsupervised operators that take in two vectors (corresponding to the embeddings of the two input words) and return a vector (corresponding to the relation between the two input words) have been studied in prior work such as concatenation [Baroni and Lenci, 2011], difference [Roller et al., 2014], addition [Weeds et al., 2014] and hybrid combinations [Turney and Mohammad, 2015].

Original: Although numerous work have reported good hypernym detection accuracies using such supervised methods [Roller et al., 2014, Weeds et al., 2014,
Revised: Although numerous work have reported good hypernym detection accuracies using such supervised methods [Roller et al., 2014, Weeds et al., 2014,

Original: Kruszewski et al., 2015, Sanchez and Riedel, 2017, Vuli´c et al., 2017].
Revised: Kruszewski et al., 2015, Sanchez and Riedel, 2017, Vuli´c et al., 2017].

Original: [Levy et al., 2015b] raise doubts as to whether these methods are truly learning the hypernymy relation or simply memorising prototypical hypernyms.
Revised: [Levy et al., 2015b] raise doubts as to whether these methods are truly learning the hypernymy relation or simply memorising prototypical hypernyms.

Original: More recently, a new line of work focusing on learning hypernymy-speciﬁc word embeddings [Yu et al., 2015, Anh et al., 2016, Nickel and Kiela, 2017, Glavaˇs and Ponzetto, 2017, Nguyen et al., 2017] has gained much popularity in the ﬁeld.
Revised: More recently, a new line of work focusing on learning hypernymy-speciﬁc word embeddings [Yu et al., 2015, Anh et al., 2016, Nickel and Kiela, 2017, Glavaˇs and Ponzetto, 2017, Nguyen et al., 2017, Vuli´c and Mrkˇsi´c, 2018] has gained much popularity in the ﬁeld.

Original: This line is the closest to our work.
Revised: This line is the closest to our work.

Original: [Yu et al., 2015] proposed a supervised distance-margin neural network for the purpose of encoding hypernymy properties in the learnt embeddings.
Revised: Yu et al. [2015] proposed a supervised distance-margin neural network for the purpose of encoding hypernymy properties in the learnt embeddings.

Original: The neural network is trained using only the pre-extracted pairwise hypernymy tuples from Probase [Wu et al., 2012], without considering any contextual information that might be available in sentences where the two words co-occur.
Revised: The neural network is trained using only the pre-extracted pairwise hypernymy tuples from Probase [Wu et al., 2012], without considering any contextual information that might be available in sentences where the two words co-occur.

Original: However, it has been shown that co-occurrence contexts provide useful semantic clues for detecting taxonomic relationships [Velardi et al., 2013, Anh et al., 2014].
Revised: However, it has been shown that co-occurrence contexts provide useful semantic clues for detecting taxonomic relationships [Velardi et al., 2013, Anh et al., 2014].

Original: Similarly, [Anh et al., 2016] proposed a dynamically weighted neural network for leaning hypernymy-speciﬁc word embeddings.
Revised: Similarly, Anh et al. [2016] proposed a dynamically weighted neural network for leaning hypernymyspeciﬁc word embeddings.

Original: The proposed method initialises by extracting pairwise hypernymy data from WordNet
Revised: The proposed method initialises by extracting pairwise hypernymy data from WordNet

Original: [Miller, 1995], followed by extracting related sentences from Wikipedia to form the training data as triples including the hypernym, hyponym and a context word between them, as an input to the neural network.
Revised: [Miller, 1995], followed by extracting related sentences from Wikipedia to form the training data as triples including the hypernym, hyponym and a context word between them, as an input to the neural network.

Original: [Nguyen et al., 2017] introduced unsupervised neural model (HyperVec) that outperformed the previous two hypernym-speciﬁc embeddings work.
Revised: Nguyen et al. [2017] introduced unsupervised neural model (HyperVec) that outperformed the previous two hypernym-speciﬁc embeddings work.

Original: HyperVec jointly learns from the extracted hypernym pairs and contextual data.
Revised: HyperVec jointly learns from the extracted hypernym pairs and contextual data.

Original: In particular, their proposed method starts by extracting all the hypernym pairs from the WordNet, and use SGNS as a contextaware objective and jointly learns the hypernymy-speciﬁc word embeddings.
Revised: In particular, their proposed method starts by extracting all the hypernym pairs from the WordNet, and use SGNS as a context-aware objective and jointly learns the hypernymy-speciﬁc word embeddings.

Original: [Nguyen et al., 2017] report an improvement over [Yu et al., 2015] and [Anh et al., 2016] methods on hypernymy relation identiﬁcation task as well as the task of distinguishing between hypernym and hyponyms in a hypernymy relation pair.
Revised: Nguyen et al. [2017] report an improvement over Yu et al. [2015] and Anh et al. [2016] methods on hypernymy relation identiﬁcation task as well as the task of distinguishing between hypernym and hyponyms in a hypernymy relation pair.

Revised: Moreover, Vuli´c and Mrkˇsi´c [2018] introduce Lexical Entailment Attract Repel (LEAR) model to learn word embeddings that encode hypernymy.

Revised: LEAR works as a post-processing model that can take any word vectors as an input and inject external constraints of hypernym relations extracted from WordNet to emphasis the hypernym relations into the given word vectors.

Revised: Whereas Nickel and

Revised: Kiela [2017] followed a different line of work and proposed Poincar´e model that learn hierarchical embeddings into a hyperbolic space, precisely into Poincar´e ball, rather than the Euclidean space.

Revised: Poincar´e make use of WordNet hypernymy and merely learn from the taxomomy without a corpus information.

Original: Although  such methods are attractive, because they jointly learn from manually created taxonomies and text corpora, a common disadvantage of such methods is that they mainly focus on pairwise hypernymy relations data in the taxonomies, ignoring the entire hierarchical path.
Revised: Although the majority of such methods are attractive, because they jointly learn from manually created taxonomies and text corpora, a common disadvantage of such methods is that they mainly focus on pairwise hypernymy relations data in the taxonomies, instead of the entire hierarchical path.

Original: A full hierarchical path of hypernymy not only gives a better understanding of the problem than a single hypernymy edge but rather empirically shown to bemore useful for a pairwise hypernymy identiﬁcation.
Revised: A full hierarchical path of hypernymy not only gives a better understanding of the problem than a single hypernymy edge but rather empirically shown to be useful for a pairwise hypernymy identiﬁcation.

Original: 3. Hierarchical Word Embeddings (HWE)
Revised: 3. Hierarchical Word Embeddings (HWE)

Original: We propose a method that learns word embeddings by encoding the hierarchy of hypernymy relations between words.
Revised: We propose a method that learns word embeddings by encoding the hierarchy of hypernymy relations between words.

Original: To explain our method, let us consider an example, given a hierarchical hypernym path ( bird → vertebrate → chordate → organism → animal ) where the pairs (bird,vertebrate) , (vertebrate,chordate) , (chordate,organism) and (organism,animal) represent a direct hypernymy relation, whereas (bird, chordate) and (vertebrate, animal) form an indirect hypernymy.
Revised: To explain our method, let us consider an example, given a hierarchical hypernym path ( bird → vertebrate → chordate → organism → animal ) where the pairs (bird,vertebrate) , (vertebrate,chordate) , (chordate,organism) and (organism,animal) represent a direct hypernymy relation, whereas (bird, chordate) and (vertebrate, animal) form an indirect hypernymy.

Original: We require our embeddings to be able to encode not only the direct hypernymy relations between the hypernym and hyponym words, but also the indirect and the full hierarchical hypernym path.
Revised: We require our embeddings to be able to encode not only the direct hypernymy relations between the hypernym and hyponym words, but also the indirect and the full hierarchical hypernym path.

Original: Given a taxonomy T and a corpus C , we propose a method for learning a hierarchical word embedding w i ∈ R d for the i -th word w i ∈ V in a vocabulary V .
Revised: Given a taxonomy T and a corpus C , we propose a method for learning a hierarchical word embedding w i ∈ R d for the i -th word w i ∈ V in a vocabulary V .

Original: We use boldface fonts to denote vectors.
Revised: We use boldface fonts to denote vectors.

Original: We assign two vectors for the word w i , respectively denoting a hyponym w i , or a hypernym ˜ w i .
Revised: We assign two vectors for the word w i , respectively denoting a hyponym w i , or a hypernym ˜ w i .

Original: The proposed method initialises by making use of a set of hierarchical hypernymy paths, which can be obtained from a taxonomy.
Revised: The proposed method initialises by making use of a set of hierarchical hypernymy paths, which can be obtained from a taxonomy.

Original: Let us assume that w i is a leaf node in the taxonomy and P ( w i ) is a path that connects w i to the root of the taxonomy.
Revised: Let us assume that w i is a leaf node in the taxonomy and P ( w i ) is a path that connects w i to the root of the taxonomy.

Original: If the taxonomy is a tree, then only one such path exists.
Revised: If the taxonomy is a tree, then only one such path exists.

Original: However, if the taxonomy is a lattice or there are multiple senses of w i represented by different synsets as in the case of WordNet, we might have multiple paths as P ( w i ) .
Revised: However, if the taxonomy is a lattice or there are multiple senses of w i represented by different synsets as in the case of WordNet, we might have multiple paths as P ( w i ) .

Original: Because a taxonomy by deﬁnition arranges concepts in a hierarchical order, we would assume that some of the information contained in a leaf node w i to be inferred from its parent nodes that fall along the path P ( w i ) .
Revised: Because a taxonomy by deﬁnition arranges concepts in a hierarchical order, we would assume that some of the information contained in a leaf node w i to be inferred from its parent nodes that fall along the path P ( w i ) .

Original: Different compositional operators could be used then to infer the semantic representation for w i using its parents such as a recurrent neural network (RNN).
Revised: Different compositional operators could be used then to infer the semantic representation for w i using its parents such as a recurrent neural network (RNN).

Original: However, for simplicity and computational efﬁciency, we represent the embedding of a leaf node as the sum of its parents’ embeddings.
Revised: However, for simplicity and computational efﬁciency, we represent the embedding of a leaf node as the sum of its parents’ embeddings.

Original: This idea can be formalised into an objective function J T for the purpose of learning hierarchical word embedding over the entire vocabulary as follows:
Revised: This idea can be formalised into an objective function J T for the purpose of learning hierarchical word embedding over the entire vocabulary as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The indirect hypernym at the top of path (i.e. the root of a taxonomy in case it is a tree or the farthest from the hyponym word w i for a truncated path)
Revised: The indirect hypernym at the top of path (i.e. the root of a taxonomy in case it is a tree or the farthest from the hyponym word w i for a truncated path)

Original: represents less (more abstract) information about the word w i than its direct hypernym.
Revised: represents less (more abstract) information about the word w i than its direct hypernym.

Original: For example, revisiting our hierarchical example above ( bird → vertebrate → chordate → animal ), the direct hypernym vertebrate represents more information about bird than the indirect hypernym animal .
Revised: For example, revisiting our hierarchical example above ( bird → vertebrate → chordate → animal ), the direct hypernym vertebrate represents more information about bird than the indirect hypernym animal .

Original: To reﬂect this in the objective deﬁned in (1), we introduce a discounting term λ ( ˜ w j ) to assign a weight for each hypernym word in the path, and update the objective function given by (1) as follows:
Revised: To reﬂect this in the objective deﬁned in (1), we introduce a discounting term λ ( ˜ w j ) to assign a weight for each hypernym word in the path, and update the objective function given by (1) as follows:

Original: [Equation]
Revised: [Equation]

Original: We set λ ( ˜ w j ) =
Revised: We set λ ( ˜ w j ) =

Original: exp( L w i − D ˜ w j ) where L w i and D ˜ w j respectively denote the length of the hierarchical hypernymy path of the word w i , and the distance in words between the word w i and its hypernym ˜ w j in the path, where the distances are measured over the shortest path from the root to word in the taxonomy.
Revised: exp( L w i − D ˜ w j ) where L w i and D ˜ w j respectively denote the length of the hierarchical hypernymy path of the word w i , and the distance in words between the word w i and its hypernym ˜ w j in the path, where the distances are measured over the shortest path from the root to word in the taxonomy.

Original: The objective function given by (2) learns the word embeddings purely from the taxonomy T .
Revised: The objective function given by (2) learns the word embeddings purely from the taxonomy T .

Original: It does not consider the contextual co-occurrences between the hyponym and hypernym words in the corpus C .
Revised: It does not consider the contextual co-occurrences between the hyponym and hypernym words in the corpus C .

Original: To address this problem, for each hypernym word ˜ w j that appears in the path of the hyponym w i , we look up their co-occurrence in the corpus.
Revised: To address this problem, for each hypernym word ˜ w j that appears in the path of the hyponym w i , we look up their co-occurrence in the corpus.

Original: For this purpose, we ﬁrst create a cooccurrence matrix X between the hyponym and hypernym words within a context window in the corpus.
Revised: For this purpose, we ﬁrst create a cooccurrence matrix X between the hyponym and hypernym words within a context window in the corpus.

Original: The element X ij of X denotes the total occurrences between the words w i and ˜ w j in the corpus.
Revised: The element X ij of X denotes the total occurrences between the words w i and ˜ w j in the corpus.

Original: We then use the Global Vector (GloVe)
Revised: We then use the Global Vector (GloVe)

Original: [Pennington et al., 2014] objective to consider the co-occurrence between the hyponym word w i and its hypernyms ˜ w j for the purpose of learning the embeddings as follows:
Revised: [Pennington et al., 2014] objective to consider the co-occurrence between the hyponym word w i and its hypernyms ˜ w j for the purpose of learning the embeddings as follows:

Original: [Equation]
Revised: [Equation]

Original: Where b i and b j are real-valued scalers biases associated with w i and ˜ w j .
Revised: Where b i and b j are real-valued scalers biases associated with w i and ˜ w j .

Original: The function f is a weighting function deﬁned as follows.
Revised: The function f is a weighting function deﬁned as follows.

Original: [Equation]
Revised: [Equation]

Original: Finally, we combine the two objectives given by (2) and (3), into a joint linearly-weighted objective as follows:
Revised: Finally, we combine the two objectives given by (2) and (3), into a joint linearly-weighted objective as follows:

Original: [Equation]
Revised: [Equation]

Original: J C (5)

Original: To minimise (5) w.r
Revised: To minimise (5) w.r.t.

Original: [Equation]

Original: t. the parameters w i , ˜ w j , b i and b j , we compute the gradient of J w.r.t.
Revised: the parameters w i , ˜ w j , b i and b j , we compute the gradient of J w.r.t.

Original: those parameters as given below.
Revised: those parameters.

Original: [Equation] We randomly initialise all parameters and perform Stochastic Gradient Descent with the learning rate scheduled via AdaGrad [Duchi et al., 2011] [Equation]
Revised: We randomly initialise all parameters and perform Stochastic Gradient Descent with the learning rate scheduled via AdaGrad [Duchi et al., 2011].

Original: 4. Experiments and Results
Revised: 4. Experiments and Results

Original: We evaluate the embeddings learnt by the proposed HWE on two main tasks: the standard supervised hypernymy identiﬁcation and a newly-proposed hierarchical path completion.
Revised: We evaluate the embeddings learnt by the proposed HWE on three main tasks: the standard supervised hypernymy identiﬁcation, graded lexical entailment prediction and a newly-proposed hierarchical path completion.

Original: In both tasks, we compare the performance of the HWE with various prior works on learning word embeddings.
Revised: In all tasks, we compare the performance of the HWE with various prior works on learning word embeddings.

Original: In Section 4.2, we ﬁrst describe the experimental setup that we follow to train the proposed method.
Revised: In Section 4.2, we ﬁrst describe the experimental setup that we follow to train the proposed method.

Original: We then introduce the word embedding models that we use to compare the proposed method with (Section 4.2).
Revised: We then introduce the word embedding models that we use to compare the proposed method with (Section 4.2).

Original: Next, in Section 4.3, we discuss the experiments conducted on the supervised hypernymy identiﬁcation task.
Revised: Next, in Section 4.3, we discuss the experiments conducted on the supervised hypernymy identiﬁcation task.

Revised: Next, we introduce the experiments of predicting the graded lexical entailment task.

Original: We then describe the newly proposed evaluation task, hierarchical path completion (Section 4.4).
Revised: We then describe the newly proposed evaluation task, hierarchical path completion (Section 4.5).

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Any taxonomy can be used as T with the proposed method provided that the hypernymy relations that exist between words is speciﬁed.
Revised: Any taxonomy can be used as T with the proposed method provided that the hypernymy relations that exist between words is speciﬁed.

Original: We use WordNet [Miller, 1995] in our experiments.
Revised: We use WordNet [Miller, 1995] in our experiments.

Original: The average path length of the WordNet words is 7 words.
Revised: The average path length of the WordNet words is 7 words.

Original: Following the recommendation in prior work on extracting taxonomic relations, we exclude the top-level hypernyms in each path.
Revised: Following the recommendation in prior work on extracting taxonomic relations, we exclude the top-level hypernyms in each path.

Original: For example, [Anh et al., 2016] found that words such as object , entity and whole at the upper level of the hierarchical path to be noisy during the learning phase.
Revised: For example, Anh et al. [2016] found that words such as object , entity and whole at the upper level of the hierarchical path to be noisy during the learning phase.

Original: In fact, for the nouns in the WordNet, the word entity is the root for all the words.
Revised: In fact, for the nouns in the WordNet, the word ‘entity’ is the root for all the words.

Original: Moreover, the words like physical entity , abstraction , object and whole appear in the hierarchical path of 58% , 47 .
Revised: Moreover, the words like physical entity , abstraction , object and whole appear in the hierarchical path of 58% , 47 .

Original: 74% and 30 .
Revised: 74% and 30 .

Original: 95% of the WordNet words respectively.
Revised: 95% of the WordNet words respectively.

Original: As such, we limit the number of words in each path to 5 hypernyms, and obtained direct and indirect hypernymy relations.
Revised: As such, we limit the number of words in each path to 5 hypernyms, and obtained direct and indirect hypernymy relations.

Original: As a results, the number of extracted relations was 256 ,forming a 59 , 908 distinct hierarchical paths, and the vocabulary size V = 80 , 673 .
Revised: As a results, the number of extracted relations was 256 , 442 forming a 59 , 908 distinct hierarchical paths, and the vocabulary size V = 80 , 673 .

Original: As a corpus C , we used the ukWaC 1 which has ca.
Revised: As a corpus C , we used the ukWaC 1 which has ca.

Original: 2 billion tokens.
Revised: 2 billion tokens.

Original: Following the recommendations made in [Levy et al., 2015a], we set the context window to 10 tokens to the both sides of the hyponym word.
Revised: Following the recommendations made in [Levy et al., 2015a], we set the context window to 10 tokens to the both sides of the hyponym word.

Original: We followed the GloVe settings and set α = 0 .
Revised: We followed the GloVe settings and set α = 0 .

Original: 75 and t max = 100 in (4).
Revised: 75 and t max = 100 in (4).

Original: 4.2 Word Embedding Models
Revised: 4.2 Word Embedding Models

Original: We compare the HWE with prior word embedding models that vary from state-of-the-art models that: (i) merely trained on text corpus (Continuous-Bag-Of-Words ( CBOW ), Skip-Gram with
Revised: We compare the HWE with prior word embedding models that vary from state-of-the-art models that: (i) merely trained on text corpus (Continuous-Bag-Of-Words ( CBOW ), Skip-Gram with

Original: Negative Sampling ( SGNS )
Revised: Negative Sampling ( SGNS )

Original: [Mikolov et al., 2013] and Global Vector (GloVe) [Pennington et al., 2014]) (ii) jointly learn from a taxonomy and a text corpus ( Retroﬁt ting [Faruqui et al., 2015], JointReps [Bollegala et al., 2016]) and (iii) learn a hypernymy-speciﬁc word embeddings ( HyperVec [Nguyen et al., 2017]).
Revised: [Mikolov et al., 2013] and Global Vector (GloVe) [Pennington et al., 2014]) (ii) jointly learn from a taxonomy and a text corpus ( Retroﬁt ting [Faruqui et al., 2015], JointReps [Bollegala et al., 2016]) and (iii) learn a hypernymy-speciﬁc word embeddings ( HyperVec [Nguyen et al., 2017], Poincar´e [Nickel and Kiela, 2017], LEAR [Vuli´c and Mrkˇsi´c, 2018]).

Original: CBOW and SGNS are two log-bilinear single-layer neural models proposed by [Mikolov et al., 2013] that exploits the local co-occurrence between the words in a large text corpus.
Revised: CBOW and SGNS are two log-bilinear single-layer neural models proposed by Mikolov et al. [2013] that exploits the local co-occurrence between the words in a large text corpus.

Original: Where CBOW objective is to predict the word given its context words, SGNS in contrast predicts the context words given the focus word.
Revised: Where CBOW objective is to predict the word given its context words, SGNS in contrast predicts the context words given the focus word.

Original: GloVe is a log-bilinear model that uses the global co-occurrences between the word and its context words to predict their embeddings, and it is represented by the objective function given by (3).
Revised: GloVe is a log-bilinear model that uses the global co-occurrences between the word and its context words to predict their embeddings, and it is represented by the objective function given by (3).

Original: Retroﬁt and JointReps are two models that combine the two resources, taxonomy and text corpus, for learning word embeddings.
Revised: Retroﬁt and JointReps are two models that combine the two resources, taxonomy and text corpus, for learning word embeddings.

Original: In Retroﬁt , a pre-trained word embeddings are fed into a taxonomy in a post-processing manner, whereas JointReps jointly learns the word embeddings from the two resources simultaneously .
Revised: In Retroﬁt , a pre-trained word embeddings are fed into a taxonomy in a post-processing manner, whereas JointReps jointly learns the word embeddings from the two resources simultaneously .

Original: On the other hand, HyperVec is an unsupervised neural model for learning hypernymy-speciﬁc word embeddings.
Revised: On the other hand, HyperVec , Poincar´e and LEAR are hypernymy-speciﬁc word embeddings.

Original: HyperVec objective is to encode hypernymy properties in the learnt word embeddings by utilising a taxonomy and a corpus.

Original: Details of this model are described in Section 2.
Revised: Details of these models are described in Section 2.

Original: We used the same ukWaC corpus that is used with the proposed method to trainall the prior methods  using the publicly available implementations by the original authors for each method.
Revised: We used the same ukWaC corpus that is used with the proposed method to train the prior methods ( CBOW , SGNS , GloVe , Retroﬁt , JointReps and HyperVec ) using the publicly available implementations by the original authors for each method.

Original: In all the experiments, we also follow the same settings used with the proposed method, and set the context window to 10 words to both sides of the focus word, and remove the words that appear less than 20 times in the corpus.
Revised: In all the experiments, we also follow the same settings used with the proposed method, and set the context window to 10 words to both sides of the focus word, and remove the words that appear less than 20 times in the corpus.

Original: Following [Levy et al., 2015a], we set the negative sampling rate to 5 in SGNS .
Revised: Following Levy et al. [2015a], we set the negative sampling rate to 5 in SGNS .

Original: We retroﬁt the embeddings learnt by SGNS and CBOW into the Retroﬁt model.
Revised: We retroﬁt the embeddings learnt by SGNS and CBOW into the Retroﬁt model.

Original: Moreover, for Retroﬁt , JointReps and HyperVec , we used the hypernym relations extracted from the WordNet provided by the original authors for each model to derive the taxonomy constraints during the training.
Revised: Moreover, for Retroﬁt , JointReps and HyperVec , we used the hypernym relations extracted from the WordNet provided by the original authors for each model to derive the taxonomy constraints during the training.

Revised: For Poincar´e model, we used the gensim implementation [ ˇReh˚uˇrek and Sojka, 2010] and set the negative samples to 10 .

Revised: In LEAR , we used the pre-trained embeddings provided by the original authors.

Original: We learn a 300 dimensional word embeddings in all experiments.
Revised: We learn a 300 dimensional word embeddings in all experiments.

Original: 4.3 Supervised Hypernymy Identiﬁcation
Revised: 4.3 Supervised Hypernymy Identiﬁcation

Original: The supervised hypernymy identiﬁcation is a standard task to evaluate the ability of the word embeddings to detect hypernymy.
Revised: The supervised hypernymy identiﬁcation is a standard task to evaluate the ability of the word embeddings to detect hypernymy.

Original: The task is modeled as a binary classiﬁcation problem, where a classiﬁer is trained using pairs of words ( x, y ) labeled as positive (i.e. a hypernymy relation exists between the words) or negative (otherwise).
Revised: The task is modeled as a binary classiﬁcation problem, where a classiﬁer is trained using pairs of words ( x, y ) labeled as positive (i.e. a hypernymy relation exists between the words) or negative (otherwise).

Original: Each word in a word-pair is represented by its pretrained word embedding.
Revised: Each word in a word-pair is represented by its pretrained word embedding.

Original: Several approaches have been proposed to combine the embeddings of the two words in each word-pair to create a single feature vector representing the relation between the two words that is subsequently given to the classiﬁer.
Revised: Several approaches have been proposed to combine the embeddings of the two words in each word-pair to create a single feature vector representing the relation between the two words that is subsequently given to the classiﬁer.

Original: For example, concatenation ( x ⊕ y )
Revised: For example, concatenation ( x ⊕ y )

Original: [Baroni et al., 2012], difference ( x − y ) and addition ( x + y )
Revised: [Baroni et al., 2012], difference ( x − y ) and addition ( x + y )

Original: [Weeds et al., 2014] have been used as operators for creating feature vectors for word-pairs.
Revised: [Weeds et al., 2014] have been used as operators for creating feature vectors for word-pairs.

Original: In our preliminary experiments, we found that concatenation to perform best among all of those operators for the task of training a hypernymy identiﬁcation.
Revised: In our preliminary experiments, we found that concatenation to perform best among all of those operators for the task of training a hypernymy identiﬁcation.

Original: Therefore, we use concatenation as the preferred operator in the remainder of the experiments described in the paper.
Revised: Therefore, we use concatenation as the preferred operator in the remainder of the experiments described in the paper.

Original: We use a binary Support Vector Machine (SVM) classiﬁer, implemented in scikit-learn [Pedregosa et al., 2011].
Revised: We use a binary Support Vector Machine (SVM) classiﬁer, implemented in scikit-learn [Pedregosa et al., 2011].

Original: We set an RBF kernel γ = 0 .
Revised: We set an RBF kernel γ = 0 .

Original: 03125 and a cost parameter C
Revised: 03125 and a cost parameter C

Original: = 8 . 0 , using an independent validation dataset.
Revised: = 8 . 0 , using an independent validation dataset.

Original: [Table]
Revised: [Table]

Original: We pick four widely used hypernym benchmark datasets, Kotlerman [Kotlerman et al., 2010],
Revised: We pick four widely used hypernym benchmark datasets, Kotlerman [Kotlerman et al., 2010],

Original: Bless [Baroni and Lenci, 2011], Baroni [Baroni et al., 2012] and Levy [Levy et al., 2014], for the supervised classiﬁcation task.
Revised: Bless [Baroni and Lenci, 2011], Baroni [Baroni et al., 2012] and Levy [Levy et al., 2014], for the supervised classiﬁcation task.

Original: To avoid any lexical memorisation , where the classiﬁer simply memorises the prototypical hypernyms rather than learning the relation, [Levy et al., 2015b] introduced a disjoint version with none lexical overlap between the test and train portions for each of the above datasets, we adopt these disjoint versions in our experiment.
Revised: To avoid any lexical memorisation , where the classiﬁer simply memorises the prototypical hypernyms rather than learning the relation, [Levy et al., 2015b] introduced a disjoint version with none lexical overlap between the test and train portions for each of the above datasets, we adopt these disjoint versions in our experiment.

Original: Details of each benchmark dataset is summarised in Table 1.
Revised: Details of each benchmark dataset is summarised in Table 1.

Original: In Table 2, we compare the quality of our embeddings ( HWE ) against that of the other methods on the hypernym identiﬁcation benchmarks.
Revised: In Table 2, we compare the quality of our embeddings ( HWE ) against that of the other methods on the hypernym identiﬁcation benchmarks.

Original: Although F1 score has been widely-used as an evaluation measure in prior work on hypernym identiﬁcation, [Sanchez and Riedel, 2017] argued that Area
Revised: Although F1 score has been widely-used as an evaluation measure in prior work on hypernym identiﬁcation, [Sanchez and Riedel, 2017] argued that Area

Original: Under the ROC Curve ( AUC ) is more appropriate as an evaluation measure for this task because some of the benchmark datasets are unbalanced in terms of the number of positive vs. negative test instances they contain.
Revised: Under the ROC Curve ( AUC ) is more appropriate as an evaluation measure for this task because some of the benchmark datasets are unbalanced in terms of the number of positive vs. negative test instances they contain.

Original: To be able to compare our results with prior work that report only F1 score, we use both F1 score as well as AUC as evaluation measures in our experiments. 
Revised: To be able to compare our results with prior work that report only F1 score, we use both F1 score as well as AUC as evaluation measures in our experiments. Model Kotlerman Bless

Revised: Baroni

Original: [Table]
Revised: Levy F AUC F

Revised: AUC

Revised: F

Original: GloVe 56.
Revised: AUC F AUC CBOW 56.

Revised: SGNS 56.

Revised: GloVe 56.

Original: Retroﬁt(CBOW) 53.
Revised: Retroﬁt(CBOW) 53.

Original: Retroﬁt(SGNS) 53.
Revised: Retroﬁt(SGNS) 53.

Original: JointReps 56.
Revised: JointReps 56.

Original: HyperVec 57.
Revised: HyperVec 57.

Original: HWE

Original: [Table] [Table]
Revised: Poincar´e 45.

Original:  From Table 2, wecan see thatexcept for the Bless benchmark dataset, the HWE reports the best scores in all the other benchmarks.
Revised: LEAR 64. HWE 57. From Table 2, we see that the HWE reports the best scores in two of the benchmark datasets.

Original: In particular, in Kotlerman dataset, HWE reports the best performance, with a slight improvement over the other methods. Similar trends can be observed with Levy dataset, as HWE achieves the highest accuracy, reporting a slightly better score than CBOW , SGNS , GloVe , and JointReps buta statistically signiﬁcant improvement (binomial test, p < 0 . 05 ) over the closest related work, the hypernymy-speciﬁc  embedding model HyperVec .
Revised: In particular, in Levy dataset, HWE reports the best performance, with a slight improvement over the ﬁrst two categories of methods in the table, but signiﬁcant improvement (binomial test, p < 0 . 05 ) over the closest related works in the third category. Similarly, HWE scores the highest in the Baroni dataset where we can observe a strong difference between the hypernymy-speciﬁc word embedding models and other methods.

Original: Ta-
Revised: In particular, HyperVec , LEAR and HWE signiﬁcantly outperforms other methods and HWE reports the best score in this dataset. This result is particularly noteworthy, because a prior extensive analysis on different benchmark datasets for hypernym identiﬁcation by Sanchez and Riedel [2017] concluded that the Baroni dataset is the most appropriate dataset for robustly evaluating hypernym identiﬁcation methods. In Kotlerman and Bless datasets, LEAR reports the best scores among all models.

Original: ble 2 shows that even the methods that were trained only with a text corpus without speciﬁcally designed to capture the hypernymy relation, perform well in the Bless dataset, reporting a better or a comparable performance to the hypernymy-speciﬁc embeddings such as HWE and HyperVec .
Revised: However, Table 2 shows that even the methods that were trained only with a text corpus without speciﬁcally designed to capture the hypernymy relation, perform well in the Bless dataset, reporting a better or a comparable performance to the hypernymyspeciﬁc embeddings such as HWE and HyperVec .

Original: Similar comparable results can be seen with Kotlerman and Levy benchmarks as well. However, from the same table, the strongest difference can be observed in theBaroni dataset, where both HyperVec and HWE signiﬁcantly outperforms other methods, and HWE reports the best score in this dataset.
Revised: It is worth noting that both of the closest related works, HyperVec and LEAR , use pairwise hypernym relations in a similar spirit to the structure of the benchmark datasets, whereas the HWE use the entire hierarchical path, nevertheless reporting a better or comparable results. 4.4 Graded Lexical Entailment

Original: This result is particularly noteworthy, because a prior extensive analysis on different benchmark datasets for hypernym identiﬁcation by [Sanchez and Riedel, 2017] concluded that the Baroni dataset is  the most appropriate dataset for robustly evaluating hypernym identiﬁcation methods.
Revised: An important aspects of the HWE embeddings is its ability to encode the hierarchical structure available in the taxonomy in the learnt embeddings. To further check this ability we use the gold standard dataset HyperLex [Vuli´c et al., 2017] to test how well the HWE embeddings capture graded lexical entailment. HyperLex focuses on the relation of graded or soft lexical entailment at a continuous scale rather than simplifying the judgements into a binary decision.

Revised: The HyperLex dataset consists of 2616 word pairs where each pair is manually annotated with a score on a scale of [0 , 10] indicating the strength of the relations of lexical entailment.

Original: These results empirically justify our proposal to use the hierarchical path in  a taxonomy, instead of merely a pairwise hypernym relation, for learning better hypernym-speciﬁc word embeddings.
Revised: Lexical entailment is asymmetric in general, therefore we need an asymmetric distance function that takes into account both vector norm and direction to provide correct entailment scores between word pairs. For this purpose, given a hypernymy pair ( x, y ) , where x is the hyponym word and y is the hypernym, we follow Vuli´c and Mrkˇsi´c [2018] and compute the entailment score as follows: [Equation]

Revised: where dcos ( x , y ) is the cosine distance 2 .

Revised: Following the standard approach for evaluating using the

Revised: HyperLex dataset, we measure Spearman ρ correlation coefﬁcient between gold standard ratings and the predicted scores.

Revised: Table 3 shows the results of the Spearman correlation coefﬁcients of HWE and the other word embeddings models on the HyperLex dataset against the human ratings.

Revised: Tableshows that HWE was able to encode the hierarchical structure in the learnt embeddings reporting a better or comparable results to all other models except for LEAR .

Revised: In particular, HWE reports an improvement over the corpus-based and joint methods as well as Poincar´e embeddings.

Revised: It is noteworthy that LEAR was trained with larger resources which allows the model to beneﬁt from a large-scale vocabulary, as we use the provided pre-trained embeddings of LEAR .

Revised: We aim to re-run the experiment in the future and use the same resources, text corpus and taxonomy data, that were used with all the models with LEAR .

Revised: [Table]

Original: 4.4 Hierarchical Path Completion
Revised: 4.5 Hierarchical Path Completion

Original: Considering that our HWEs are designed to capture hierarchical information about words, it remains an open question as to what extent the word embeddings learnt using the proposed method encode the hierarchical structure.
Revised: Considering that our HWEs are designed to capture hierarchical information about words, it remains an open question as to what extent the word embeddings learnt using the proposed method encode the hierarchical structure.

Original: The supervised hypernymy identiﬁcation task discussed in the previous section provides only a partial answer to this question because all the benchmarks used in the experiments there were speciﬁcally annotated for hypernymy relation between two words, ignoring the taxonomic structure.
Revised: The supervised hypernymy identiﬁcation and the graded lexical entailment tasks discussed in the previous sections (4.3 and 4.4) provides only a partial answer to this question because all the benchmarks used in the experiments there were speciﬁcally annotated for hypernymy relation between two words, ignoring the taxonomic structure.

Original: Unfortunately, none of the prior work has looked into this aspect of word embeddings and to the best of our knowledge there exists no benchmark dataset for conducting such a hierarchical evaluation over a taxonomy.
Revised: Unfortunately, none of the prior work has looked into this aspect of word embeddings and to the best of our knowledge there exists no benchmark dataset for conducting such a hierarchical evaluation over a taxonomy.

Original: To address this issue, we create a novel dataset by ﬁrst sampling paths from the WordNet, which connect a hypernym to a hyponym via a path not exceeding a maximum path length L max .
Revised: To address this issue, we create a novel dataset by ﬁrst sampling paths from the WordNet, which connect a hypernym to a hyponym via a path not exceeding a maximum path length L max .

Original: We limit the paths to contain words that are unigrams, bigrams or trigrams, and sample the paths such that words with different frequency ranges are covered, thereby considering paths that contain frequent as well as rare words.
Revised: We limit the paths to contain words that are unigrams, bigrams or trigrams, and sample the paths such that words with different frequency ranges are covered, thereby considering paths that contain frequent as well as rare words.

Original: This enables us to conduct a fair and a balanced evaluation of the hierarchical word embeddings.
Revised: This enables us to conduct a fair and a balanced evaluation of the hierarchical word embeddings.

Original: We remove any paths that are used as training data when computing J c in (1).
Revised: We remove any paths that are used as training data when computing J c in (1).

Original: By following this procedure we created a dataset that contains 330 paths.
Revised: By following this procedure we created a dataset that contains 330 paths.

Original: If a path contains only unigrams then it is considered as a unigram path.
Revised: If a path contains only unigrams then it is considered as a unigram path.

Original: If a path contains at least one bigram but no trigrams, we consider it to be a bigram path and if a path contains a trigram then it is considered as a trigram path.
Revised: If a path contains at least one bigram but no trigrams, we consider it to be a bigram path and if a path contains a trigram then it is considered as a trigram path.

Original: There are respectively 150 , 120 and 60 unigram, bigram and trigram paths in the created dataset.
Revised: There are respectively 150 , 120 and 60 unigram, bigram and trigram paths in the created dataset.

Original: Inspired by the word analogy prediction task that is widely used to evaluate word embeddings [Mikolov et al., 2013], we propose a hierarchical path completion task as follows.
Revised: Inspired by the word analogy prediction task that is widely used to evaluate word embeddings [Mikolov et al., 2013], we propose a hierarchical path completion task as follows.

Original: Given a hierarchical path ( a → b → c → d → e ) where a is the hyponym word and b , c , d and e are its hypernyms, the task is to predict the hyponym word a given its hypernyms b , c , d and e .
Revised: Given a hierarchical path ( a → b → c → d → e ) where a is the hyponym word and b , c , d and e are its hypernyms, the task is to predict the hyponym word a given its hypernyms b , c , d and e .

Original: However, in the taxonomy, we might observe several hyponym words that have the exact same hierarchical path.
Revised: However, in the taxonomy, we might observe several hyponym words that have the exact same hierarchical path.

Original: For example, the hypernyms ( b → c → d → e ) might represent the hierarchical path to both a and a (cid:48) .
Revised: For example, the hypernyms ( b → c → d → e ) might represent the hierarchical path to both a and a (cid:48) .

Original: In fact, for each hierarchical path in the WordNet, there are, on average, 8 hyponym words represented by that particular hierarchical path.
Revised: In fact, for each hierarchical path in the WordNet, there are, on average, 8 hyponym words represented by that particular hierarchical path.

Original: Therefore, for each hierarchical path instance, if a method predicts a candidate a that belongs to any of the hyponyms with that path, we consider it to be a correct prediction.
Revised: Therefore, for each hierarchical path instance, if a method predicts a candidate a that belongs to any of the hyponyms with that path, we consider it to be a correct prediction.

Original: Given a set of word embeddings learnt using some word embedding learning method, to solve a hierarchical path completion problem, we used several prediction methods to predict a hyponym word a using the hypernyms b , c , d and e as described next.
Revised: Given a set of word embeddings learnt using some word embedding learning method, to solve a hierarchical path completion problem, we used several prediction methods to predict a hyponym word a using the hypernyms b , c , d and e as described next.

Original: ADD : We used an additive method ( ADD ), in which we compute the cosine similarity between b + c + d + e and each word x in a ﬁxed vocabulary, and select the word a ∗ with the highest cosine similarity as the prediction.
Revised: ADD : We used an additive method ( ADD ), in which we compute the cosine similarity between b + c + d + e and each word x in a ﬁxed vocabulary, and select the word a ∗ with the highest cosine similarity as the prediction.

Original: If a ∗ equals to a (or a ’s), then we consider it to be a correct prediction for the given test instance.
Revised: If a ∗ equals to a (or a ’s), then we consider it to be a correct prediction for the given test instance.

Original: SUB : In addition to the ADD method, and following the same procedure, we further consider several other prediction methods such as b + c + d − e , b
Revised: SUB : In addition to the ADD method, and following the same procedure of computing the cosine similarity, we further consider several other prediction methods such as b + c + d − e , b

Original: + c − d + e , b − c − d − e and b − c + d + e , for predicting a using the hypernyms b , c , d and e .
Revised: + c − d + e , b − c − d − e and b − c + d + e , for predicting a using the hypernyms b , c , d and e .

Original: We refer to these methods as the subtraction ( SUB ).
Revised: We refer to these methods as the subtraction ( SUB ).

Original: We found that the variant b + c + d − e of SUB to perform consistently better than other SUB variations with all embedding models.
Revised: We found that the variant b + c + d − e of SUB to perform consistently better than other SUB variations with all embedding models.

Original: Considering that the further the subtracted candidate in the hierarchical path from a the lesser its inﬂuence on the prediction.
Revised: Considering that the further the subtracted candidate in the hierarchical path from a the lesser its inﬂuence on the prediction.

Original: For example, the variant b + c + d − e reports a better prediction result as compared to b − c + d + e , presumably because of the subtracted hypernym c being closer to the hyponym a as compared to the farthest hyponym e .
Revised: For example, the variant b + c + d − e reports a better prediction result as compared to b − c + d + e , presumably because of the subtracted hypernym c being closer to the hyponym a as compared to the farthest hyponym e .

Original: The remainder of the experiments in the paper use the variant b + c + d − e as SUB .
Revised: The remainder of the experiments in the paper use the variant b + c + d − e as SUB .

Original: DH :
Revised: DH :

Original: In this method, we use only the direct hypernym b to predict a (or a ’s), and refer to this method as the direct hypernym ( DH ).
Revised: In this method, we use only the direct hypernym b to predict a (or a ’s), and refer to this method as the direct hypernym ( DH ).

Revised: We follow the same procedure of computing the cosine similarity as used in the ADD and SUB methods.

Original: If the predicted candidate is in the set of hyponyms for a particular test hierarchical path, then we consider it to be a correct prediction.
Revised: If the predicted candidate is in the set of hyponyms for a particular test hierarchical path, then we consider it to be a correct prediction.

Original: We then deﬁne the accuracy of a prediction method that uses word embeddings learnt from a particular word embedding learning method as the percentage of the correctly answered test instances.
Revised: We then deﬁne the accuracy of a prediction method that uses word embeddings learnt from a particular word embedding learning method as the percentage of the correctly answered test instances.

Original: [Table] Uni+Bi Uni+Bi+Tri Uni Uni+Bi Uni+Bi+Tri
Revised: In Table 4, we report the accuracies of different word embedding learning methods on the hierarchical path completion dataset using different prediction methods over different n -gram lengths.

Original: Uni Uni+Bi Uni+Bi+Tri CBOW 38.25 36.
Revised: According to Clopper-Pearson conﬁdence intervals computed at p < 0 . 05 , the proposed HWE signiﬁcantly outperforms all other word embedding learning methods compared in Table 4 irrespective of the prediction method being used.

Original: SGNS 32.77 31. GloVe 29.60 27. Retroﬁt(CBOW) 38.58 38. Retroﬁt(SGNS) 34.73 33.
Revised: Surprisingly, in contrast to the results found on the supervised classiﬁcation task (4.3) and the graded lexical entailment task (4.4), where the prior word embeddings methods, including hypernymy-speciﬁc methods HyperVec and LEAR were performing constantly well on pairwise datasets, they seem to suffer capturing an upper level hierarchy in the vector space.

Original: JointReps 31.19 30. HyperVec 19.09 18.
Revised: Whereas, on other side, Poincar´e which was suffering with pairwise data in the previous tasks, seems to learn a better upper level hierarchy and works well with this hierarchical completion task.

Original: HWE 88.30 85.
Revised: The fact that

Original: In Table 3, we report the accuracies of different word embedding learning methods on the hierarchical path completion dataset using different prediction methods over different n -gram lengths.
Revised: Poincar´e embeddings, a hierarchical word embeddings learn method, reports good performance on this hierarchical path completion task, suggests that this is an appropriate task for evaluating hierarchies and embeddings.

Original: According to Clopper-Pearson conﬁdence intervals computed at p < 0 . 05 , the proposed HWE signiﬁcantly outperforms all other word embedding learning methods compared in Table 3 irrespective of the prediction method being used. Surprisingly, in contrast to the results found on the supervised classiﬁcation task (4.3), where the prior word embeddings methods, including hypernymy-speciﬁc method HyperVec was performing constantly well on pairwise datasets, they seem to suffer capturing an upper level hierarchy in the vector space.
Revised: [Table]

Original: Using the ADD method, HWE reports an improvement as high as 50% on accuracy over Retroﬁt(CBOW) , which is the highest score among the other methods.
Revised: Using the ADD method [Table] HWE reports an improvement as high as 28% on accuracy over Poincar´e , which is the highest score among the other methods.

Original: Table 3 shows that SUB slightly increases the other word embedding methods’ results and decreases the performance of HWE .
Revised: Table 4 shows that SUB slightly increases the other word embedding methods’ results and decreases the performance of HWE .

Original: DH signiﬁcantly improves the results for all word embeddings, except for HWE  , whereas ADD applied to HWE reports the best overall performance in Table 3.
Revised: DH signiﬁcantly improves the results for all word embeddings, except for HWE and Poincar´e , whereas ADD applied to HWE reports the best overall performance in Table 4.

Original: This result shows that the proposed HWE not relying merely on the direct hypernym but considers information available in the entire hierarchical path composed into a single vector via addition.
Revised: This result shows that the proposed HWE not relying merely on the direct hypernym but considers information available in the entire hierarchical path composed into a single vector via addition.

Original: Furthermore, Table 3 shows that including bigram and trigram hypernyms on the paths slightly hurt the performance, possibly because of data sparseness.[Table] [Table] dress kafﬁyeh,
Revised: Furthermore, Table 4 shows that including bigram and trigram hypernyms on the paths slightly hurt the performance, possibly because of data sparseness.

Original: [Table]

Original: recruiter,

Original: [Table]

Original: , distributor market stockist,caterer

Original: [Table]

Original: ape ape

Original: [Table]

Original: orangutan, gorilla

Original: [Table] ,chimpanzee [Table] To further demonstrate the ability of the proposed method for completing the hierarchical paths, we qualitatively analyse the predictions of HWE and Retroﬁt(CBOW) , which report the best ac- curacy among all the other methods according to Table 3.
Revised: To further demonstrate the ability of the proposed method for completing the hierarchical paths, we qualitatively analyse the predictions of HWE and Retroﬁt(CBOW) .

Original: Due to the limited availability of space, only a few randomly selected examples are shown in Table 4.
Revised: Due to the limited availability of space, only a few randomly selected examples are shown in Table 5.

Original: The hyponym column represents gold standard answers (i.e. correct hyponym words).
Revised: The hyponym column represents gold standard answers (i.e. correct hyponym words).

Original: Due to space limitations, we show only a maximum of 4 correct hyponyms in the table.
Revised: Due to space limitations, we show only a maximum of 4 correct hyponyms in the table.

Original: If a particular path has more than 4 hyponyms, we randomly select 4 , otherwise all possible hyponyms are listed.
Revised: If a particular path has more than 4 hyponyms, we randomly select 4 , otherwise all possible hyponyms are listed.

Original: We see that HWE accurately predicts the correct word in many cases where Retroﬁt(CBOW) fails (shaded rows in the table).
Revised: We see that HWE accurately predicts the correct word in many cases where Retroﬁt(CBOW) fails (shaded rows in the table).

Original: We can also see Retroﬁt(CBOW) tends to predict the average nearest neighbour of a word’s hypernyms to be the missing word, not necessarily completing the hierarchical path.
Revised: We can also see Retroﬁt(CBOW) tends to predict the average nearest neighbour of a word’s hypernyms to be the missing word, not necessarily completing the hierarchical path.

Original: For example, given the path ( ?
Revised: For example, given the path ( ?

Original: → head dress → clothing → consumer goods → commodity ), HWE correctly predicts the missing word to be hat , whereas Retro(CBOW) incorrectly predicts dress .
Revised: → head dress → clothing → consumer goods → commodity ), HWE correctly predicts the missing word to be hat , whereas Retro(CBOW) incorrectly predicts dress .

Original: More interestingly, Tableshows by considering the hierarchical path we can reduce the ambiguity of the prediction.
Revised: More interestingly, Tableshows by considering the hierarchical path we can reduce the ambiguity of the prediction.

Original: For instance, given the path ( ?
Revised: For instance, given the path ( ?

Original: → book → publication → work → product ) the Retroﬁt(CBOW) predicts the word nature to complete the path, presumably associating this with the scientiﬁc journal Naturebecause the path includes publication , however HWE correctly predicted the word to be textbook . 
Revised: → book → publication → work → product ) the Retroﬁt(CBOW) predicts the word nature to complete the path, presumably associating this with the scientiﬁc journal Naturebecause the path includes publication , however HWE correctly predicted the word to be textbook . Hypernym(b) Hypernym(c) Hypernym(d)

Revised: Hypernym(e)

Original: Moreover, from Table 4, we can see that in some cases HWE struggled to predict the correct words, while Retroﬁt(CBOW) has managed to accurately complete the path.
Revised: Hyponym(s) (a’s) HWEPrediction RetroﬁtPrediction book publication work product booklet,textbook, textbook nature storybook,catalog success attainment accomplishment action winning, winning enjoyment ﬂying colors [Table]

Revised: dress clothing consumer goods commodity turban,cap, hat dress kafﬁyeh,hat supplier businessperson capitalist person recruiter,distributor, distributor market stockist,caterer great ape anthropoid ape ape primate orangutan, gorilla chimp gorilla,chimpanzee executive administrator head leader corporate executive, minister chair minister,rainmaker

Revised: [Table]

Revised: crash landing crash landing landing

Revised: [Table]

Revised: ,carnivore, carnivore ecology lagomorph,rodnet

Revised: [Table]

Revised: attribute discouragement, discouragement

Revised: [Table]

Revised: pessimism

Revised: [Table]

Revised: act event

Revised: [Table]

Revised: , boondoggle enjoyment boondoggle

Revised: [Table]

Revised: separation bank

Revised: [Table]

Revised: bank run withdrawal

Revised: [Table]

Revised: abduction, kidnapping kidnapping kidnapping

Revised: [Table]

Revised: [Table]

Revised: , card lyric lipogram

Revised: [Table]

Revised: trip voyage

Revised: [Table]

Revised: Moreover, from Table 5, we can see that in some cases HWE struggled to predict the correct words, while Retroﬁt(CBOW) has managed to accurately complete the path.

Original: For example, HWE failed to predict the word(s) voyage , cruise or maiden voyage to complete the path ( ?
Revised: For example, HWE failed to predict the word(s) voyage , cruise or maiden voyage to complete the path ( ?

Original: → ocean trip → world travel → travel → motion ) while Retroﬁt(CBOW) was able to do so.
Revised: → ocean trip → world travel → travel → motion ) while Retroﬁt(CBOW) was able to do so.

Original: Considering the fact that ghost and apparition are registered as synonyms in the WordNet, Retroﬁt(CBOW) could be simply picking up this synonym, without considering any hierarchical information in this case.[Figure]
Revised: Considering the fact that ghost and apparition are registered as synonyms in the WordNet, Retroﬁt(CBOW) could be simply picking up this synonym, without considering any hierarchical information in this case.

Original: Furthermore, we study the effect of each hypernym word in the hierarchical path towards the prediction accuracy.
Revised: Furthermore, we study the effect of each hypernym word in the hierarchical path towards the prediction accuracy.

Original: In particular, given a hierarchical path ( a → b → c → d → e ) , the goal is to study which of the given hypernyms b , c , d and e is having the strongest effect on predicting the correct word a .
Revised: In particular, given a hierarchical path ( a → b → c → d → e ) , the goal is to study which of the given hypernyms b , c , d and e is having the strongest effect on predicting the correct word a .

Original: For this purpose, we used the same hierarchical path completion dataset including bigram and trigram hypernyms and follow the same procedure used in the above experiment, but we hold out one of the hypernym while computing the cosine similarity between b + c + d + e and a .
Revised: For this purpose, we used the same hierarchical path completion dataset including bigram and trigram hypernyms and follow the same procedure used in the above experiment, but we hold out one of the hypernym while computing the cosine similarity between b + c + d + e and a .

Original: We vary the excluded hypernym between b , c , d and e .
Revised: We vary the excluded hypernym between b , c , d and e .

Original: When we hold out b we refer to this as the direct hypernym exclusion, whereas holding out one of c , d or e is referred to as the indirect hypernym exclusion.
Revised: When we hold out b we refer to this as the direct hypernym exclusion, whereas holding out one of c , d or e is referred to as the indirect hypernym exclusion.

Original: In Figure 1, we report the accuracy result on the hierarchical path completion task using the ADD method with each exclusion variation.
Revised: In Figure 1, we report the accuracy result on the hierarchical path completion task using the ADD method with each exclusion variation.

Original: Not surprisingly, excluding the direct hypernym  ( b ) drastically effect the performance.
Revised: Not surprisingly, excluding the direct hypernym [Figure] ( b ) drastically effect the performance.

Original: Whereas, the least effect can be observed with the farthest hypernym ( e ).
Revised: Whereas, the least effect can be observed with the farthest hypernym ( e ).

Original: For example, in a hierarchical path of a hyponym word of only uigram hypernyms, excluding the ﬁrst direct hypernym results on 68% accuracy prediction, whereas holding out the farthest hypernym signiﬁcantly improves the prediction score reaching 88% accuracy.
Revised: For example, in a hierarchical path of a hyponym word of only uigram hypernyms, excluding the ﬁrst direct hypernym results on 68% accuracy prediction, whereas holding out the farthest hypernym signiﬁcantly improves the prediction score reaching 88% accuracy.

Original: We also investigate how the dimensionality d effects the proposed method.
Revised: We also investigate how the dimensionality d effects the proposed method.

Original: Similar to the previous experiments, we report the accuracy of predicting the hyponym word in each hierarchical hypernym path.
Revised: Similar to the previous experiments, we report the accuracy of predicting the hyponym word in each hierarchical hypernym path.

Original: From Figure 2, we see that the proposed method is able to reach as high as 76% with as small as 25 dimensions.
Revised: From Figure 2, we see that the proposed method is able to reach as high as 76% with as small as 25 dimensions.

Original: The performance is then increase with the dimensionality, reaching its peak with 200 dimensions reporting 88% accuracy.
Revised: The performance is then increase with the dimensionality, reaching its peak with 200 dimensions reporting 88% accuracy.

Original: It is worth noting that, adding more dimensions does not negatively effect the performance.
Revised: It is worth noting that, adding more dimensions does not negatively effect the performance.

Original: Moreover, Figure 2 shows that including bigram and trigram hypernym words in the paths report a slight decrease in the performance, but similar trend as to the unigram only is observed.
Revised: Moreover, Figure 2 shows that including bigram and trigram hypernym words in the paths report a slight decrease in the performance, but similar trend as to the unigram only is observed.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We presented a method to learn a Hierarchical Word Embedding ( HWE ) for identifying the hypernymy relations between words.
Revised: We presented a method to learn a Hierarchical Word Embedding ( HWE ) for identifying the hypernymy relations between words.

Original: For this purpose, we introduced a joint objective that make use of both a taxonomy and a large text corpus to learn hierarchical word embeddings.
Revised: For this purpose, we introduced a joint objective that make use of both a taxonomy and a large text corpus to learn hierarchical word embeddings.

Original: We evaluated HWE on the standard supervised hypernymy identiﬁcation and a newly-proposed hierarchical hypernymy path completion tasks.
Revised: We evaluated HWE on the standard supervised hypernymy identiﬁcation and a newly-proposed hierarchical hypernymy path completion tasks.

Original: The experiments conducted in this paper on the above mentioned two tasks demonstrate that HWE was able to encode the hypernymy relations between words into the learnt embeddings, and reports an improvement over several previously proposed methods that learn either general word embeddings or hypernymy-speciﬁc word embeddings. 
Revised: The experiments conducted in this paper on the above mentioned two tasks demonstrate that HWE was able to encode the hypernymy relations between words into the learnt embeddings and reports an improvement over several previously proposed methods that learn either general word embeddings or hypernymy-speciﬁc word embeddings. Appendix A. Gradients The gradients of the objective given by (5) w.r.t. the variables w i , w j , b i and b j are computed as follows:

Revised: [Equation]