U NSUPERVISED S ENTENCE E MBEDDING U SING D OCUMENT S TRUCTURE - BASED C ONTEXT
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a new unsupervised method for learning general-purpose sentence embeddings.
Unlike existing methods which rely on local contexts, such as words inside the sentence or immediately neighboring sentences, our method selects, for each target sentence, inﬂuential sentences in the entire document based on a document structure.
We identify a dependency structure of sentences using metadata or text styles.
Furthermore, we propose a novel out-of-vocabulary word handling technique to model many domain-speciﬁc terms, which were mostly discarded by existing sentence embedding methods.
We validate our model on several tasks showing 30% precision improvement in coreference resolution in a technical domain, and 7.5% accuracy increase in paraphrase detection compared to baselines.
I NTRODUCTION
Distributed representations are ever more leveraged to understand text (Mikolov et al., 2013a;b; Levy & Goldberg, 2014; Pennington et al., 2014).
Recently, Kiros et al.
(2015) proposed a neural network model, S KIP -T HOUGHT , that embeds a sentence without supervision by training the network to predict the next sentence for a given sentence.
However, unlike human reading with broader context and structure in mind, the existing approaches focus on a small continuous context of neighboring sentences.
These approaches work well on less structured text like movie transcripts, but do not work well on structured documents like encylopedic articles and technical reports.
To better support semantic understanding of such technical documents, we propose a new unsupervised sentence embedding framework to learn general-purpose sentence representations by leveraging long-distance dependencies between sentences in a document.
We observe that understanding a sentence often requires understanding of not only the immediate context but more comprehensive context, including the document title, previous paragraphs or even related articles as shown in Figure 1.
For instance, all the sentences in the document can be related to the title of the document (1(a)).
The ﬁrst sentence of each item in a list structure can be inﬂuenced by the sentence introducing the list (1(b)).
Moreover, html documents can contain hyperlinks to provide more information about a certain term (1(c)).
With the contexts obtained from document structure, we can connect ransomware with payment (1(a)) and the four hashes with Locky (1(b)).
[Table]
[Figure]
Our approach leveraging such structural elements has several advantages.
First, it can learn from technical documents containing several subtopics that may cause sudden context changes.
Some sentences have dependences to distant ones if a different perspective of the topic is introduced. Using [Table] et al
[Table] [Table] et [Table]
[Table]
[Table]
Our work
[Table]
only small neighboring context results in insufﬁcient input to the neural network.
Using long distance dependencies, we can provide a broader context.
Second, we can consider out-of-vocabulary (OOV) words using their information extracted from the structural context.
The vocabulary in a neural network is always limited due to costly training time and memory use.
An existing method discarding low frequency words results in losing important keywords in the technical domain.
We validate our model on several NLP tasks using a Wikipedia corpus.
When trained with the
Wikipedia corpus, our model produces much lower loss than S KIP -T HOUGHT in the target sentence prediction task, conﬁrming that training with only local context does not work well for such documents.
We also compare the performance of the learned embedding on several NLP tasks including coreference resolution and paraphrase identiﬁcation.
For coreference resolution, our model shows roughly 30% improvement in precision over a state-of-the-art deep learning-based approach on cybersecurity domain, and produces 7.5% increase in accuracy compared with S KIP -T HOUGHT for paraphrase identiﬁcation.
The main contributions of the paper include:
• We propose a general-purpose sentence embedding method which leverages long distance sentence dependencies extracted from the document structure.
• We developed a rule-baed dependency annotator to automatically determine the document structure and extract all governing sentences for each sentence.
• We also present a new OOV handling technique based on the document structure.
• We have applied our methods to several NLP applications using cybersecurity datasets.
The experiments show that our model consistently outperform existing methods.
R ELATED WORK
Distributed representation of sentences, which is often called sentence embedding, has gained much attention recently, as word-level representations (Mikolov et al., 2013a;b; Levy & Goldberg, 2014;
Pennington et al., 2014) are not sufﬁcient for many sentence-level or document-level tasks, such as machine translation, sentiment analysis and coreference resolution.
Recent approaches using neural networks consider some form of dependencies to train the network.
Dependencies can be continuous (relating two adjacent words or sentences) or discontinuous (relating two distant words or sentences), and intra-sentence (dependency of words within a sentence) or inter-sentence (dependency between sentences).
Many sentence embedding approaches leverage these dependencies of words to combine word embeddings, and can be categorized as shown in 1.
One direct extension of word embedding to sentences is combining words vectors in a continuous context window.
Kusner et al.
(2015) use a weighted average of the constituent word vectors.
Wieting & Gimpel (2017), Conneau et al.
(2017), and Palangi et al.
(2016) use supervised approaches to train a long short-term memory (LSTM) network that merges word vectors.
Kalchbrenner et al.
(2014) and Kim (2014) use convolutional neural networks (CNN) over continuous context window to generate sentence representations.
Le & Mikolov (2014) include a paragraph vector in the bag of word vectors, and apply a word embedding approaches (Mikolov et al., 2013a;b).
Recently, several researchers have proposed dependency-based embedding methods using a dependency parser to consider discontinuous intra-sentence relationships (Socher et al., 2011; Ma et al., 2015; Tai et al., 2015).
Socher et al.
(2011) uses recursive neural network to consider discontinuous dependencies.
Ma et al.
(2015) proposes a dependency-based convolutional neural network which concatenate a word with its ancestors and siblings based on the dependency tree structure.
Tai et al.
(2015) proposes tree structured long short-term memory networks.
These studies show that dependency-based (discontinuous) networks outperform their sequential (continuous) counterparts.
Unlike these approaches, considering only intra-sentence dependencies, S KIP -T HOUGHT (Kiros et al., 2015) joins two recurrent neural networks, encoder and decoder.
The encoder combines the words in a sentence into a sentence vector, and the decoder generates the next sentence.
Our approach is similar to S KIP -T HOUGHT since both approaches are unsupervised and use inter-sentential dependencies.
However, S KIP -T HOUGHT considers only continuous dependency.
D OCUMENT S TRUCTURED - BASED C ONTEXT
Previous methods use intra-sentence dependencies such as dependency tree, or immediately neighboring sentences for sentence embedding.
However, we identify more semantically related content to a target sentence based on the document structure as shown in Figure 1.
In this section, we describe a range of such inter-sentence dependencies that can be utilized for sentence embedding and the techniques to automatically identify them.
We use the following notations to describe the extraction of document structure-based context for a given sentence.
Suppose we have a document
D = { S 1 , . .
. , S | D | } , which is a sequence of sentences.
Each sentence S i is a sequence of words: s i, 1 , . . .
, s i, | S
i | .
For each target sentence S t ∈ D , there can be a subset G ⊂ D that S t depends on (For simplicity, we use G to denote a S t speciﬁc set).
We call such a sentence in G a governing sentence of S t , and say G i governs S t , or S t depends on G i .
Each G i is associated with S t through one of the dependency types in D described below.
3. T ITLES
The title of a document, especially a technical document, contains the gist of the document, and all other sentences support the title in a certain way.
For instance, the title of the document can clarify the meaning of a deﬁnite noun in the sentence.
Section titles play a similar role, but, mostly to the sentences within the section.
We detect different levels of titles, starting from the document title to chapter, section and subsection titles.
Then, we identify the region in the document which each title governs and incorporate the title in the embedding of all the sentences in the region.
To identify titles in a document, we use various information from the metadata and the document content.
Document Metadata ( D TM ):
We extract a document title from the < title > tag in a HTML document or from the title ﬁeld in Word or PDF document metadata.
Since the document title inﬂuences all sentences in a document, we consider a title obtained from D TM governs every sentence in D .
Heading Tag ( D THn ):
The heading tags < h1 > to < h6 > in HTML documents are often used to show document or section titles.
We consider all the sentences between a heading tag and the next occurrence of the same level tag are considered under the inﬂuence of the title.
Table Of Contents ( D TC ): Many documents contain a table of contents (TOC) providing the overall structure of the document.
To detect the titles based on the table of contents, we ﬁrst recognize a phrase indicating TOC, such as “table of contents”, “contents” or “index”.
Then, we parse the content following the cue phrase and check if it contains a typical TOC pattern such as “Chapter– Introduction” or “Introduction · · · · · · · · · 8”.
The range of each section can be easily identiﬁed from the TOC.
If the document is a HTML ﬁle, each line in the TOC tends to have a hyperlink to the corresponding section.
For non-HTML documents, we can extract the page number from the TOC ( e.g. , page 8) and locate the corresponding content if the document includes the page numbers.
Header and Footer ( D TR ): Technical documents often contain the document or section titles in the headers or footers.
Thus, if the same text is repeated in the header or in the footer in many pages, we take the text as a title and consider all the sentences appearing in these pages belong to the title.
Text Styles ( D TS ): Titles often have a distinctive text style including a different font style ( e.g. , italics or bold ), a bigger font size or capitalized words.
They also tend to be shorter than typical sentences and have no period at the end.
We apply this heuristic to detect titles when the document does not contain any tags or a TOC.
Then, we split the document based on the detected titles and treat each slice as a section.
3. L ISTS
Authors often employ a list structure to describe several elements of a subject.
These list structures typically state the main concept ﬁrst, and, then, the supporting points are described in a bulleted, numbered or in-text list as illustrated in Figure 2.
In these lists, an item is conceptually more related to the introductory sentence than the other items in the list, but the distance can be long because of other items.
Once list items are identiﬁed, we consider the sentence appearing prior to the list items as the introductory sentence and assume that it governs all the items in the list.
The categories of the products State Farm offers are as follows:
• We have property and casualty insurance.
• We offer comprehensive types of life and health insurances.
• We have bank products.
Formatted List ( D LF ):
To extract numbered or bulleted lists, we use the list tags ( e.g. , < ul > , < ol > , < li > ) for HTML documents.
For non-HTML documents, we detect a number sequence ( i.e. , 1, 2, ...) or bullet symbols ( e.g. , -, · ) repeating in multiple lines.
In-text List ( D LT ): We also identify in-text lists such as “First(ly), . . . .
Second(ly), . . . .
Last(ly), . . . ”
by identifying these cue words.
We consider the sentence appearing prior to the list items as the introductory sentence and assume that it governs the list items.
3. L INKS
Hyperlinks ( D H ): Some sentences contain hyperlinks or references to provide additional information or clarify the meaning of the sentence.
We can enrich the representation of the sentence using the linked document.
In this work, we use the title of the linked document in the embedding of the sentence.
Alternatively, we can use the embedding of the linked document.
Footnotes and In-document Links ( D F ): Footnotes also provide additional information for the target sentence.
In an HTML document, such information is usually expressed with in-document hyperlinks, which ends with “#dest”.
In this case, we identify a sentence marked with “#dest” and add a dependency between the two sentences.
3. W INDOW - BASED C ONTEXT ( D Wn ):
We also consider the traditional sequential dependency used in previous methods (Kiros et al., 2015;
Gan et al., 2017).
Given a document D = { S 1 , . . .
, S | D | } , the target sentence S t is considered to be governed by n sentences prior to ( n < 0 ) or following ( n > 0 ) S t .
In our implementation, we use only one left sentence.
N EURAL N ETWORK M ODELS
Similarly to S KIP -T HOUGHT (Kiros et al., 2015), we train our model to generate a target sentence
S t using a set of governing sentences G .
However, S KIP -T HOUGHT takes into account only the window-based context ( D Wn ), while our model considers diverse long distance context.
Furthermore, we handle (OOV) words based on their occurrences in the context.
Our model has several encoders (one encoder for each G i ∈ G ), a decoder and an OOV handler as shown in Figure 3.
The input to each cell is a word, represented as a dense vector.
In this work, we use the pre-trained vectors from the CBOW model (Mikolov et al., 2013b), and the word vectors can be optionally updated during the training step.
Unlike existing sentence embedding methods, which include only a small fraction of words (typically high frequency words) in the vocabulary and map all other words to one OOV word by averaging all word vectors, we introduce a new OOV handler in our model.
The
OOV handler maps all OOV words appearing in governing sentences to placeholders and extend the vocabulary with the OOV placeholders. More details about OOV handler is described in Section 5. We now formally describe the model given a target sentence S t and a set G of its governing sentences. We ﬁrst describe the encoders that digest each G i ∈ G .
Given the i -th governing sentence
[Figure]
for [Figure] sentence ………
……………… 𝑠 " #$ 𝑠 % ………
Title 1 Footnote … 𝐺 " #$ 𝐺 ’() 𝐺 *
S t 𝑔 *,) 𝑔 *,- 𝑔 *,. / 𝑔 " #$ ,) 𝑔 " #$ ,- 𝑔 " #$ ,.
0#$ 𝑔 ’(),) 𝑔 ’(),- 𝑔 ’(),.
12$ ℎ ’() ℎ " #$ ℎ * ℎ4 5 𝜆 ’()
OOV
[Figure]
OOV Mapper 𝑑 ’() 𝑑 " #$ 𝑑 *
OOV
[Figure]
G i =
( g i, 1 , . . .
, g i, | G i | ) let w ( g i,t ) be the word representation (pre-trained or randomly initialized) of word g i,t .
Then, the following equations deﬁne the encoder for S i .
[Equation]
where RC is a recurrent neural network cell (e.g., LSTM or GRU) that updates the memory
h
i,t ; θ E is the parameters for the encoder RC ; λ i is an OOV weight vector that decides how much we rely on out-of-vocabulary words; d i denotes the OOV features for G i ; U and g are linear regression parameters; S ( · ) is the sigmoid function; u dep and a dep are dependency-speciﬁc weight parameters; W
[Equation] ( i ) is the transformation matrix for the dependency type dep( i ) ; and ¯ h 0 is the aggregated information of G and is passed to the decoder for target sentence generation.
Now, we deﬁne the decoder as follows:
[Equation]
where RC is a recurrent neural network cell that updates the memory ¯ h t and generates the output
o t ; θ D is a set of parameters for the decoder RC ; σ ( · ) is the softmax function; and V o t + c transforms the output into the vocabulary space.
That is, V o t + c generates logits for words in the vocabulary set and is used to predict the words in the target sentence.
To strike a balance between the model accuracy and the training time, we use K randomly chosen governing sentences from G for all target sentence.
We use the cross entropy between y t and o t as the optimization function and update θ E , W dep( i ) , b, V, c, θ D and optionally w ( · ) .
O UT - OF - VOCABULARY (OOV) MAPPING
Incorporating all the words from a large text collection in deep learning models is infeasible, since the amounts of memory use and training time will be too costly.
Existing sentence embedding techniques reduce the vocabulary size mainly by using only high frequency words.
Then, they create an arbitrary word vector by averaging all the word vectors in the vocabulary and map all the discarded words into the vector.
However, this frequency-based ﬁltering can loose many important words including domain-speciﬁc words and proper nouns and can produce unsatisfactory results for technical documents.
Speciﬁcally, OOV word handling is desired in the following three places: (1)
Input embeddings to encode the governing sentences ( G ); (2) Input embeddings to decode the target sentence ( S t ); and (3) Output logits to compute the loss with respect to S t .
For the input embeddings of G or S t , we map all OOV words to the average vector of the embeddings of all other words in the sentence in the same way as other methods.
However, when we produce the word labels (output logits) for S t , we handle the OOV words as follows. We ﬁrst select N most frequent words in the training corpus as an initial vocabulary V 0 in the same way as existing methods. Note that N (typically, tens of thousands) is much smaller than the size of a training corpus (typically, millions or billions). How can we model this many OOV words without deteriorating the
Function BuildOOVMap ( G, Vocabulary)
Input : A governing sentence set G = { G 1 , . .
, G | G | } and a vocabulary V
Output: OOV Map foreach G i ∈
G do
OOVWords i ← { w j ∈ G i | w j / ∈ V, j = 1 , . . .
, η }
WtoPL i ← { w j
→ O i ( j ) | w j ∈ OOVWords i , j = 1 , . . .
, | OOVWords i |} return (cid:83) i
WtoPL i system performance?
We note that only the OOV words appearing in governing sentences inﬂuence in model training, and many semantically important words tend to appear in the beginning or at the end of the governing sentences.
Based on these observations, we select only the ﬁrst and the last η OOV words in a governing sentences, but, other selection methods can be used.
Further, to minimize the vocabulary size increase, we map individual OOV words to OOV variables based on the dependency type and the location of the OOV words.
We denote OOV variables by O i ( j ) , where i denotes the i -th dependency type and j is the index of the OOV word in the governing sentence.
The size of the augmented vocabulary is clear.
Let D denote the a set of all possible dependency types.
For each dependency i ∈ D , we add 2 × η OOV variables, resulting in 2 × η × |D| new words, and, thus, the extended vocabulary V = V 0 ∪ { 2 × η × |D| OOV variables } ).
We conduct experiments with 10 dependency types as described in Section 3 and set η to 4 adding 80 unique
OOV variables to the vocabulary.
Note that the mapping between OOV words and OOV variables is many-to-many.
For example, suppose “We discuss Keras ﬁrst’ is a target sentence S t , and, “Slim and
Keras are two tools you must know” is extracted as the document title by the dependency type D TS , “PLA’s weekly review: Slim and Keras are two tools you must know” is extracted as the document title by D TM for S t , and, words ‘Slim’, ‘Keras’ and ‘PLA’ are OOV words.
Then, we map the ‘Slim’ and ‘Keras’ from the ﬁrst title to OOV variable O TS (1) and O TS (2) and ‘PLA’, ‘Slim’ and ‘Keras’ from the second title to O TM (1) , O TM (2) , and O TM (3) respectively.
As a result, ‘Keras’ in S t is mapped to O TS (1) and O TM (3) .
Our OOV handler performs the following steps.
First, we build an OOV map to convert OOV words to OOV variables and vice versa.
Algorithm 1 summarizes the steps to build a map which converts the ﬁrst η OOV words into OOV variables.
To model the last η OOV words, we reverse the words in each G i , and index them as w − 1 , w − 2 , . .
, then pass them to BuildOOVMap to construct
O i ( − 1) , O i ( − 2) , . . .
, O i ( − η ) .
Once we have the OOV mapping and the augmented vocabulary, we can formulate an optimization goal taking into account the OOV words.
The optimization goal of each RNN cell without OOV words is to predict the next word with one correct answer.
In contrast, our model allows multiple correct answers, since an OOV word can be mapped to multiple OOV variables.
We use the cross entropy with soft labels as the optimization loss function.
The weight of each label is determined by the inverse-square law, i.e., the weight is inversely proportional to the square of the number of words associated with the label.
This weighting scheme gives a higher weight to less ambiguous dependency.
One additional component we add related to OOV words is a weight function for the governing sentences based on occurrences of proper nouns ( λ i in Equation 1).
Instead of equally weighing all governing sentences, we can give a higher weight to sentences with proper nouns, which are more likely to be OOV words.
Thus, we introduce a feature vector representing the number of OOV proper nouns in the i -th governing sentence ( d i in Equation 1).
Currently, the features include # of OOV words whose initials are uppercased, # of OOV words that are uppercased, and # of OOV words with any of the letters are uppercased.
Together with the linear regression parameters, U and g , the model learns the weights for different dependency types.
E XPERIMENTS
In this section, we empirically evaluate our approach on various NLP tasks and compare the results with other existing methods.
We trained the proposed model (O URS ) and the baseline systems on [Table] [Table] 807,647 randomly selected documents from the 2009 Wikipedia dump, after removing the discussion and resource ( e.g. , images) sections from the wiki pages.
Since our approach leverages HTML tags to identify document structures, our model used the raw HTML ﬁles.
For the baseline systems, we provided plain text version of the same articles.
All models were trained for 300K steps with 64-sized batches and the Adagrad optimizer (Duchi et al., 2011).
For the evaluation, we used up-togoverning sentences as the context for a target sentence.
When a sentence has more than 8 governing sentences, we randomly choose 8 sentences.
We set the maximum number of words in a sentence to be 30 and padded each sentence with special start and end of sentence symbols.
6. T ARGET SENTENCE PREDICTION
Unlike most other approaches, our model and S KIP -T HOUGHT (Kiros et al., 2015) can learn application-independent sentence representations without task-speciﬁc labels.
Both models are trained to predict a target sentence given a set of context.
For a quantitative evaluation between the two models, we compare the prediction losses by using the same loss function.
We randomly chose 640,000 target sentences for evaluation and computed the average loss over the 640K sentences.
Since S KIP -T HOUGHT cannot handle OOV words, we excluded the loss values corresponding to the OOV words from both models and denoted them as O URS -OOV and S KIP -T HOUGHT -OOV.
However, OOV denotes our model including the loss values of OOV words.
Table 2 shows the comparison of the three models.
As we can see, both O URS -OOV and O URS outperform S KIPT HOUGHT -OOV resulting in 19% and 2.5% reduction in the loss values respecitively.
6. P ARAPHRASE DETECTION
Further, we compare our model with S KIP -T HOUGHT on a paraphrase detection task using the
Microsoft Research Paraphrase corpus (Microsoft, 2016).
The data consists of 5,801 sentence pairs extracted from news data and the assessment (if a sentence is a paraphrase or not), which were determined by three assessors using majority voting.
We used 4,076 pairs for training and 1,725 pairs for testing.
Since the data sets contains sentence pairs only and no structural context, we evaluate only the effectiveness of the trained encoder.
To compare the qualities of sentence embeddings by the two models, we use the same logistic regression classiﬁer with features based on embedded sentences as in (Kiros et al., 2015).
Given a pair of sentences S 1 and S 2 , the features are the two embeddings of S 1 and S 2 , their entry-wise absolute difference, and their entry-wise products.
Our model shows a 5% points higher accuracy than S KIP -T HOUGHT in paraphrase detection (Table 3), demonstrating the effectiveness of our encoder trained with the structural dependencies.
Note that
S KIP -T HOUGHT trained on the Wikipedia corpus performs worse than a model trained on books or movie scripts due to more sophisticated and less sequential structure in Wikipedia documents.
6. C OREFERENCE RESOLUTION
Traditionally, the coreference resolution problem is considered as a pairwise classiﬁcation problem or clustering problem using handcrafted features (Clark & Manning, 2016).
We show that our approach can be used to extract features for coreference resolution and perform coreference resolution without handcrafted features and supervision.
We treat the coreference resolution problem as an inference problem given the context .
We ﬁrst identify a pronoun or an entity reference ( e.g. , a deﬁnite noun phrase) and build a list of candidate referents that conform to the pronoun type and the entity type of the reference.
Then, we replace the entity reference with each of the candidate referents and choose the one with the lowest loss value as the result, if the ratio of its loss to the original sentence loss value is less than a threshold value θ .
[Table]
[Figure]
[Table]
[Figure]
We compare our approach with the Stanford Deep Coreference Resolution tool (Clark & Manning, 2016) (denoted as D EEP C OREF henceafter), that leverages both handcrafted features and word embeddings to build distributed representations of coreference cluster pairs.
We conducted experiments for several cybersecurity related entity types such as ‘Malware’ and ‘Operating System’ in addition to general entity types including ‘Person’ and ‘Organization’.
The evaluation dataset consists ofcoreferences from 38 Wikipedia articles about malware programs that appeared after year 2009 and were not included in the training document set.
For the evaluation, we set θ to 0 .
99 and 1 . 00 .
Table 4 summarizes the results of the two systems.
Our model achieves higher precision and recall than D EEP C OREF .
Since D EEP C OREF was trained for a general domain, its overall performance on domain speciﬁc documents is very low.
Figure 4 shows the two systems’ performance on different entity types.
As we can see, O URS works well for domain speciﬁc entities such as ‘Malware’ and ‘Vulnerability’, while D EEP C OREF shows higher precision for ‘Person’ and ‘Organization’.
The reason O URS performs worse for ‘Person’ and ‘Organization’ is the security documents do not provide much information to ﬁnd dependencies as evidence to infer the coreferences.
6. D EPENDENCY IMPORTANCE
Note that u dep in Equation 1 denotes the importance level of a dependency dep .
In Table 5, we show the relative importance of the different dependencies compared to the sequential dependency ( D W, − 1 ), which is used in other methods.
The values show their relative importance levels when the importance of D W, − 1 is set to 1.
As we can see, all levels of document and section titles, except the fourth level subsection title, play a much more signiﬁcant role than the sequential dependency.
The reason the title from the metadata, ( D TM ), does not have a high weight as the title from the heading 1 tag ( D TH 1 ) is that the metadata contains extra text, “- Wikipedia”, in the title for Wikipedia articles ( e.g. , “George W. Bush - Wikipedia” instead of “George W. Bush”).
Further, hyperlinks ( D H ), indocument links ( D F ) and formatted lists ( D LF ) are all shown to have a similar inﬂuence as the sequence dependency.
The remaining dependencies, D TC , D TR , D TS , and D LT are scarcely found in the Wikipedia corpus, and thus, did not converge or were not updated.
[Table]
C ONCLUSION
In this paper, we presented a novel sentence embedding technique exploiting diverse types of structural contexts and domain-speciﬁc OOV words.
Our method is unsupervised and applicationindependent, and it can be applied to various NLP applications.
We evaluated the method on several
NLP tasks including coreference resolution, paraphrase detection and sentence prediction.
The results show that our model consistently outperforms the existing approaches conﬁrming that considering the structural context generates better quality sentence representations.