Compositional embedding models build a representation ( or embedding ) for a linguistic structure based on its component word embeddings . We propose a Feature-rich Compositional Embedding Model ( FCM ) for relation extraction that is expressive , generalizes to new domains , and is easy-to-implement . The key idea is to combine both ( unlexicalized ) handcrafted features with learned word embeddings . The model is able to directly tackle the difficulties met by traditional compositional embeddings models , such as handling arbitrary types of sentence annotations and utilizing global information for composition . We test the proposed model on two relation extraction tasks , and demonstrate that our model outperforms both previous compositional models and traditional feature rich models on the ACE 2005 relation extraction task , and the SemEval 2010 relation classification task . The combination of our model and a loglinear classifier with hand-crafted features gives state-of-the-art results . We made our implementation available for general use 1 . Two common NLP feature types are lexical properties of words and unlexicalized linguistic / structural interactions between words . Prior work on relation extraction has extensively studied how to design such features by combining discrete lexical properties ( e.g. the identity of a word , ⇤ ⇤ Gormley and Yu contributed equally . 1 https://github.com / mgormley / pacaya its lemma , its morphological features ) with aspects of a word 's linguistic context ( e.g. whether it lies between two entities or on a dependency path between them ) . While these help learning , they make generalization to unseen words difficult . An alternative approach to capturing lexical information relies on continuous word embeddings2 as representative of words but generalizable to new words . Embedding features have improved many tasks , including NER , chunking , dependency parsing , semantic role labeling , and relation extraction ( Miller et al . , 2004 ; Turian et al . , 2010 ; Koo et al . , 2008 ; Roth and Woodsend , 2014 ; Sun et al . , 2011 ; Plank and Moschitti , 2013 ; Nguyen and Grishman , 2014 ) . Embeddings can capture lexical information , but alone they are insufficient : in state-of-the-art systems , they are used alongside features of the broader linguistic context . In this paper , we introduce a compositional model that combines unlexicalized linguistic context and word embeddings for relation extraction , a task in which contextual feature construction plays a major role in generalizing to unseen data . Our model allows for the composition of embeddings with arbitrary linguistic structure , as expressed by hand crafted features . In the following sections , we begin with a precise construction of compositional embeddings using word embeddings in conjunction with unlexicalized features . Various feature sets used in prior work ( Turian et al . , 2010 ; Nguyen and Grishman , 2014 ; Hermann et al . , 2014 ; Roth and Woodsend , 2014 ) A feature that depends on the embedding for this context word could generalize to other lexical indicators of the same relation ( e.g. " operating " ) that do n't appear with ART during training . But lexical information alone is insufficient ; relation extraction requires the identification of lexical roles : where a word appears structurally in the sentence . In ( 2 ) , the word " of " between " suburbs " and " Baghdad " suggests that the first entity is part of the second , yet the earlier occurrence after " direction " is of no significance to the relation . Even finer information can be expressed by a word 's role on the dependency path between entities . In ( 3 ) we can distinguish the word " died " from other irrelevant words that do n't appear between the entities . tured as special cases of this construction . Adding these compositional embeddings directly to a standard log-linear model yields a special case of our full model . We then treat the word embeddings as parameters giving rise to our powerful , efficient , and easy-to-implement log-bilinear model . The model capitalizes on arbitrary types of linguistic annotations by better utilizing features associated with substructures of those annotations , including global information . We choose features to promote different properties and to distinguish different functions of the input words . The full model involves three stages . First , it decomposes the annotated sentence into substructures ( i.e. a word and associated annotations ) . Second , it extracts features for each substructure ( word ) , and combines them with the word 's embedding to form a substructure embedding . Third , we sum over substructure embeddings to form a composed annotated sentence embedding , which is used by a final softmax layer to predict the output label ( relation ) . The result is a state-of-the-art relation extractor for unseen domains from ACE 2005 ( Walker et al . , 2006 ) and the relation classification dataset from SemEval-2010 Task 8 ( Hendrickx et al . , 2010 ) . Contributions This paper makes several contributions , including : 1 . We introduce the Yu and Dredze ( 2015 ) and Yu et al . ( 2015 ) . Additionally , we have extended FCM to incorporate a low-rank embedding of the features ( Yu et al . , 2015 ) , which focuses on fine-grained relation extraction for ACE and ERE . This paper obtains better results than the low-rank extension on ACE coarse-grained relation extraction . We have presented FCM , a new compositional model for deriving sentence-level and substructure embeddings from word embeddings . Compared to existing compositional models , FCM can easily handle arbitrary types of input and handle global information for composition , while remaining easy to implement . We have demonstrated that FCM alone attains near state-of-the-art performances on several relation extraction tasks , and in combination with traditional feature based loglinear models it obtains state-of-the-art results . Our next steps in improving FCM focus on enhancements based on task-specific embeddings or loss functions as in Hashimoto et al . ( 2015 ; dos Santos et al . ( 2015 ) . Moreover , as the model provides a general idea for representing both sentences and sub-structures in language , it has the potential to contribute useful components to various tasks , such as dependency parsing , SRL and paraphrasing . Also as kindly pointed out by one anonymous reviewer , our FCM can be applied to the TAC-KBP ( Ji et al . , 2010 ) tasks , by replacing the training objective to a multi-instance multilabel one ( e.g. Surdeanu et al . ( 2012 ) ) . We plan to explore the above applications of FCM in the future .

Challenge: Existing methods for relation extraction do not work well on unseen words, and word embeddings alone are not sufficient without the broader linguistic contexts.
Approach: They propose a compositional embedding model which combines hand-crafted features with word embeddings by summing over substructure embeddings to form a composed annotated sentence embedding.
Outcome: The proposed model outperforms compositional models and traditional feature rich models on the ACE 2005 relation extraction task and SemEval 2010 relation classification task.