In this paper , we present a recursive neural network ( RNN ) model that works on a syntactic tree . Our model differs from previous RNN models in that the model allows for an explicit weighting of important phrases for the target task . We also propose to average parameters in training . Our experimental results on semantic relation classification show that both phrase categories and task-specific weighting significantly improve the prediction accuracy of the model . We also show that averaging the model parameters is effective in stabilizing the learning and improves generalization capacity . The proposed model marks scores competitive with state-of-the-art RNN-based models . Recursive Neural Network ( RNN ) models are promising deep learning models which have been applied to a variety of natural language processing ( NLP ) tasks , such as sentiment classification , compound similarity , relation classification and syntactic parsing ( Hermann and Blunsom , 2013 ; Socher et al . , 2012 ; Socher et al . , 2013 ) . RNN models can represent phrases of arbitrary length in a vector space of a fixed dimension . Most of them use minimal syntactic information ( Socher et al . , 2012 ) . Recently , Hermann and Blunsom ( 2013 ) proposed a method for leveraging syntactic information , namely CCG combinatory operators , to guide composition of phrases in RNN models . While their models were successfully applied to binary sentiment classification and compound similarity tasks , there are questions yet to be answered , e.g. , whether such enhancement is beneficial in other NLP tasks as well , and whether a similar improvement can be achieved by using syntactic information of more commonly available types such as phrase categories and syntactic heads . In this paper , we present a supervised RNN model for a semantic relation classification task . Our model is different from existing RNN models in that important phrases can be explicitly weighted for the task . Syntactic information used in our model includes part-of-speech ( POS ) tags , phrase categories and syntactic heads . POS tags are used to assign vector representations to word-POS pairs . Phrase categories are used to determine which weight matrices are chosen to combine phrases . Syntactic heads are used to determine which phrase is weighted during combining phrases . To incorporate task-specific information , phrases on the path between entity pairs are further weighted . The second contribution of our work is the introduction of parameter averaging into RNN models . In our preliminary experiments , we observed that the prediction performance of the model often fluctuates significantly between training iterations . This fluctuation not only leads to unstable performance of the resulting models , but also makes it difficult to fine-tune the hyperparameters of the model . Inspired by Swersky et al . ( 2010 ) , we propose to average the model parameters in the course of training . A recent technique for deep learning models of similar vein is dropout ( Hinton et al . , 2012 ) , but averaging is simpler to implement . Our experimental results show that our model per-Figure 1 : A recursive representations of a phrase " a word vector " with POS tags of the words ( DT , NN and NN respectively ) . For example , the two word-POS pairs " word NN " and " vector NN " with a syntactic category N are combined to represent the phrase " word vector " . forms better than standard RNN models . By averaging the model parameters , our model achieves performance competitive with the MV-RNN model in Socher et al . ( 2012 ) , without using computationally expensive word-dependent matrices . We have presented an averaged RNN model for semantic relation classification . Our experimental results show that syntactic information such as phrase categories and heads improves the performance , and the task-specific weighting is also beneficial . The results also demonstrate that averaging the model parameters not only stabilizes the learning but also improves the generalization capacity of the model . As future work , we plan to combine deep learning models with richer information such as predicateargument structures .

Challenge: Incorporating syntactic information into RNN models to guide the composition of phrases is shown effective for classification tasks but not studied for other tasks.
Approach: They propose a supervised RNN model which uses syntactic information to explicitly weight important phrases for a semantic relation classification task.
Outcome: Experiments show that using syntactic information such as phrase categories and heads, and also task-specific weighting contribute to improve performance.