This paper presents a deep semantic similarity model ( DSSM ) , a special type of deep neural networks designed for text analysis , for recommending target documents to be of interest to a user based on a source document that she is reading . We observe , identify , and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents , which we collect from commercial Web browser logs . The DSSM is trained on millions of Web transitions , and maps source-target document pairs to feature vectors in a latent space in such a way that the distance between source documents and their corresponding interesting targets in that space is minimized . The effectiveness of the DSSM is demonstrated using two interestingness tasks : automatic highlighting and contextual entity search . The results on large-scale , real-world datasets show that the semantics of documents are important for modeling interestingness and that the DSSM leads to significant quality improvement on both tasks , outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models . Tasks of predicting what interests a user based on the document she is reading are fundamental to many online recommendation systems . A recent survey is due to Ricci et al . ( 2011 ) . In this paper , we exploit the use of a deep semantic model for two such interestingness tasks in which document semantics play a crucial role : automatic highlighting and contextual entity search . Automatic Highlighting . In this task we want a recommendation system to automatically discover the entities ( e.g. , a person , location , organi-zation etc . ) that interest a user when reading a document and to highlight the corresponding text spans , referred to as keywords afterwards . We show in this study that document semantics are among the most important factors that influence what is perceived as interesting to the user . For example , we observe in Web browsing logs that when a user reads an article about a movie , she is more likely to browse to an article about an actor or character than to another movie or the director . Contextual entity search . After identifying the keywords that represent the entities of interest to the user , we also want the system to recommend new , interesting documents by searching the Web for supplementary information about these entities . The task is challenging because the same keywords often refer to different entities , and interesting supplementary information to the highlighted entity is highly sensitive to the semantic context . For example , " Paul Simon " can refer to many people , such as the singer and the senator . Consider an article about the music of Paul Simon and another about his life . Related content about his upcoming concert tour is much more interesting in the first context , while an article about his family is more interesting in the second . At the heart of these two tasks is the notion of interestingness . In this paper , we model and make use of this notion of interestingness with a deep semantic similarity model ( DSSM ) . The model , extending from the deep neural networks shown recently to be highly effective for speech recognition ( Hinton et al . , 2012 ; Deng et al . , 2013 ) and computer vision ( Krizhevsky et al . , 2012 ; Markoff , 2014 ) , is semantic because it maps documents to feature vectors in a latent semantic space , also known as semantic representations . The model is deep because it employs a neural network with several hidden layers including a special convolutional-pooling structure to identify keywords and extract hidden semantic features at different levels of abstractions , layer by layer . The semantic representation is computed through a deep neural network after its training by backpropagation with respect to an objective tailored to the respective interestingness tasks . We obtain naturally occurring " interest " signals by observing Web browser transitions , from a source document to a target document , in Web usage logs of a commercial browser . Our training data is sampled from these transitions . The use of the DSSM to model interestingness is motivated by the recent success of applying related deep neural networks to computer vision ( Krizhevshy et al . 2012 ; Markoff , 2014 ) , speech recognition ( Hinton et al . 2012 ) , text processing ( Collobert et al . 2011 ) , and Web search ( Huang et al . 2013 ) . Among them , ( Huang et al . 2013 ) is most relevant to our work . They also use a deep neural network to map documents to feature vectors in a latent semantic space . However , their model is designed to represent the relevance between queries and documents , which differs from the notion of interestingness between documents studied in this paper . It is often the case that a user is interested in a document because it provides supplementary information about the entities or concepts she encounters when reading another document although the overall contents of the second documents is not highly relevant . For example , a user may be interested in knowing more about the history of University of Washington after reading the news about President Obama 's visit to Seattle . To better model interestingness , we extend the model of Huang et al . ( 2013 ) in two significant aspects . First , while Huang et al . treat a document as a bag of words for semantic mapping , the DSSM treats a document as a sequence of words and tries to discover prominent keywords . These keywords represent the entities or concepts that might interest users , via the convolutional and max-pooling layers which are related to the deep models used for computer vision ( Krizhevsky et al . , 2013 ) and speech recognition ( Deng et al . , 2013a ) but are not used in Huang et al . 's model . The DSSM then forms the high-level semantic representation of the whole document based on these keywords . Second , instead of directly computing the document relevance score using cosine similarity in the learned semantic space , as in Huang et al . ( 2013 ) , we feed the features derived from the semantic representations of documents to a ranker which is trained in a supervised manner . As a result , a document that is not highly relevant to another document a user is reading ( i.e. , the distance between their derived feature 1 We stress here that , although the click signal is available to form a dataset and a gold standard ranker ( to be described in vectors is big ) may still have a high score of interestingness because the former provides useful information about an entity mentioned in the latter . Such information and entity are encoded , respectively , by ( some subsets of ) the semantic features in their corresponding documents . In Sections 4 and 5 , we empirically demonstrate that the aforementioned two extensions lead to significant quality improvements for the two interestingness tasks presented in this paper . Before giving a formal description of the DSSM in Section 3 , we formally define the interestingness function , and then introduce our data set of naturally occurring interest signals . Modeling interestingness is fundamental to many online recommendation systems . We obtain naturally occurring interest signals by observing Web browsing transitions where users click from one webpage to another . We propose to model this " interestingness " with a deep semantic similarity model ( DSSM ) , based on deep neural networks with special convolutional-pooling structure , mapping source-target document pairs to feature vectors in a latent semantic space . We train the DSSM using browsing transitions between documents . Finally , we demonstrate the effectiveness of our model on two interestingness tasks : automatic highlighting and contextual entity search . Our results on large-scale , real-world datasets show that the semantics of documents computed by the DSSM are important for modeling interestingness and that the new model leads to significant improvements on both tasks . DSSM is shown to outperform not only the classic document models that do not use ( latent ) semantics but also stateof-the-art topic models that do not have the deep and convolutional architecture characterizing the DSSM . One area of future work is to extend our method to model interestingness given an entire user session , which consists of a sequence of browsing events . We believe that the prior browsing and interaction history recorded in the session provides additional signals for predicting interestingness . To capture such signals , our model needs to be extended to adequately represent time series ( e.g. , causal relations and consequences of actions ) . One potentially effective model for such a purpose is based on the architecture of recurrent neural networks ( e.g. , Mikolov et al . 2010 ; Chen and Deng , 2014 ) , which can be incorporated into the deep semantic model proposed in this paper .

Challenge: Recommending interesting documents to users is a fundamental feature of many systems, however; current systems do not directly model it but relevance.
Approach: They propose a CNN-based model that models documents as a sequence of words in latent space and trains it on millions of Web transitions.
Outcome: They show that their model outperforms classic/recent models on two tasks that evaluate the interestingness of documents significantly.