Original: M ULTI -T ASK L EARNING FOR D OCUMENT R ANKING AND Q UERY S UGGESTION
Revised: M ULTI -T ASK L EARNING FOR D OCUMENT R ANKING AND Q UERY S UGGESTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a multi-task learning framework to jointly learn document ranking and query suggestion for web search.
Revised: We propose a multi-task learning framework to jointly learn document ranking and query suggestion for web search.

Original: It consists of two major components,  document ranker and  query recommender.
Revised: It consists of two major components, a document ranker, and a query recommender.

Original: Document ranker combines current query and session information and compares the combined representation with document representation to rank the documents.
Revised: Document ranker combines current query and session information and compares the combined representation with document representation to rank the documents.

Original: Query recommender tracks users’ query reformulation sequence considering all previous in-session queries using a sequence to sequence approach.
Revised: Query recommender tracks users’ query reformulation sequence considering all previous in-session queries using a sequence to sequence approach.

Original: Both components are trained across search sessions by sharing parameters through session recurrence, which encodes session information.
Revised: As both tasks are driven by the users’ underlying search intent, we perform joint learning of these two components through session recurrence, which encodes search context and intent.

Original: Comprehensive experiments including rigorous comparisons with state-of-the-art techniques are performed on the public AOL search log, and the promising results endorse the effectiveness of the joint learning framework.
Revised: Extensive comparisons against state-of-the-art document ranking and query suggestion algorithms are performed on the public AOL search log, and the promising results endorse the effectiveness of the joint learning framework.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Understanding users’ information need is the key to optimize a search engine for providing relevant search results.
Revised: Understanding users’ information need is the key to optimize a search engine for providing relevant search results.

Original: Search engine logs have become an important resource to mine users’ search intent and click behavior (Baeza-Yates et al., 2004; Croft et al., 2010).
Revised: Search engine logs have been extensively used to mine users’ search intent, reﬂected in their search result click preferences and query reformulations (Baeza-Yates et al., 2004; Croft et al., 2010).

Original: In particular, user query logs are partitioned into search sessions, i.e., sequences of queries  issued by the same user and within a short time interval.
Revised: In particular, user query logs are partitioned into search sessions, i.e., sequences of queries and clicks issued by the same user and within a short time interval.

Original: In a search session, a user submits a sequence of queries and clicks on the ranked documents that he/she believes can best serve his/her information need.

Original: Thus, leveraging the user search behaviors within a query session, a.k.a.

Original: context-awareness provides useful  information about user intent and helps to narrow down ambiguity while ranking documents for the current query and predicting next query that users will submit (Jiang et al., 2014).
Revised: Search sessions provide useful contextual information about user intent and help to narrow down ambiguity while ranking documents for the current query and predicting next query that users will submit, a.k.a. contextawareness (Jiang et al., 2014).

Original: Since both  user’s click behavior and query reformulation are driven by the underlying  intent, we argue thatby jointly modeling both tasks can beneﬁt each other.
Revised: Since both a user’s click behavior and query reformulation are driven by the underlying search intent, we argue that jointly modeling both tasks can beneﬁt each other.

Original: In this work, we propose a joint learning framework, called multi-task neural session relevance framework (M-NSRF) to predict users’  future queries and clicks.
Revised: In this work, we propose a joint learning framework, called multi-task neural session relevance framework (M-NSRF), to predict users’ result clicks and future queries in search sessions.

Original: We assume a search  session is composed of several queries and corresponding clicks that can supplement a user’s underlying intent for a search task and thus helps to improve document ranking and query suggestion. Inspired by recent works (Collobert & Weston, 2008; Liu et al., 2015), we consider a deep neural network approach that allows us to learn multiple tasks by sharing model parameters, latent states, and optimizing a joint objective function.
Revised: We model search context within a session via a recurrent latent state in a deep neural network (Collobert & Weston, 2008; Liu et al., 2015), which governs the generation of result clicks in the current query and formation of next query. By sharing the latent states across the tasks of document ranking and query suggestion, we learn the representations of queries, documents and user intent carrying over the whole session jointly, i.e., multi-task learning.

Original: The joint learning approach improves generalizationon both tasks, and allows us to learn latent representations of user intent carrying over the whole session.
Revised: Given the fact that web search queries are usually very short and the number of clicks in each query is also very few, our joint learning framework greatly reduces the sample complexity in learning each individual task and effectively improves generalization of both learnt models.

Original: The general workﬂow of M-NSRF is illustrated in Figure 1.
Revised: The general workﬂow of M-NSRF is illustrated in Figure 1.

Original: Given a sequence of queries from the same search session, “cheap furniture”, “craig list virginia” , M-NSRF is trained to predict the  next query, “cheap furniture for sale” and corresponding result clicks.
Revised: Given a sequence of queries from the same search session, “cheap furniture” and “craig list virginia” , M-NSRF is trained to predict the result clicks under the current query, and the next query, “cheap furniture for sale.”

Original: It is evident that in this search session the user kept reformulating the queries because his/her information need has not been satisﬁed by the  previously clicked documents, which is reﬂected in the added and removed query terms; and such revisions suggest what he/she might want to click next.
Revised: It is evident that in this search session the user kept reformulating the queries because his/her information need has not been met by the result clicks in the previously submitted queries, which is reﬂected in the added and removed query terms. And such revisions suggest what he/ she might want to click next.

Original: We argue that modeling session-level latent states which carry information about previous queries and clicks is crucial in understanding the true user intent, and the learning of such latent states can be aided by both document ranking and query prediction tasks.
Revised: As a result, the modeling of result clicks and query reformulations mutually reinforce each other in revealing users’ underlying search intent. In M-NSRF, we model user search intent as a sessionlevel recurrent state, the learning of which is aided by both document ranking and query prediction tasks.

Original: Our proposed framework is not restricted to any speciﬁc architecture to represent queries and documents and can be trained end-to-end on search sessions.
Revised: Our proposed framework is not restricted to any speciﬁc architecture to represent queries and documents and can be trained end-to-end on search sessions.

Original: We evaluate the effectiveness of the proposed framework using the publicly available AOL search log and compare with several well-known classical retrieval models, as well as variousstate-of-theart deep neural  models speciﬁcally designed for ad-hoc retrieval.
Revised: We evaluate the effectiveness of the proposed framework using the publicly available AOL search log and compare with several well-known classical retrieval models, as well as various neural retrieval models speciﬁcally designed for ad-hoc retrieval.

Original: We also compare M-NSRF with several baseline models for thequery prediction task.
Revised: We also compare M-NSRF with several baseline models for the prediction task.

Original: The empirical results show that M-NSRF outperforms state-of-the-art models signiﬁcantly
Revised: empirical forms state- -art ad task performance

Original: To summarize, the key contributions of this work include:
Revised: To summarize the key contributions of this work include:

Original: We propose a novel multi-task neural session relevance model that is jointly trained on document ranking and query suggestion tasks by utilizing in-session queries and clicks in a holistic way.
Revised: We propose a novel multi-task neural session relevance model that is jointly trained on document ranking and query suggestion tasks by utilizing in-session queries and clicks in a holistic way.

Original: We conduct rigorous comparisons over classical and neural state-of-the-art retrieval models and query prediction techniques using AOL search log and demonstrate the joint framework can improve existing models in both tasks.

Original: We provide detailed analysis and will release the code and  data  to facilitate future research.
Revised: We provide detailed experiment analysis, and will release the implementation and the data processing tool to facilitate future research.

Original:  R ELATED W ORK AND B ACKGROUND
Revised:  R ELATED W ORK AND B ACKGROUND

Original: Ad-hoc Retrieval.
Revised: Ad-hoc Retrieval.

Original: Traditional retrieval models such as query likelihood (Ponte & Croft, 1998) and
Revised: Traditional retrieval models such as query likelihood (Ponte & Croft, 1998) and

Original: BM25 (Robertson et al., 2009) are based on exact matching of query and document words with a variety of smoothing, weighting and normalization techniques.
Revised: BM25 (Robertson et al., 2009) are based on exact matching of query and document words with a variety of smoothing, weighting and normalization techniques.

Original: Recently, deep neural network based approaches demonstrate strong performance in ad-hoc retrieval.
Revised: Recently, deep neural network based approaches demonstrate strong advance in ad-hoc retrieval.

Original: Existing neural ranking models fall into two categories: representation focused (Huang et al., 2013; Gao et al., 2014) and interaction focused (Guo et al., 2016b).
Revised: Existing neural ranking models fall into two categories: representation focused (Huang et al., 2013; Gao et al., 2014) and interaction focused (Guo et al., 2016b).

Original: The earlier focus of neural IR was mainly on representation based models (Hu et al., 2014;
Revised: The earlier focus of neural ranking models was mainly on representation based models (Hu et al., 2014;

Original: Shen et al., 2014), in which the query and documents are ﬁrst embedded into continuous vectors, and the ranking is calculated from their embeddings’ similarity.
Revised: Shen et al., 2014), in which the query and documents are ﬁrst embedded into continuous vectors, and the ranking is calculated based on their embeddings’ similarity.

Original: The interaction focused neural models (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016a), on the other hand, learn query-document matching patterns from word-level interactions.
Revised: The interaction focused neural models (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016a), on the other hand, learn query-document matching patterns from word-level interactions.

Original: Both the interaction and representation focused models can be combined for further improvements (Mitra et al., 2017).
Revised: Both the interaction and representation focused models can be combined for further improvements (Mitra et al., 2017).

Original: Similarly, Jaech et al.
Revised: Similarly, Jaech et al.

Original: (2017) captures both local relevance matching and global topicality signals when computing relevance of a document to a query.
Revised: (2017) captures both local relevance matching and global topicality signals when computing relevance of a document to a query.

Original: Our work falls into the representation focused 
Revised: Our work falls into the representation focused approach to form query and document representations and jointly models the two tasks through session representation learning.

Original: Query Suggestion.
Revised: Query Suggestion.

Original: In general, query suggestion algorithms use clustering methods to ﬁnd  similar queries so that they can be used as suggestions for one another (Wen et al., 2001; Baeza-Yates et al., 2004).
Revised: In general, query suggestion algorithms deﬁne various distance metrics between queries to ﬁnd the most similar ones as suggestions for one another (Wen et al., 2001; Baeza-Yates et al., 2004).

Original: The closest work to ours is the end-to-end hierarchical recurrent encoder-decoder architecture (HRED-qs)
Revised: The closest work to ours is the end-to-end hierarchical recurrent encoder-decoder architecture (HRED-qs)

Original: (Sordoni et al., 2015), which ranks candidates for context-aware query suggestion.
Revised: (Sordoni et al., 2015), which ranks candidates for context-aware query suggestion.

Original: Our proposed frameworkmainly differs from HRED-qs as it exploits user clicks as contextual information.
Revised: Our proposed framework differs from HRED-qs as it integrates another important type of user search behavior, i.e., result clicks, into the joint modeling, which provides additional contextual information for query suggestion.

Original: Similarly, Mitra & Craswell (2015) proposed a candidate generation approach for rare preﬁxes using frequently observed query sufﬁxes and suggested a neural model to generate ranking features along with n-gram features. 
Revised: Mitra & Craswell (2015) proposed a candidate generation approach for rare preﬁxes using frequently observed query sufﬁxes and suggested a neural model to generate ranking features along with n-gram features. Similarly, a large pool of previous works investigated task or context-aware approaches (Cao et al., 2008; He et al., 2009; Feild & Allan, 2013; Chen et al., 2017; Garigliotti & Balog, 2017) for query suggestion.

Original: Multi-task Learning.
Revised: Multi-task Learning.

Original: The goal of multi-task learning is to improve generalization on the target task by leveraging the domain-speciﬁc information contained in the training signals of related tasks (Caruana, 1998).
Revised: The goal of multi-task learning is to improve generalization on the target task by leveraging the domain-speciﬁc information contained in the training signals of related tasks (Caruana, 1998).

Original: Multi-task learning in combination with deep neural networks has been successfully used in many application scenarios, including natural language processing (Collobert & Weston, 2008; Liu et al., 2016; Peng et al., 2017), speech recognition (Deng et al., 2013; Thanda & Venkatesan, 2017) and computer vision (Girshick, 2015).
Revised: Multi-task learning in combination with deep neural networks has been successfully used in many application scenarios, including natural language processing (Collobert & Weston, 2008; Liu et al., 2016a;b; Peng & Dredze, 2017; Peng et al., 2017), speech recognition (Deng et al., 2013; Thanda & Venkatesan, 2017) and computer vision (Girshick, 2015).

Original: However, it has been less explored in  information retrieval domain.
Revised: However, it has been less explored in the information retrieval domain.

Original: Liu et al. (2015) proposed a multi-task deep neural approach to combine query classiﬁcation and document ranking, and reported improvement on both the tasks.
Revised: Liu et al. (2015) proposed a multi-task deep neural approach to combine query classiﬁcation and document ranking, and reported improvement on both the tasks.

Original: Bai et al.
Revised: Bai et al.

Original: (2009) used multi-task learning in learning to rank for web search.
Revised: (2009) used multi-task learning in learning to rank for web search.

Original: To better capture latent intent embedded in users’ search behaviors, we propose to jointly learn document ranking and query suggestion via multi-task learning.
Revised: We propose to jointly learn document ranking and query suggestion via multi-task learning to better capture latent intent embedded in users’ search behaviors.

Original:  M ULTI - TASK N EURAL S ESSION R ELEVANCE F RAMEWORK
Revised:  M ULTI - TASK N EURAL S ESSION R ELEVANCE F RAMEWORK

Original: We propose a multi-task neural session relevance framework (M-NSRF), speciﬁcally designed to jointly learn for document ranking and query suggestion.
Revised: We propose a multi-task neural session relevance framework (M-NSRF), speciﬁcally designed to jointly learn for document ranking and query suggestion in web search.

Revised: Document ranking refers to the task of ordering the retrieved results with respect to their relevance to the given query under the users’ search intent.

Revised: Accurate modeling of relevance between a document and a query is the key in this task.

Original: We consider a session as a sequence of  queries, Q = { Q 1 , ..., Q n } , submitted by a user in chronological order to satisfy a speciﬁc search intent. 
Revised: On the other hand, query suggestion refers to the task of generating related/alternative queries to users, so as to help them explore the search space. The key challenge is to maintain the semantic consistency between the suggested queries and users’ original queries, with respect to their search intent. Traditional IR approaches consider document ranking and query suggestion tasks independently (Huang et al., 2013; Shen et al., 2014;

Original: As the  users’ true intention is unobservable in the query log, we use 30-minutes inactive time threshold to segment the sequence into sessions, by assuming queries in the same time-based session share the same information need.
Revised: Sordoni et al., 2015), although both the tasks are driven by users’ underlying search intent. In the following, we discuss our joint learning model in details. We consider search context embedding in sessions, which have been proved to be a useful concept in modeling users’ search intent (Cao et al., 2008; Mitra, 2015; Li et al., 2014). Session is typically deﬁned as a sequence of queries, Q = { Q 1 , ..., Q n } , submitted by a user in a short time interval in a chronological order to satisfy a speciﬁc search intent.

Original: Every query Q i in a session is associated with a set of related documents D
Revised: Every query Q i in a session is associated with a set of related documents D

Original: = { D 1 , ..., D m } which need to be ranked by its relevance to the query and o = { o 1 , ..., o m } is the set of binary click labels for each document.
Revised: = { D 1 , ..., D m } which need to be ranked by its relevance to the query, and o = { o 1 , ..., o m } is the set of relevance labels for each document in D .

Original: A query Q i and a  document 
Revised: In typical search engine logs, o i is usually approximated via user clicks, e.g., clicked documents’ o i = 1 , otherwise 0. In this work, we model ranking of related documents as a pointwise classiﬁcation problem where the goal is to predict the relevance label based on the matching between the query and the document representations. But our developed solution can be easily extended to pairwise or list-wise ranking models (Liu et al., 2009).

Original:  D j is a sequence of words Q i = { w 1 i , ..., w qi } , D j = { w 1 j , ..., w dj }  where q = | Q i |is the query length and d = | D j | is the  document length.
Revised: A query Q i and a document D j consist of a sequence of words, i.e., Q i = { w 1 i , ..., w qi } and D j = { w 1 j , ..., w dj } , where q = | Q i | and d = | D j | are the query and document lengths respectively.

Original: V is the size of vocabulary constructed over queries and relevant documents.
Revised: V is the size of vocabulary constructed over queries and relevant documents.

Original: A detailed architecture of the multi-task neural session relevance framework (M-NSRF) is provided inappendix (see Appendix A).
Revised: The detailed architecture of our proposed multi-task neural session relevance framework (M-NSRF) is provided in Appendix A.

Original: M-NSRF is composed of two major components, document ranker and  query recommender.
Revised: M-NSRF is composed of two major components – a document ranker and a query recommender, which are connected via the shared session recurrent states.

Original: Given all the previous information in the same session and the current query submitted by a user with a set of candidate documents, the document ranker is trained to predict user clicks in the candidate document list. At the same time, the query recommender is trained in a sequence to sequence fashion (Sutskever et al., 2014) to predict the user’s next query.
Revised: Based on the latent session state inferred from all the previous queries and clicks in the same session, the document ranker is trained to predict user clicks from the candidate documents for the current query and the query recommender is trained in a sequence to sequence fashion (Sutskever et al., 2014) to predict the user’s next query.

Original: The process is repeated  for all the queries in a session.
Revised: The process is repeated sequentially for all the queries in the same session.

Original: In the following, we discuss each component of the  neural session relevance model in details.
Revised: In the following, we discuss each component of the multi-task neural session relevance framework in details.

Original: 3. D OCUMENT R ANKER
Revised: 3. D OCUMENT R ANKER

Revised: Ranking the retrieved documents for an input query requires encoding the query and documents into a shared representation space.

Revised: In addition to the search intent carried by the current query, search context, which is reﬂected in the previously submitted queries in the same session, should also be accounted in ranking the documents.

Revised: In M-NSRF, we model the latent user search intent in a sequence of queries via a series of session recurrent states.

Original: The document ranker  in M-NSRF is made up of a query encoder, a document encoder, a session encoder and a ranker.
Revised: As a result, the document ranker component in M-NSRF consists of a query encoder, a document encoder, a session encoder and a ranker.

Revised: The ranker sub-component combines the latent representations of query and session and match them with document representations to generate the ranking score of documents, based on which the documents are ordered.

Original: The technical details of each constituent element are given as follows.
Revised: The technical details of each constituent element are given as follows.

Original: Query Encoder .
Revised: Query Encoder .

Revised: The query encoder encodes a query into a vector.

Revised: To encode a sequence of words, various approaches have been studied.

Original: Following Conneau et al. (2017), we use bidirectional LSTMnetwork with max pooling technique to form query representation.
Revised: We follow (Conneau et al., 2017) to adopt a bidirectional LSTM with max pooling (BiLSTM-max), due to its superior practical performance.

Original: Considering query as a sequence of words Q i = { w 1{+
+} i, ..., w q i } , the encoder composed of forward and backward LSTM read the sequence in two opposite directions.
Revised: Considering query as a sequence of words Q i = { w {+
+} i ..., w qi } , the encoder composed of forward and backward LSTM reads the sequence in two opposite directions,

Original: [Equation]
Revised: [Equation]

Original: where h t ∈ R 2 d is the query-level recurrent state, d is the dimensionality of the LSTM hidden unitwhich is initialized to  zero vector.
Revised: where h t ∈ R 2 d is the query-level recurrent state, d is the dimensionality of the LSTM hidden unit initialized to a zero vector.

Original: To form a ﬁxed-size vector representation of variable length query, maximum value is selected over each dimension of the hidden units,
Revised: To form a ﬁxed-size vector representation of variable length queries, maximum value is selected over each dimension of the hidden units,

Original: [Equation]
Revised: [Equation]

Original: where Q i,k is the k -th element of the latent vector Q i .
Revised: where Q i,k is the k -th element of the latent vector Q i .

Original: Document Encoder .
Revised: Document Encoder .

Original: Document encoder isidentical to query encoder.
Revised: The goal of the document encoder is to encode documents into continuous vectors. We use the same BiLSTM-max technique that is utilized in encoding queries as the document encoder.

Original: The only difference is  the dimensionality of the LSTM hidden units.
Revised: The only difference is in the dimensionality of the LSTM hidden units.

Original: In general, documents (body or title) are longer than  query,so we use dense vector of  larger size to represent documents.
Revised: In general, because a document (body or title) is longer than a query, we consider dense vector of a larger size as the continuous representations of the documents.

Original: Session Encoder .
Revised: Session Encoder .

Original: Unlike the  query and document encoder, we use a vanilla LSTM  for session encoding.
Revised: The session encoder generates a representation to encode the queries that the system has received so far. Unlike query and document encoders that can read the complete encoded queries and documents, the session encoder does not have information from the future queries. Therefore, we use a unidirectional LSTM (Hochreiter & Schmidhuber, 1997) for session encoding.

Original: The session encoder takes the sequence of query representations Q 1 , ..., Q n as input and computes the sequence of session-level recurrent states.
Revised: The session encoder takes the sequence of query representations Q 1 , ..., Q n as input and computes the sequence of session-level recurrent states.

Original: [Equation]
Revised: [Equation]

Original: where S t ∈ R
Revised: where S t ∈ R

Original: d is the session-level recurrent state, d is the dimensionality of the LSTM hidden unit which is initialized to  zero vector.
Revised: d is the session-level recurrent state initialized to a zero vector.

Original: The number of session-level recurrent states S t is n , the number of queries in the session. The session-level recurrent state S t summarizes the queries that have been processed up to position t .
Revised: As a result, each query in the session has its session-level recurrent state, summarizing the user’s information need that has been processed up to query t .

Original: Ranker .
Revised: Ranker .

Original: To rank a set of relevant documents, D = { D 1 , ..., D M } , we concatenate the current query representation Q t with previous session-level recurrent state S t − 1 and apply a non-linear transformation. 
Revised: We ﬁrst concatenate the current query representation Q t with previous session-level recurrent state S t − 1 via a non-linear transformation. This combined representation reﬂects the search intent reﬂected in the current query and the past ones in the same session.

Original: Finally, to compute relevance score (click probability) between a query and adocument, we use sigmoid function.
Revised: Then we compute the ranking score of document D i under the query as its probability of being relevant via a sigmoid function (with binary relevance labels), [Equation]

Original: [Equation]
Revised: [Equation]

Original: where W r ∈ R ( d q + d s ) × d d and b r ∈ R d d where d q , d s and d d are the dimensionality of the query encoder, session encoder and document encoder hidden units and σ is  sigmoid function.
Revised: where W r ∈ R ( d q + d s ) × d d , b r ∈ R d d , and d q , d s and d d are the dimensionality of the query encoder, session encoder and document encoder hidden units and σ is the sigmoid function.

Revised: A list of retrieved documents can therefore be ordered by this ranking score.

Original: 3. Q UERY 
Revised: 3. Q UERY R ECOMMENDER

Original: [Equation]

Original: ECOMMENDER
Revised: Query recommender suggests related queries to users by inferring the underlying search intent through utilizing in-session previous queries embedded in latent vectors.

Original: Following Sutskever et al.
Revised: Following Sutskever et al.

Original:  and Bahdanau et al.
Revised: (2014) and Bahdanau et al.

Original: (2014), query recommender in M-NSRM predicts users’ next query using a sequence to sequence approach.
Revised: , the query recommender in M-NSRF predicts users’ next query in a sequence to sequence manner.

Original: Basically the query recommender module estimates the probability of the next query Q n = { w 1 , ..., w q } , given all the previous queries, Q 1: n − 1 up to position n − 1 in a session as follows,
Revised: Basically the query recommender module estimates the probability of the next query Q t = { w 1 , ..., w q } , given all the previous queries up to position t − 1 in a session as follows,

Original: [Equation]
Revised: [Equation]

Original: We use LSTM as a fundamental building block for the query recommender.
Revised: We use LSTM as a basic building block for the query recommender.

Original: Information about all the previous queries represented through a session vector S t are passed to the query recommender.
Revised: Information about all the previous queries represented through a session vector S t is passed to the query recommender.

Original: To this end, the recurrent state of the query recommender is initialized with a non-linear transformation of S t , h 0 =
Revised: To this end, the recurrent state of the query recommender is initialized with a non-linear transformation of S t , h 0 =

Original: tanh( W q S t + b q ) ,
Revised: tanh( W q S t + b q ) ,

Original: where h 0 ∈ R
Revised: where h 0 ∈ R

Original: d is the initial recurrent state, d is the dimensionality of the LSTM hidden unit.
Revised: d is the initial recurrent state.

Original: Then the query recommender’s recurrence is computed by, h t =
Revised: Then the query recommender’s recurrence is computed by h t =

Original: LSTM t ( h t − 1 , w t − 1 ) , where h t − 1 is the previous hidden state, w t − 1 is the previous query term.
Revised: LSTM t ( h t − 1 , w t − 1 ) , where h t − 1 is the previous hidden state, w t − 1 is the previous query term.

Original: Finally, each recurrent state is mapped to a probability distribution over the vocabulary, V using a combination of linear transformation and softmax function.
Revised: Finally, each recurrent state is mapped to a probability distribution over the vocabulary of size V using a combination of linear transformation and softmax function.

Original: Word with the highest probability is chosen as the next word in sequence.
Revised: Word with the highest probability is chosen as the next word in sequence.

Original: [Equation]
Revised: [Equation]

Original: where w t = arg max w P ( w | w 1: t − 1 , Q 1: n − 1 ) and g ( z ) j = e zj (cid:80) Kk =1 e zk , j = 1 , ..., K .
Revised: where w t = arg max w P ( w | w 1: t − 1 , Q 1: n − 1 ) and g ( z ) j = [Equation] = [Equation]

Revised: , j = 1 , ..., K .

Original: This formulation has shown to be beneﬁcial for language modelling tasks (Mikolov et al., 2010).
Revised: This formulation has shown to be beneﬁcial for language modelling tasks (Mikolov et al., 2010).

Original: Query Suggestion .
Revised: Query Suggestion .

Original: We consider the query suggestion task as an inference problem (Sordoni et al., 2015).
Revised: We consider the query suggestion task as a sequence inference problem (Sordoni et al., 2015).

Original: Given a sequence of queries up to position n − 1 , a suggested query Q n is:
Revised: Given a sequence of queries up to position t − 1 , a suggested query Q t is:

Original: [Equation]
Revised: [Equation]

Original: where Q is the space of all possible queries.
Revised: where Q is the space of all possible queries.

Original: To generate query suggestions of variable lengths, we use standard word-level decoding techniques (Cho et al., 2014).
Revised: To generate query suggestions of variable lengths, we use standard word-level decoding techniques (Cho et al., 2014).

Original: We iteratively consider the best preﬁx w 1: m up to length m and extend it by sampling the most probable word given the distribution in Eq.
Revised: We iteratively consider the best preﬁx w 1: m up to length m and extend it by sampling the most probable word given the distribution in Eq.

Original: The process ends when we obtain a well-formed query containing the special end-ofquery token.
Revised: The process ends when we obtain a well-formed query containing the special end-ofquery token.

Original: 3. L EARNING E ND - TO -E ND
Revised: 3. L EARNING E ND - TO -E ND

Original: Neural session relevance framework ranks documents  and predicts next query given a query and a set of candidate documents.
Revised: Within a session, M-NSRF ranks documents in a set of candidates and predicts next query given the current query sequence.

Original: Therefore, the training objective of M-NSRF consists of two terms.
Revised: Therefore, the training objective of M-NSRF consists of two terms.

Original: The ﬁrst term is the binary cross entropy loss for the document ranker
Revised: The ﬁrst term is the binary cross entropy loss from the document ranker,

Original: [Equation]
Revised: [Equation]

Original: where t i and o i represents binary click label and click probability for the i -th document ( P ( D i | Q t ) deﬁned in Eq.
Revised: where t i and o i represents binary click label and click probability for the i -th document, i.e., P ( D i | Q t ) deﬁned in Eq.

Original: The second term is the regularized negative log-likelihood loss for the query suggestion model
Revised: The second term is the regularized negative log-likelihood loss from the query suggestion model,

Original: [Equation]
Revised: [Equation]

Original:  where current query Q n = { w  1 , ..., wq } is of size | q |  , Q 1: n − 1 is all the previous queries in the session S , and [Equation] is the regularization term to avoid the distribution of words in Eq.
Revised: (3) where L R = − λ (cid:80) w ∈ V P ( w | w 1: i − 1 , Q 1: t − 1 ) log P ( w | w 1: i − 1 , Q 1: t − 1 ) is the regularization term to avoid the distribution of words in Eq.

Original: (2) from  highly peaky. λ is a hyper-parameter.
Revised: (2) from being highly skewed, and λ is a hyper-parameter to control the regularization term.

Original: For the sake of simplicity, the ﬁnal objective is the summation of L 1 and L 2 .
Revised: The ﬁnal objective is the summation of L 1 and L 2 .

Original: Note that the document ranker and query recommenders share the same document, query, and session encoders, and the training of M-NSRF can be done in an online manner using the following procedure.
Revised: Note that the document ranker and query recommender share the same document, query, and session encoders, and the training of M-NSRF can be done in an online manner using the following procedure.

Original: In the forward pass, M-NSRF computes the query and corresponding document encodings, updates session-level recurrent states, click probability for each candidate document and the log-likelihood of each query in the session given the previous ones.
Revised: In the forward pass, M-NSRF computes the query and corresponding document encodings, updates session-level recurrent states, click probability for each candidate document and the loglikelihood of each query in the session given the previous ones.

Original: In the backward pass, the gradients are computed and the parameters are updated based on ADAM update rule (Kingma & Ba, 2014).
Revised: In the backward pass, the gradients are computed and the parameters are updated based on ADAM update rule (Kingma & Ba, 2014).

Original: Details of implementation can be found in Section 4.
Revised: Details of implementation can be found in Section 4.

Original: 3. MULTI - TASK L EARNING F RAMEWORK FOR N EURAL IR M ODELS
Revised: 3. G ENERALIZING M-NSRF FOR N EURAL IR M ODELS

Original: The proposed multi-task learning framework is general and can be applied to combine different types of document rankers and query recommenders.
Revised: The proposed multi-task learning framework is general and can be applied to combine different types of document rankers and query recommenders.

Original: As mentioned previously, our proposed framework isvery close to thequery suggestion model proposed in (Sordoni et al., 2015) with a multi-task learning framework embedded in.
Revised: For example, as mentioned previously, our query suggestion component is close to the model proposed in (Sordoni et al., 2015) with a multitask learning framework embedded in.

Original: So, a query suggestion model working in  sequence to sequence fashion can be readily extended to our proposed multi-task learning architecture.
Revised: Hence, any neural query suggestion model working in the sequence to sequence fashion can be readily extended to our proposed multi-task learning architecture.

Original: On the other hand, most of the neural IR models are built on the notion of comparing query and document representation in  latent space to compute the matching degree.
Revised: On the other hand, most of the neural IR models are built on the notion of learning query and document representations in a latent space for relevance modeling.

Original: Extending such models to our proposed multi-task learning framework is straightforward but due to distinctive nature of different neural IR models, we need to decide how to incorporate context-awareness in the ﬁnal architecture.
Revised: Incorporating such models into our proposed multi-task learning framework is straightforward; but due to distinctive nature of different neural IR models, careful study is required while adding context-awareness into the ﬁnal architecture.

Original: In the experiment, we take the Match Tensor model (Jaech et al., 2017), a recently proposed neural relevance model  as an example and extend it to a multi-task-Match Tensor (M-Match Tensor) model by incorporating the query recommender module to follow our proposed multi-task learning framework. 
Revised: In this paper, we take the Match-Tensor model (Jaech et al., 2017), a recently proposed neural relevance model for document ranking as an example and extend it to a multi-task Match-Tensor (M-Match-Tensor) model by incorporating the query recommender component within our proposed multi-task learning framework. The document ranker in Match-Tensor works as follows, [Equation]

Revised: where C represents a sequence of convolutional operation (Lawrence et al., 1997) and max-pooling on the product of query and document vectors.

Revised: The detailed architecture of the M-Match-Tensor model (M-Match-Tensor) is provided in Appendix B.

Revised: To keep M-Match-Tensor model simple, we do not involve session recurrence in the document ranker component.

Original: We take the query and document recurrence state produced by bi-LSTM following a linear projection in original Match Tensor model and feed them to the session-level encoder.
Revised: We bring the query-level recurrent states produced by bi-LSTM following a linear projection into original Match-Tensor model and feed them to the session-level encoder.

Original: In this way, we follow the same relevance computation process as in  original Match Tensor model but jointly train M-Match Tensor model on document ranking and query suggestion.
Revised: In this way, we follow the same relevance computation process as in the original Match-Tensor model but jointly train M-Match-Tensor model on document ranking and query suggestion.

Original:  E XPERIMENTS 4. D ATA S ETS AND E VALUATION M ETHODOLOGY
Revised:  E XPERIMENTS 4. D ATA S ETS AND E VALUATION M ETHODOLOGY

Original: We conduct our experiments on thewell-known publicly available AOL search log (Pass et al., 2006).
Revised: We conduct our experiments on the publicly available AOL search log (Pass et al., 2006).

Original: The queries in this dataset were sampled between 1 March, 2006 and 31 May, 2006.
Revised: The queries in this dataset were sampled between 1 March, 2006 and 31 May, 2006.

Original: In total there are 16,946,938 queries submitted by 657,426 unique users.
Revised: In total there are 16,946,938 queries submitted by 657,426 unique users.

Original: We remove all nonalphanumeric characters from the queries, apply word segmentation and lowercasing.
Revised: We removed all non-alphanumeric characters from the queries, applied word segmentation and lowercasing.

Original: We follow (Jansen Bernard et al., 2007) to deﬁnethe end of a session by a 30 minute window of idle time. We ﬁltered sessions based on session length (minimum 2, maximum 10).
Revised: We followed (Jansen Bernard et al., 2007) to deﬁne a session by a 30-minute window of inactive time, and ﬁltered sessions by their lengths (minimum 2, maximum 10).

Original: We only keep the most frequent | V | = 100 k words and map all other words to an < unk > token while constructing the vocabulary.
Revised: We only kept the most frequent | V | = 100 k words and mapped all other words to an < unk > token when constructing the vocabulary.

Original: In total, we get 1,032,459 sessions for training, 129,053 sessions for development and 91,108 sessions for testing.
Revised: We randomly selected 1,032,459 sessions for training, 129,053 sessions for development and 91,108 sessions for testing, with no overlapping.

Original: In total train, development and test sessions consists of 2,987,486; 287,138  and 259, queries respectively.
Revised: In total, there are 2,987,486 queries for training, 287,138 for development, and 259,117 for testing.

Original: We consider two tasks in multi-task learning framework – document ranking and query suggestion. In the document ranking task, our goal is to rank candidate documents’ titles based on their relevance with the query.
Revised: The average length of the queries and documents (only the title ﬁeld) are 3. and 6.77 respectively.

Original: We model the document ranking task as a binary classiﬁcation task. Essentially, given a query and document, we predict if the  document will be clicked by a user.
Revised: In our experiments, we set maximum allowable length of query and document to 10 and 20 respectively. In the document ranking task, we need to rank the most relevant (e.g., most clickable) document on top.

Original: The ﬁnal predictions are evaluated using two ranking metrics, mean average precision (MAP) and normalized discounted cumulative gain (NDCG) metric computed at positions one, three and ten.
Revised: We used three standard ranking metrics, mean average precision (MAP), mean reciprocal rank (MRR) and normalized discounted cumulative gain (NDCG) metric computed at positions one, three, ﬁve and ten, to measure the performance.

Original: Since, AOL search log only contains clicks (positive examples), we selected negative examples through random sampling from the top 100 documentsretrieved by BM25.
Revised: Since AOL search log only contains clicked documents, we constructed the ranking candidates by the top ranked documents by BM25 (Robertson et al., 2009).

Original: Each query in the test set consists ofcandidates including the clicked documents.
Revised: Each query in the test set consists of 50 candidate documents including the clicked ones.

Original: However, to reduce the training time and memory use, each query in training and development set only contains 5 candidates.
Revised: However, to reduce the training time and memory use, each query in training and development set only contains 5 candidates.

Original: In the query suggestion task, we aim at predicting the next query in the same session.
Revised: In the query suggestion task, we need to suggest the most semantically related query to users.

Original: For evaluation, we computed the BLEU scores (Papineni et al., 2002).
Revised: As we do not have user feedback on the query suggestion part, we treated the users’ next submitted query as ground-truth (Sordoni et al., 2015), and used the BLEU scores (Papineni et al., 2002) as the evaluation metric.

Original: 4. B ASELINES AND I MPLEMENTATION D ETAILS
Revised: 4. B ASELINES AND I MPLEMENTATION D ETAILS

Original: Document Ranking Baselines .
Revised: Document Ranking Baselines .

Original: We compare M-NSRF and M-Match Tensor with word-based baselines and neural network-based baselines.
Revised: We compared M-NSRF and M-Match Tensor with word-based baselines and neural network-based baselines.

Original: Word-based baselines include query likelihood model based on Dirichlet smoothing (QL)
Revised: Word-based baselines include query likelihood model based on Dirichlet smoothing (QL)

Original: (Ponte & Croft, 1998) and BM25 (Robertson et al., 2009).
Revised: (Ponte & Croft, 1998) and BM25 (Robertson et al., 2009).

Original: In addition, following (Mitra et al., 2016), we investigated the ranking performance of a simple word embedding-based model using GloVe word embeddings (Pennington et al., 2014).
Revised: In addition, following (Mitra et al., 2016), we investigated the ranking performance of a simple word embedding-based model using GloVe word embeddings (Pennington et al., 2014).

Original: To compare with neural models, we consider baselines broadly categorized in representation-focused, interactionfocused and a combination of both.
Revised: To compare with neural models, we consider baselines broadly categorized in representation-focused, interactionfocused and a combination of both.

Original: Representation-focused neural baselines include: DSSM (Huang et al., 2013), CLSM (Shen et al., 2014), ARC-I (Hu et al., 2014) and interaction-focused baselines include: ARC-II (Hu et al., 2014) and DRMM (Guo et al., 2016a).
Revised: Representation-focused neural baselines include: DSSM (Huang et al., 2013), CLSM (Shen et al., 2014), ARC-I (Hu et al., 2014) and interaction-focused baselines [Table] (this [Table]

Original: A combination of representation and interaction focused models include: DUET (Mitra et al., 2017) and Match Tensor (Jaech et al., 2017).
Revised: (this paper) [Table] (Hu et al. [Table]

Revised: [Table]

Revised: (Guo et al., 2016a).

Revised: A combination of representation and interaction focused models include: DUET (Mitra et al., 2017) and Match Tensor (Jaech et al., 2017).

Original: Details of these models are provided in appendix (see B). We implemented all the baseline models in PyTorch.
Revised: Details of these models are provided in Appendix C. We implemented all the baseline models in PyTorch.

Original: Query Suggestion Baselines .
Revised: Query Suggestion Baselines .

Original: To evaluate  performance on query suggestion task, we consider three baseline methods including Seq2seq model proposed by Bahdanau et al.
Revised: To evaluate the performance on query suggestion task, we consider three baseline methods including Seq2seq model proposed by Bahdanau et al.

Original: (2014), Seq2seq with global attention mechanism (Luong et al., 2015) and HRED-qs
Revised: (2015), Seq2seq with global attention mechanism (Luong et al., 2015) and HRED-qs

Original: (Sordoni et al., 2015).
Revised: (Sordoni et al., 2015).

Original: Details of these models are provided in appendix (see C).
Revised: Details of these models are provided in appendix (see D).

Original: We implemented all three baselines in PyTorch and optimized using negative log-likelihood loss as in Eq.
Revised: We implemented all three baselines in PyTorch and optimized using negative log-likelihood loss as in Eq.

Original: Implementation Details of M-NSRF .
Revised: Implementation Details of M-NSRF .

Original: Themulti-task neural session relevance model was trained end-to-end and we used mini-batch SGD with Adam (Kingma & Ba, 2014) for optimization with the two momentum parameters set to 0.9 and 0.999 respectively.
Revised: The model was trained end-to-end and we used mini-batch SGD with Adam (Kingma & Ba, 2014) for optimization with the two momentum parameters set to 0.9 and 0.999 respectively.

Original: We use 300-dimentional word vectors trained with GloVe (Pennington et al., 2014) on 840 billion of tokens to initialize the word embeddings.
Revised: We use 300-dimensional word vectors trained with GloVe (Pennington et al., 2014) on 840 billion of tokens to initialize the word embeddings.

Original: Out-of-vocabulary words were randomly initialized by sampling values from a zeromean unit-variance normal distribution.
Revised: Out-of-vocabulary words were randomly initialized by sampling values from a zero-mean unit-variance normal distribution.

Original: All training used a mini-batch size of 32 to ﬁt in single GPU memory.
Revised: All training used a mini-batch size of 32 to ﬁt in single GPU memory.

Original: Learning rate was ﬁxed to 0.001.
Revised: Learning rate was ﬁxed to 0.001.

Original: We used dropout (0.10) (Srivastava et al., 2014) and early stopping with a patience of 5 epochs were used for regularization.
Revised: We used dropout (0.10) (Srivastava et al., 2014) and early stopping with a patience ofepochs for regularization.

Original: M-NSRF is implemented in PyTorch and it runs on a single GPU (TITAN X) with roughly a runtime of 90 minutes per epoch.
Revised: M-NSRF is implemented in PyTorch and it runs on a single GPU (TITAN X) with roughly a runtime of 90 minutes per epoch.

Original: In general, M-NSRF runs up to 20 epochs and we select the model that achieves the minimum loss on the development set.
Revised: In general, M-NSRF runs up to 20 epochs and we select the model that achieves the minimum loss on the development set.

Original: 4. E VALUATION R ESULTS
Revised: 4. E VALUATION R ESULTS

Original: Document Ranking Accuracy .
Revised: Document Ranking Quality .

Original: Table 1 shows the performance of NSRF, M-NSRF, M-Match Tensor and other baseline models.
Revised: Table 1 shows the performance of NSRF, M-NSRF, M-Match-Tensor and other baseline models.

Original: NSRF signiﬁcantly outperforms all the baselines except the Match Tensor model. However, the model size of NSRF is much smaller than Match Tensor.
Revised: NSRF signiﬁcantly outperforms all the baselines except the MatchTensor model, and the model size of NSRF is much smaller than Match-Tensor.

Original: With Multitask learning, both M-NSRF and M-Match Tensor outperform NSRF and Match Tensor, respectively.To avoid complexity in M-Match Tensor model, we did not consider session recurrence in [Table] (this paper)
Revised: With multi-task learning, both M-NSRF and M-Match-Tensor outperform NSRF and Match-Tensor, respectively.

Original: [Table]

Original: (this paper

Original: [Table] [Table] document ranking, rather investigated the utility added by the Multi-task learning principle. To study the advantage of Multi-task learning on our proposed approach, we trained M-NSRF only on document ranking task (noted as NSRF in Table 1) and observed signiﬁcant performance drop which endorses the beneﬁts of Multi-task learning.
Revised: To study the advantage of multi-task learning on our proposed approach, we trained M-NSRF only on document ranking task (noted as NSRF in Table 1) and observed signiﬁcant performance drop which endorses the beneﬁts of multi-task learning.

Original: Previous state-of-the-art techniques DRMM and DUET architecture performed suboptimal  in our experimental settings.
Revised: Previous techniques, like DRMM and DUET architecture, achieved suboptimal performance in our experiments.

Original: We believe because of  simple architecture of DRMM with few hundreds  parameters, the model fall behind to show competitive performance on the evaluation dataset.
Revised: We believe because of the simple architecture of DRMM with few hundreds of parameters, the model fell behind to show competitive performance on the evaluation dataset.

Original: On the other hand, we suspect that the use of smaller number of top character n -graphs (in our case, 5000) by DUET architecture limits its’ effectiveness in modeling representation and interaction focused features to compute matching degree between query and document.
Revised: On the other hand, we believe that the use of smaller number of top character n -graphs (in our case, 5000) by DUET architecture limits its effectiveness in modeling representation and interaction focused features to compute matching degree between query and document.

Original: In addition, use of document title while computing relevance between query and document in our experimental setting may limit the performance of DRMM and DUET architecture.

Original: It is important to note that, some negative results have been reported for document title-based ad hoc retrieval tasks (Guo et al., 2016a), so in our future work, we want to investigate M-NSRF and all baseline model’s performance by leveraging document body. 
Revised: We have to note that, some negative results have been reported for document title-based ad hoc retrieval tasks (Guo et al., 2016a); and thus in our future work, we plan to investigate M-NSRF and all baseline model’s performance by leveraging document body. [Table] [Table]

Original: Query Suggestion Accuracy Given the all the  previous queries  in a session, whether M-NSRF or baseline modelscan predict the user’s next query is evaluated and results are presented in table 2.
Revised: Query Suggestion Accuracy . Examples of the predicted queries by M-NSRF given previous queries from same session is presented in table 2 (more examples are provided in Appendix E) [Table] The quantitative comparison results in this task between our proposed framework and baseline models are presented in Table 3.

Original: While M-NSRF and HRED-qs considers previous query information from  same session, the other two baselines only consider the current query and predict  next one.
Revised: While M-NSRF and HRED-qs consider previous query information from the same session, the other two baselines only consider the current query to predict the next one.

Original: Table 2 shows that M-NSRF outperformed all baselines though the performance is still far from good.
Revised: Table 3 shows that M-NSRF outperformed seq2seq and HRED-qs baselines in all measured

Original: From our observation, we found that Seq2seq with global attention mechanism performed relatively well but the entropy regularization technique helped M-NSRF to achieve little improvement over the model’s performance.
Revised: BLUE scores, but slightly worse than Seq2Seq with global attention in BLEU-3 and BLEU-4.

Original: Though we found Multi-task learning helped M-NSRFto achieve better performance in document ranking, we could not see much gain on the query suggestion task.
Revised: We observed that entropy-based regularization helped M-NSRF in predicting the next query, as shown in Table 3 that M-NSRF without regularization achieved worse performance in all measured

Original: Adding attention over session information and personalization to improve query suggestions are two future directions, we are interested to work on. Examples of predicted queries by M-NSRF given previous queries from same session is presented in table 3 (more examples are provided in appendix, see D).
Revised: BLUE scores.

Original: [Table] [Table]
Revised: We further investigated the utility of regularization for the baselines.

Original: [-) 0.232 ( − 0.336) 0.296 ( − 0.348) GloVe 6B 100d 0.312 ( − 0.269) 0.241 ( − 0.282) 0.
 ( − 0.295) 0.306 ( − 0.308) GloVe 6B 200d 0.378 ( − 0.203) 0.356 ( − 0.167) 0.447-] ( − 0.121) 0.498 ( − 0.116)
Revised: [-) 0.232 ( − 0.336) 0.296 ( − 0.348) GloVe 6B 100d 0.312 ( − 0.269) 0.241 ( − 0.282) 0.
 ( − 0.295) 0.306 ( − 0.308) GloVe 6B 200d 0.378 ( − 0.203) 0.356 ( − 0.167) 0.447-]We found signiﬁcant ( ∼ 1.2%) improvement for the HRED-qs, but the performance improvement for the seq2seq and seq2seq with global attention mechanism is rather marginal ( ∼ 0.2%).

Original: [Table] [Table] [Table] [Table]
Revised: We hypothesize that the regularization is more useful when a user’s search history is employed in modeling users’ search intent that may potentially introduce bias in the prediction of next query.

Original: [Table] [Table] [Table] 544 ( −
Revised: Comparing to HREDqs, we conclude multi-task learning also helps M-NSRF achieve improved performance in query suggestion task, as the baseline employs a very similar model as ours for query suggestion.

Original: [Table] [Table] [Table] [Table]
Revised: The comparison with Seq2Seq with global attention suggests adding attention over session information is a promising direction to further improve query suggestions quality (and maybe also document ranking).

Original: [Table] [Table] [Table] [Table]
Revised: We will pursue this as our future work.

Original: 4. A BLATION S TUDY ON M-NSRF
Revised: 4. A BLATION S TUDY ON M-NSRF

Original: We conducted experiments to better understand the effectiveness of different components in the MNSRF.
Revised: We conducted experiments to better understand the effectiveness of different components in the MNSRF.

Original: We also analyzed the impact of word embeddings and hidden units dimension in M-NSRF’s performance.
Revised: We also analyzed the impact of word embeddings and hidden units dimension in M-NSRF’s performance.

Original: Our ﬁndings are presented in table 4.
Revised: Our ﬁndings are presented in Table 4.

Original: Impact of Different Model Components .
Revised: Impact of Different Model Components .

Original: To study the effect of different model components, we compare the full M-NSRF with several simpler versions of the model.
Revised: To study the effect of different model components, we compared the full M-NSRF with several simpler versions of the model.

Original: At ﬁrst, we turned off training for word embeddings and found a large drop in performance.
Revised: At ﬁrst, we turned off training for word embeddings and found a signiﬁcant decrease in performance.

Original: In our training dataset, we have roughly | OOV | = 26 k out-of-vocabulary words and so, training the word embeddings turned out to be very important for our experimental setting.
Revised: In our training dataset, we have roughly | OOV | = 26 k out-of-vocabulary words and thus, training the word embeddings turned out to be very important on this data set.

Original: Also, we investigated the role of pre-trained word embeddings (ex., GloVe embeddings) and found signiﬁcant performance drop (27.9% drop in MAP) if we train M-NSRF without any pre-trained word embeddings.
Revised: Also, we investigated the role of pre-trained word embeddings (ex., GloVe embeddings) and found signiﬁcant performance decrease (27.9% drop in MAP) if we train M-NSRF without any pre-trained word embeddings.

Original: Hence, we can conclude that  use of pre-trained word embeddings and training them further is important to achieve better performance in M-NSRF.
Revised: Hence, we can conclude that the use of pre-trained word embeddings and training them further is important to achieve better performance in M-NSRF.

Original: Secondly, we investigated the advantages of using max-pooling over mean-pooling and considering  last hidden recurrent state for query and document representation.
Revised: Second, we investigated the advantages of using max-pooling over mean-pooling and considering the last hidden recurrent state for query and document representa- [Table] ( − 0.116)

Revised: Q 128 D 256 S0.562 ( − 0.019) 0.507 ( − 0.016) 0.544 ( − 0.024) 0.582 ( − 0.032)

Revised: Q 512 D 1024 S0.586 ( + 0.005) 0.528 ( + 0.005) 0.571 ( + 0.003) 0.617 ( + 0.003) tion.

Original: We observed that max-pooling and mean-pooling provide almost  same performance while biLSTM-last approach lags slightly.
Revised: We observed that max-pooling and mean-pooling provide almost the same performance while biLSTM-last approach lags slightly.

Original: To further analyze the features identiﬁed by the query and document encoders using the max-pooling technique, we follow the idea of visualization proposed in (Conneau et al., 2017).
Revised: To further analyze the features identiﬁed by the query and document encoders using the max-pooling technique, we followed the idea of visualization proposed in (Conneau et al., 2017).

Original: We provide an example in ﬁgure 2 where document 1 is clicked by the user (a positive example) and document 2, 3 is retrieved by BM25 and considered as unclicked document (negative examples).
Revised: We provide an example in Figure 2 where document 1 is clicked by the user (a positive example) and document 2, 3 is retrieved by BM25 but not clicked (negative examples).

Original: We observed that the query and document encoders identify distinguishing features (ex., the word priceline in the ﬁrst document’s title is most important) which helpto differentiate between clicked and unclicked documents.
Revised: We observed that the query and document encoders identiﬁed distinct features (e.g., the word priceline in the ﬁrst document’s title is most important) which help differentiate between clicked and unclicked documents.

Original: In another variant of M-NSRF, we did not use  session recurrent state while computing relevance score for the candidate documents to examine the inﬂuence of previous information from  same session on the ranking performance.
Revised: In another variant of M-NSRF, we did not use the session recurrent state when computing relevance score for the candidate documents to examine the inﬂuence of previous queries from the same session on the ranking performance.

Original: We call this variant of M-NSRF as  neural relevance model (M-NRM).
Revised: We refer to this variant of M-NSRF as multi-task neural relevance model (M-NRF).

Original: From table 4 we can see, without session information the performance drops by 2.8% in terms of MAP.
Revised: From Table 4 we can ﬁnd that without session information the performance drops by 2.8% in terms of MAP.

Original: We further investigated and found  session information helps particularly for longer sessions (session length > 5 ).
Revised: We further investigated and found the session information helps particularly for longer sessions (session length > 5 ).

Original: In our evaluation dataset, we had roughly 5000 sessions of length greater than 5, so the performance difference is not reasonably high as we expected.
Revised: In our evaluation dataset, we have roughlysessions of length greater than 5 (91 thousands in total).

Original: We will investigate whether directly incorporating click information in the session recurrence helps in improving the performance signiﬁcantly.

Original: Impact of Dimensionality .
Revised: Impact of Dimensionality .

Original: We further study the impact of dimensionality of the word embeddings, query, document and session latent vectors.
Revised: We further study the impact of dimensionality of the word embeddings, query, document and session latent vectors.

Original: In M-NSRF, we set the dimension of query, document and session latent vectors as 256, 512 and 1024 respectively.
Revised: In M-NSRF, we set the dimension of query, document and session latent vectors to 256, 512 and 1024 respectively.

Original: As shown in table 4, decreasing the dimensions of latent vectors, drops the performance while increasing the dimensions further, does not affect the performance signiﬁcantly.
Revised: As shown in Table 4, decreasing the dimensions of latent vectors, decreases the performance; while increasing the dimensions further, does not affect the performance signiﬁcantly.

Original: We also experimented with different dimensions of pretrained word embeddings (50d, 100d and 200d GloVe (Pennington et al., 2014) embeddings).
Revised: We also experimented with different dimensions of pre-trained word embeddings (50d, 100d and 200d GloVe (Pennington et al., 2014) embeddings).

Original: Word embeddings of different dimensionality provide different granularity of semantic similarity; with lower dimensionality, the similarity between word embeddings might be coarse and thus hard to capture matching between two text sequences.
Revised: Word embeddings of different dimensions provide different granularity of semantic similarity; with lower dimensionality, the similarity between word embeddings might be coarse and thus hard to capture matching between two text sequences.

Original: In our experiment, we found 300 dimension for embeddings works signiﬁcantly better than other dimensionality.
Revised: In our experiment, we found 300 dimension for embeddings works signiﬁcantly better than other dimensionality on this data set.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Existing deep neural models for ad-hoc retrieval often omit session information and are only trained on  query-document pairs.
Revised: Existing deep neural models for ad-hoc retrieval often omit session information and are only trained on individual query-document pairs.

Original: In this work, we propose a context-aware multi-task neural session relevance framework which works in a sequence to sequence fashion and show that sharing session-level latent recurrent states across document ranking and query suggestion task beneﬁts each other.
Revised: In this work, we propose a context-aware multi-task neural session relevance framework which works in a sequence to sequence fashion and show that sharing session-level latent recurrent states across document ranking and query suggestion tasks beneﬁts each other.

Original: Our experiments and analysis not only demonstrated the effectiveness of the proposed framework, but also provides useful intuitions about the advantages of multi-task learning involving deep neural networks for IR tasks.
Revised: Our experiments and analysis not only demonstrate the effectiveness of the proposed framework, but also provide useful intuitions about the advantages of multi-task learning involving deep neural networks for two different information retrieval tasks.

Original: For future work, we would like to leverage  document body  to train M-NSRF  so that we can further explore the potential of the proposed framework for ad-hoc retrieval.
Revised: As our future work we would like to leverage the content from document body and click sequence to update M-NSRF (especially the session-level recurrent states) so that we can further explore the potential of the proposed framework for ad-hoc retrieval.

Revised: As attention mechanism shows promise in improving query suggestion performance, we also also explore it in our multi-task learning setting.

Original: In addition, a broad research direction would be to ﬁnd ways to summarize individual users’ search log to model  long-term search goals to enhance personalized search results and query suggestions 
Revised: In addition, a broad research direction is to go beyond session boundaries to model users’ long-term search goals to enhance personalized search results and query suggestions B M ULTITASK M ATCH -T ENSOR A RCHITECTURE

Original: B D OCUMENT R ANKING B ASELINES
Revised: C D OCUMENT R ANKING B ASELINES

Original: Deep semantic similarity model, DSSM (Huang et al., 2013) maps words to letter tri-grams using a word-hashing technique and uses a feed-forward neural network to build representations for both query and document.
Revised: Deep semantic similarity model, DSSM (Huang et al., 2013) maps words to letter tri-grams using a word-hashing technique and uses a feed-forward neural network to build representations for both query and document.

Original: Similarity, convolutional latent semantic model, CLSM (Shen et al., 2014) uses word-hashing technique and uses convolutional neural networks (CNN) to build query and document representations.
Revised: Similarity, convolutional latent semantic model, CLSM (Shen et al., 2014) uses word-hashing technique and uses convolutional neural networks (CNN) to build query and document representations.

Original: To compute relevance between query and document, both DSSM and
Revised: To compute relevance between query and document, both DSSM and

Original: CLSM uses cosine similarity.
Revised: CLSM uses cosine similarity.

Original: ARC-I (Hu et al., 2014) uses CNN to form query and document representations and employs a multi-layer perceptron to compute relevance score.
Revised: ARC-I (Hu et al., 2014) uses CNN to form query and document representations and employs a multi-layer perceptron to compute relevance score.

Original: In our implementation, we used 128 convolution ﬁlters of size 1, 2 and 256 ﬁlters of size 3.
Revised: In our implementation, we used 128 convolution ﬁlters of size 1, 2 and 256 ﬁlters of size 3.

Original: ARC-II (Hu et al., 2014) was proposed by focusing on learning hierarchical matching patterns from local interactions using a CNN.
Revised: ARC-II (Hu et al., 2014) was proposed by focusing on learning hierarchical matching patterns from local interactions using a CNN.

Original: To keep the ARC-II model simple, we use two layers of 2d convolution and max-pooling each and two-layer feed forward neural network to compute relevance score.
Revised: To keep the ARC-II model simple, we use two layers of 2d convolution and max-pooling each and two-layer feed forward neural network to compute relevance score.

Original: DRMM (Guo et al., 2016a) aims to perform term matching over histogram-based features ignoring the actual position of matches.
Revised: DRMM (Guo et al., 2016a) aims to perform term matching over histogram-based features ignoring the actual position of matches.

Original: In DRMM, histogram-based features are computed using exact term matching and pretrained word embeddings based cosine similarities.
Revised: In DRMM, histogram-based features are computed using exact term matching and pretrained word embeddings based cosine similarities.

Original: In principal, the histogram counts the number of word pairs at different similarity levels.
Revised: In principal, the histogram counts the number of word pairs at different similarity levels.

Original: The counts are combined by a feed forward network to produce ﬁnal ranking scores.
Revised: The counts are combined by a feed forward network to produce ﬁnal ranking scores.

Original: The DUET (Mitra et al., 2017) model composed of a local and distributed model where the distributed model projects the query and the document text into an embedding space before matching, while the local model operates over an interaction matrix comparing every query term to every document term.
Revised: The DUET (Mitra et al., 2017) model composed of a local and distributed model where the distributed model projects the query and the document text into an embedding space before matching, while the local model operates over an interaction matrix comparing every query term to every document term.

Original: Similarly, Match Tensor (Jaech et al., 2017) model incorporates both immediate and larger contexts in a given document when comparing document to a query.
Revised: Similarly, Match-Tensor (Jaech et al., 2017) model incorporates both immediate and larger contexts in a given document when comparing document to a query.

Original: C Q UERY S UGGESTION B ASELINES
Revised: D Q UERY S UGGESTION B ASELINES

Original: Seq2seq model proposed by Bahdanau et al.
Revised: Seq2seq model proposed by Bahdanau et al.

Original: (2014) is a general neural network architecture that can be applied to the task where both input and output consist of a sequence of tokens.
Revised: (2015) is a general neural network architecture that can be applied to the task where both input and output consist of a sequence of tokens.

Original: This method have been shown successful in machine translation and sequential tagging.
Revised: This method have been shown successful in machine translation and sequential tagging.

Original: Because different input tokens may contribute to each output token differently, attention mechanism which learns a weight between each input-output token pair can further improve the Seq2seq model.
Revised: Because different input tokens may contribute to each output token differently, attention mechanism which learns a weight between each input-output token pair can further improve the Seq2seq model.

Original: In this paper, we consider a seq2seq with global attention method proposed by Luong et al.
Revised: In this paper, we consider a seq2seq with global attention method proposed by Luong et al.

Original: (2015), which is suitable for short text such as web queries.
Revised: (2015), which is suitable for short text such as web queries.

Original: HRED-qs suggested by Sordoni et al.
Revised: HRED-qs suggested by Sordoni et al.

Original: (2015) is very close to our work which proposed to use a hierarhical recurrent encoder-decoder approach by considering session information for context-aware query suggestion.
Revised: (2015) is very close to our work which proposed to use a hierarhical recurrent encoder-decoder approach by considering session information for context-aware query suggestion.

Original: D M ORE E XAMPLES OF Q UERY S UGGESTION BY M-NSRF
Revised: E M ORE E XAMPLES OF Q UERY S UGGESTION BY M-NSRF

Original: Previous session queries discount pet supplies, homes for rent smyrna georgia
Revised: Previous session queries discount pet supplies, homes for rent smyrna georgia

Original: Next user query homes for rent atlanta georgia
Revised: Next user query homes for rent atlanta georgia

Original: Suggested next query pet friendly rentals in georgia
Revised: Suggested next query pet friendly rentals in georgia

Original: Previous session queries language aptitude test, foreign language aptitude test
Revised: Previous session queries language aptitude test, foreign language aptitude test

Original: Next user query american idol
Revised: Next user query american idol

Original: Suggested next query american language association
Revised: Suggested next query american language association

Original: Previous session queries saturday night fever, saturday night fever nj band
Revised: Previous session queries saturday night fever, saturday night fever nj band

Original: Next user query new jersey cover band
Revised: Next user query new jersey cover band

Original: Suggested next query saturday night live
Revised: Suggested next query saturday night live

Original: Previous session queries pregnancy, abortion, abortion clinics
Revised: Previous session queries pregnancy, abortion, abortion clinics

Original: Next user query tampa abortion
Revised: Next user query tampa abortion

Original: Suggested next query abortion clinics in ﬂorida
Revised: Suggested next query abortion clinics in ﬂorida

Original: Previous session queries ncaa basketball, ncaa basketball trees, ncaa mens basketball bracket, sportscenter Next user query mens ncaa basketball odds
Revised: Previous session queries ncaa basketball, ncaa basketball trees, ncaa mens basketball bracket, sportscenter Next user query mens ncaa basketball odds

Original: Suggested next query espn
Revised: Suggested next query espn

Original: Previous session queries childhood autism rating scale, childhood autism rating scale free, autism screening questionnaire Next user query pervasive developmental disorder
Revised: Previous session queries childhood autism rating scale, childhood autism rating scale free, autism screening questionnaire Next user query pervasive developmental disorder

Original: Suggested next query how to do questionnaire
Revised: Suggested next query how to do questionnaire