Original: M ULTITASK

Original: L EARNING

Original: FOR

Original:  D OCUMENT R ANKING AND Q UERY S UGGESTION
Revised: M ULTI -T ASK L EARNING FOR D OCUMENT R ANKING AND Q UERY S UGGESTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a multitask learning framework to jointly learn document ranking and query suggestion.
Revised: We propose a multi-task learning framework to jointly learn document ranking and query suggestion for web search.

Original: The proposed model leverages previous queries and clickthrough information from the same search session and improves performance on both tasks.
Revised: The proposed model leverages previous queries and click-through information from the same search session to improve performance on both tasks.

Original: Our framework consists of two major components, document ranker and query recommender.
Revised: Our framework consists of two major components, document ranker and query recommender.

Original: Document ranker combines current query and previous session information and compares the combined representation with the document representation to rank the documents.
Revised: Document ranker combines current query and previous session information and compares the combined representation with the document representation to rank the documents.

Original: Query recommender tracks users’ query reformulation sequence considering all previous in-session queries using a sequence to sequence approach.
Revised: Query recommender tracks users’ query reformulation sequence considering all previous in-session queries using a sequence to sequence approach.

Original: Both components are trained across search sessions by sharing parameters through session recurrence, which encodes session information.
Revised: Both components are trained across search sessions by sharing parameters through session recurrence, which encodes session information.

Original: Comprehensive experiments including rigorous comparisons with state-of-the-art techniques are performed on the public AOL search log, and the promising results endorse the effectiveness of the joint learning framework.
Revised: Comprehensive experiments including rigorous comparisons with state-of-the-art techniques are performed on the public AOL search log, and the promising results endorse the effectiveness of the joint learning framework.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Understanding users’ information need is the key to optimize a search engine for providing relevant search results.
Revised: Understanding users’ information need is the key to optimize a search engine for providing relevant search results.

Original: Search engine logs have become an important resource to mine users’ search intent and click behavior (Baeza-Yates et al., 2004; Croft et al., 2010).
Revised: Search engine logs have become an important resource to mine users’ search intent and click behavior (Baeza-Yates et al., 2004; Croft et al., 2010).

Original: In particular, user query logs are partitioned into search sessions, i.e. sequences of queries issued by a unique user and submitted within a short time interval.
Revised: In particular, user query logs are partitioned into search sessions, i.e. sequences of queries issued by a unique user and submitted within a short time interval.

Original: In a search session, a user submits a sequence of queries and clicks on ranked documents that he/she believes can serve his/her information need.
Revised: In a search session, a user submits a sequence of queries and clicks on ranked documents that he/she believes can serve his/her information need.

Original: Thus, leveraging the user search behaviors within a query session, a.k.a.
Revised: Thus, leveraging the user search behaviors within a query session, a.k.a.

Original: context-awareness provides useful information about user intent and helps to narrow down ambiguity while ranking documents for the current query and predicting next query that users will submit (Jiang et al., 2014).
Revised: context-awareness provides useful information about user intent and helps to narrow down ambiguity while ranking documents for the current query and predicting next query that users will submit (Jiang et al., 2014).

Original: Since, both user’s click behavior and query reformulation are driven by the underlying intent, we argue that by jointly modeling both tasks can beneﬁt each other.
Revised: Since, both user’s click behavior and query reformulation are driven by the underlying intent, we argue that by jointly modeling both tasks can beneﬁt each other.

Original: In this work, we propose a joint learning framework, called multitask neural session relevance framework (M-NSRF) to predict users’ future queries and clicks.
Revised: In this work, we propose a joint learning framework, called multitask neural session relevance framework (M-NSRF) to predict users’ future queries and clicks.

Original: We assume, a search session is composed of several queries and corresponding clicks that can supplement a user’s underlying intent for a search task and thus helps to improve document ranking and query suggestion.
Revised: We assume, a search session is composed of several queries and corresponding clicks that can supplement a user’s underlying intent for a search task and thus helps to improve document ranking and query suggestion.

Original: Inspired by recent works (Collobert & Weston, 2008; Liu et al., 2015), we consider a deep neural network approach that allows us to learn multiple tasks by sharing model parameters, latent states, and optimizing a joint objective function.
Revised: Inspired by recent works (Collobert & Weston, 2008; Liu et al., 2015), we consider a deep neural network approach that allows us to learn multiple tasks by sharing model parameters, latent states, and optimizing a joint objective function.

Original: The joint learning approach improves generalization on both tasks, and allows us to learn latent vector representations of user intent carrying over the whole session.
Revised: The joint learning approach improves generalization on both tasks, and allows us to learn latent vector representations of user intent carrying over the whole session.

Original: General work ﬂow of M-NSRF is illustrated in Figure 1.
Revised: General work ﬂow of M-NSRF is illustrated in Figure 1.

Original: Given a sequence of user queries from the same search session (collected from AOL search log), “how to terminate a pregnancy at home”, “abortions in orlando ﬂorida”, “pregnancy signs” , M-NSRF is trained to predict the next query, “non medical pregnancy termination” and the clicks on a set of relevant documents.
Revised: Given a sequence of user queries from the same search session (collected from AOL search log), “how to terminate a pregnancy at home”, “abortions in orlando ﬂorida”, “pregnancy signs” , M-NSRF is trained to predict the next query, “non medical pregnancy termination” and the clicks on a set of relevant documents.

Original: We argue that understanding the true user intent can be aided by both click and query prediction task.
Revised: We argue that understanding the true user intent can be aided by both click and query prediction task.

Original: In addition, modeling session-level latent states which carry information about previous queries and clicks help to better understand users’ search intents.
Revised: In addition, modeling session-level latent states which carry information about previous queries and clicks help to better understand users’ search intents.

Original: Our proposed framework can be adapted to any text representation and generation model and trained end-to-end on search sessions.
Revised: Our proposed framework can be adapted to any text representation and generation model and trained end-to-end on search sessions.

Original: We evaluate the effectiveness of the proposed framework using the publicly available AOL search log and compare with several well-known traditional retrieval models, as well as several state-ofthe-art deep neural models speciﬁcally designed for ad-hoc retrieval.
Revised: We evaluate the effectiveness of the proposed framework using the publicly available AOL search log and compare with several well-known traditional retrieval models, as well as several state-ofthe-art deep neural models speciﬁcally designed for ad-hoc retrieval.

Original: We also compare M- with several baseline models for query prediction task.
Revised: We also compare M- with several baseline models for query prediction task.

Original: empirical results
Revised: empirical results

Original: M-
Revised: M-

Original: The key contributions of this work include:
Revised: The key contributions of this work include:

Original: We propose a novel multitask neural session relevance model that is jointly trained on document ranking and query suggestion tasks by utilizing in-session historical query and click information.
Revised: We propose a novel multitask neural session relevance model that is jointly trained on document ranking and query suggestion tasks by utilizing in-session historical query and click information.

Original: We conduct rigorous comparisons over traditional and neural state-of-the-art retrieval models and query prediction techniques using AOL search log and demonstrate the joint framework can improve existing models, such as Match Tensor (Jaech et al., 2017).
Revised: We conduct rigorous comparisons over traditional and neural state-of-the-art retrieval models and query prediction techniques using AOL search log and demonstrate the joint framework can improve existing models, such as Match Tensor (Jaech et al., 2017).

Original: We provide detailed analysis and will release the code and data to facilitate future research.
Revised: We provide detailed analysis and will release the code and data to facilitate future research.

Original:  R ELATED W ORK AND B ACKGROUND
Revised:  R ELATED W ORK AND B ACKGROUND

Original: Ad-hoc Retrieval.
Revised: Ad-hoc Retrieval.

Original: Traditional retrieval models such as query likelihood (Ponte & Croft, 1998) and
Revised: Traditional retrieval models such as query likelihood (Ponte & Croft, 1998) and

Original: BM25 (Robertson et al., 2009) are based on exact matching of query and document words with a variety of smoothing, weighting and normalization techniques.
Revised: BM25 (Robertson et al., 2009) are based on exact matching of query and document words with a variety of smoothing, weighting and normalization techniques.

Original: Recently, deep neural network based approaches demonstrate strong performance in ad-hoc retrieval.
Revised: Recently, deep neural network based approaches demonstrate strong performance in ad-hoc retrieval.

Original: Existing neural ranking models fall into two categories: representation focused (Huang et al., 2013; Gao et al., 2014) and interaction focused (Guo et al., 2016b).
Revised: Existing neural ranking models fall into two categories: representation focused (Huang et al., 2013; Gao et al., 2014) and interaction focused (Guo et al., 2016b).

Original: The earlier focus of neural IR was mainly on representation based models (Hu et al., 2014;
Revised: The earlier focus of neural IR was mainly on representation based models (Hu et al., 2014;

Original: Shen et al., 2014), in which the query and documents are ﬁrst embedded into continuous vectors, and the ranking is calculated from their embeddings’ similarity.
Revised: Shen et al., 2014), in which the query and documents are ﬁrst embedded into continuous vectors, and the ranking is calculated from their embeddings’ similarity.

Original: The interaction focused neural models (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016a), on the other hand, learn query-document matching patterns from word-level interactions.
Revised: The interaction focused neural models (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016a), on the other hand, learn query-document matching patterns from word-level interactions.

Original: Both the interaction and representation focused models can be combined for further improvements (Mitra et al., 2017).
Revised: Both the interaction and representation focused models can be combined for further improvements (Mitra et al., 2017).

Original: Similarly, Jaech et al.
Revised: Similarly, Jaech et al.

Original: (2017) captures both local relevance matching and global topicality signals when computing relevance of a document to a query.
Revised: (2017) captures both local relevance matching and global topicality signals when computing relevance of a document to a query.

Original: In our work, we followed representation focused approach to form query and document representations and jointly model two tasks through session representations.
Revised: In our work, we followed representation focused approach to form query and document representations and jointly model two tasks through session representations.

Original: Query Suggestion.
Revised: Query Suggestion.

Original: In general, query suggestion algorithms use clustering methods to ﬁnd similar queries so that they can be used as suggestions for one another (Wen et al., 2001; Baeza-Yates et al., 2004).
Revised: In general, query suggestion algorithms use clustering methods to ﬁnd similar queries so that they can be used as suggestions for one another (Wen et al., 2001; Baeza-Yates et al., 2004).

Original: The closest to our work is the end-to-end hierarchical recurrent encoder-decoder architecture (HRED-qs)
Revised: The closest to our work is the end-to-end hierarchical recurrent encoder-decoder architecture (HRED-qs)

Original: (Sordoni et al., 2015) to rank candidates for context-aware query suggestion.
Revised: (Sordoni et al., 2015) to rank candidates for context-aware query suggestion.

Original: Our proposed framework mainly differs from HRED-qs as it exploits user clicks and includes it in the contextual information.
Revised: Our proposed framework mainly differs from HRED-qs as it exploits user clicks and includes it in the contextual information.

Original: Similarly, Mitra & Craswell (2015) proposed a candidate generation approach for rare preﬁxes using frequently observed query sufﬁxes and suggested a neural model to generate ranking features along with n-gram features.
Revised: Similarly, Mitra & Craswell (2015) proposed a candidate generation approach for rare preﬁxes using frequently observed query sufﬁxes and suggested a neural model to generate ranking features along with n-gram features.

Original: To handle rare queries, different methods are proposed in previous works (Vahabi et al., 2013; Broccolo et al., 2012).

Original: Like other deep neural models, our proposed framework can handle rare queries as long as their words appear in the model vocabulary.

Original: Multitask Learning.
Revised: Multitask Learning.

Original: Multitask learning consider learning multiple related tasks jointly with the aim of mutual beneﬁt.
Revised: Multitask learning consider learning multiple related tasks jointly with the aim of mutual beneﬁt.

Original: The goal of multitask learning is to improve generalization on the original task by leveraging the domain-speciﬁc information contained in the training signals of related tasks (Caruana, 1998).
Revised: The goal of multitask learning is to improve generalization on the original task by leveraging the domain-speciﬁc information contained in the training signals of related tasks (Caruana, 1998).

Original: Multitask learning in combination with deep neural networks have been successfully used in many applications of machine learning, including natural language processing (Collobert &
Revised: Multitask learning in combination with deep neural networks have been successfully used in many applications of machine learning, including natural language processing (Collobert &

Original: Weston, 2008; Liu et al., 2016), speech recognition (Deng et al., 2013; Thanda & Venkatesan, 2017) and computer vision (Girshick, 2015).
Revised: Weston, 2008; Liu et al., 2016; Peng et al., 2017), speech recognition (Deng et al., 2013; Thanda & Venkatesan, 2017) and computer vision (Girshick, 2015).

Original: Despite of wide adoption of multitask learning with deep neural networks in related areas, it has been less explored in information retrieval domain.
Revised: Despite of wide adoption of multitask learning with deep neural networks in related areas, it has been less explored in information retrieval domain.

Original: Liu et al.
Revised: Liu et al.

Original: (2015) proposed a multitask deep neural approach to combine query classiﬁcation and document ranking and found improvements on both the tasks.
Revised: (2015) proposed a multitask deep neural approach to combine query classiﬁcation and document ranking and found improvements on both the tasks.

Original: Also, multitask learning is used in learning to rank in web search (Bai et al., 2009).
Revised: Also, multitask learning is used in learning to rank in web search (Bai et al., 2009).

Original: To further investigate the utility of multitask learning, we proposea deep neural approach to jointly learn document ranking and query suggestion.
Revised: To further investigate the utility of multitask learning, we propose to jointly learn document ranking and query suggestion.

Original:  N EURAL S ESSION R ELEVANCE M ODEL
Revised:  N EURAL S ESSION R ELEVANCE M ODEL

Original: We propose a multitask neural session relevance framework (M-NSRF), speciﬁcally designed to jointly learn for document ranking and query suggestion tasks.
Revised: We propose a multitask neural session relevance framework (M-NSRF), speciﬁcally designed to jointly learn for document ranking and query suggestion tasks.

Original: We consider a session as a sequence of queries, Q = { Q 1 , ..., Q n } submitted by a user in chronological order to satisfy a query intent.
Revised: We consider a session as a sequence of queries, Q = { Q 1 , ..., Q n } submitted by a user in chronological order to satisfy a query intent.

Original: As we do not have the access to users’ true intention from the query log, we assume the queries submitted within 30 minutes time frame by a user are in the same session.
Revised: As we do not have the access to users’ true intention from the query log, we assume the queries submitted within 30 minutes time frame by a user are in the same session.

Original: Every query Q i in a session is associated with a set of relevant documents D
Revised: Every query Q i in a session is associated with a set of relevant documents D

Original: = { D 1 , ..., D m } which needs to be ranked by its relevance to the query Q i and O = { O 1 , ..., O m } is the set of binary click labels for each relevant document.
Revised: = { D 1 , ..., D m } which needs to be ranked by its relevance to the query Q i and o = { o 1 , ..., o m } is the set of binary click labels for each relevant document.

Original: A query Q i and a document D j is a sequence of words Q i = { w 1 i , ..., w qi } ,
Revised: A query Q i and a document D j is a sequence of words Q i = { w 1 i , ..., w qi } ,

Original: D j = { w 1 j , ..., w dj } where q = | Q i | is the query length and d = | D j | is the document length.
Revised: D j = { w 1 j , ..., w dj } where q = | Q i | is the query length and d = | D j | is the document length.

Original: V is the size of vocabulary constructed over queries and relevant documents.
Revised: V is the size of vocabulary constructed over queries and relevant documents.

Original: A detailed architecture of the multitask neural session relevance framework (M-NSRF) is provided in appendix (see  A).
Revised: A detailed architecture of the multitask neural session relevance framework (M-NSRF) is provided in appendix (see Appendix A).

Original: M-NSRF is composed of two major components, document ranker and query recommender.
Revised: M-NSRF is composed of two major components, document ranker and query recommender.

Original: Given all the previous information in a session and the current query submitted by a user with a set of candidate documents, the document ranker is trained to predict user clicks in the candidate document list.
Revised: Given all the previous information in a session and the current query submitted by a user with a set of candidate documents, the document ranker is trained to predict user clicks in the candidate document list.

Original: At the same time, the query recommender is trained in a sequence to sequence fashion (Sutskever et al., 2014) to predict the user’s next query.
Revised: At the same time, the query recommender is trained in a sequence to sequence fashion (Sutskever et al., 2014) to predict the user’s next query.

Original: The process is repeated for all the queries in a session.
Revised: The process is repeated for all the queries in a session.

Original: In the following, we discuss each component of the neural session relevance model in details.
Revised: In the following, we discuss each component of the neural session relevance model in details.

Original: 3. D OCUMENT R ANKER
Revised: 3. D OCUMENT R ANKER

Original: The document ranker in M-NSRF is made up of a query encoder, a document encoder, a session encoder and a ranker.
Revised: The document ranker in M-NSRF is made up of a query encoder, a document encoder, a session encoder and a ranker.

Original: The technical details of each constituent element is given as follows.
Revised: The technical details of each constituent element is given as follows.

Original: Query Encoder .
Revised: Query Encoder .

Original: Following Conneau et al.
Revised: Following Conneau et al.

Original: (2017), we use bidirectional LSTM network with max pooling technique to form query representation.
Revised: (2017), we use bidirectional LSTM network with max pooling technique to form query representation.

Original: Considering query as a sequence of words Q i = { w 1 i , ..., w qi } , the encoder composed of forward and backward LSTM read the sequence in two opposite directions.
Revised: Considering query as a sequence of words Q i = { w 1 i , ..., w qi } , the encoder composed of forward and backward LSTM read the sequence in two opposite directions.

Original: [Equation]
Revised: [Equation]

Original: where h t ∈ R 2 d is the query-level recurrent state, d is the dimensionality of the LSTM hidden unit which is initialized to zero vector.
Revised: where h t ∈ R 2 d is the query-level recurrent state, d is the dimensionality of the LSTM hidden unit which is initialized to zero vector.

Original: To form a ﬁxed-size vector representation of variable length query, maximum value is selected over each dimension of the hidden units,
Revised: To form a ﬁxed-size vector representation of variable length query, maximum value is selected over each dimension of the hidden units,

Original: [Equation]
Revised: [Equation]

Original: where Q i,k is the k -th element of the latent vector Q i .
Revised: where Q i,k is the k -th element of the latent vector Q i .

Original: Document Encoder .
Revised: Document Encoder .

Original: Document encoder is identical to query encoder.
Revised: Document encoder is identical to query encoder.

Original: The only difference is the dimensionality of the LSTM hidden units.
Revised: The only difference is the dimensionality of the LSTM hidden units.

Original: In general, documents (body or title) are lengthier than query, so we use dense vector of bigger size to represent documents.
Revised: In general, documents (body or title) are lengthier than query, so we use dense vector of bigger size to represent documents.

Original: Session Encoder .
Revised: Session Encoder .

Original: Unlike the query and document encoder, we use a vanilla LSTM for session encoding.
Revised: Unlike the query and document encoder, we use a vanilla LSTM for session encoding.

Original: The session encoder takes the sequence of query representations Q 1 , ..., Q n as input and computes the sequence of session-level recurrent states.
Revised: The session encoder takes the sequence of query representations Q 1 , ..., Q n as input and computes the sequence of session-level recurrent states.

Original: [Equation]
Revised: [Equation]

Original: where S t ∈ R
Revised: where S t ∈ R

Original: d is the session-level recurrent state, d is the dimensionality of the LSTM hidden unit which is initialized to zero vector.
Revised: d is the session-level recurrent state, d is the dimensionality of the LSTM hidden unit which is initialized to zero vector.

Original: The number of session-level recurrent states S t is n , the number of queries in the session.
Revised: The number of session-level recurrent states S t is n , the number of queries in the session.

Original: The session-level recurrent state S t summarizes the queries that have been processed up to position t .
Revised: The session-level recurrent state S t summarizes the queries that have been processed up to position t .

Original: Ranker .
Revised: Ranker .

Original: To rank a set of relevant documents, D = { D 1 , ..., D M } , we concatenate the current query representation Q t with previous session-level recurrent state S t − 1 and apply a non-linear transformation.
Revised: To rank a set of relevant documents, D = { D 1 , ..., D M } , we concatenate the current query representation Q t with previous session-level recurrent state S t − 1 and apply a non-linear transformation.

Original: Finally, to compute relevance score (click probability) between a query and a document, we use sigmoid function.
Revised: Finally, to compute relevance score (click probability) between a query and a document, we use sigmoid function.

Original: [Equation]
Revised: [Equation]

Original: where W r ∈ R ( d q + d s ) × d d and b r ∈ R d d where d q , d s and d d are the dimensionality of the query encoder, session encoder and document encoder hidden units and σ is sigmoid function.
Revised: where W r ∈ R ( d q + d s ) × d d and b r ∈ R d d where d q , d s and d d are the dimensionality of the query encoder, session encoder and document encoder hidden units and σ is sigmoid function.

Original: 3. Q UERY R ECOMMENDER
Revised: 3. Q UERY R ECOMMENDER

Original: Following Sutskever et al.
Revised: Following Sutskever et al.

Original: and Bahdanau et al.
Revised: and Bahdanau et al.

Original: (2014), query recommender in M-NSRM predicts users’ next query using a sequence to sequence approach.
Revised: (2014), query recommender in M-NSRM predicts users’ next query using a sequence to sequence approach.

Original: Basically the query recommender module estimates the probability of the next query Q n = { w 1 , ..., w q } , given all the previous queries, Q 1: n − 1 up to position n − 1 in a session as follows,
Revised: Basically the query recommender module estimates the probability of the next query Q n = { w 1 , ..., w q } , given all the previous queries, Q 1: n − 1 up to position n − 1 in a session as follows,

Original: [Equation]
Revised: [Equation]

Original: In principal, a simple LSTM is the fundamental building block of the query recommender.
Revised: We use LSTM as a fundamental building block for the query recommender.

Original: Information about all the previous queries represented through a session vector S t are passed to the query recommender.
Revised: Information about all the previous queries represented through a session vector S t are passed to the query recommender.

Original: To this end, the recurrent state of the query recommender is initialized with a non-linear transformation of S t , 
Revised: To this end, the recurrent state of the query recommender is initialized with a non-linear transformation of S t , h 0 =

Original: [Equation]
Revised: tanh( W q S t + b q ) ,

Original: where h 0 ∈ R d is the initial recurrent state, d is the dimensionality of the LSTM hidden unit.
Revised: where h 0 ∈ R d is the initial recurrent state, d is the dimensionality of the LSTM hidden unit.

Original: Then the query recommender’s recurrence is computed as follows,
Revised: Then the query recommender’s recurrence is computed by, h t =

Original: [Equation] where h t − 1 is the previous hidden state, w t − 1 is the previous query term.
Revised: LSTM t ( h t − 1 , w t − 1 ) , where h t − 1 is the previous hidden state, w t − 1 is the previous query term.

Original: Finally, each recurrent state is mapped to a probability distribution over the vocabulary, V using a combination of linear transformation and softmax function.
Revised: Finally, each recurrent state is mapped to a probability distribution over the vocabulary, V using a combination of linear transformation and softmax function.

Original: Word with the highest probability is chosen as the next word in sequence.
Revised: Word with the highest probability is chosen as the next word in sequence.

Original: [Equation]
Revised: [Equation]

Original: where w t = arg max w P ( w | w 1: t − 1 , Q 1: n − 1 ) and g ( z ) j = [Equation] , j = 1 , ..., K .
Revised: where w t = arg max w P ( w | w 1: t − 1 , Q 1: n − 1 ) and g ( z ) j = e zj (cid:80) Kk =1 e zk , j = 1 , ..., K .

Original: This formulation has shown to be beneﬁcial for language modelling tasks (Mikolov et al., 2010).
Revised: This formulation has shown to be beneﬁcial for language modelling tasks (Mikolov et al., 2010).

Original: Query Suggestion .
Revised: Query Suggestion .

Original: In our framework, we consider the query suggestion task as an inference problem (Sordoni et al., 2015).
Revised: We consider the query suggestion task as an inference problem (Sordoni et al., 2015).

Original: Given a sequence of queries up to position n − 1 , a suggested query Q n is:
Revised: Given a sequence of queries up to position n − 1 , a suggested query Q n is:

Original: [Equation]
Revised: [Equation]

Original: where Q is the space of all possible queries.
Revised: where Q is the space of all possible queries.

Original: To generate query suggestions of variable lengths, we use standard word-level decoding techniques (Cho et al., 2014).
Revised: To generate query suggestions of variable lengths, we use standard word-level decoding techniques (Cho et al., 2014).

Original: We iteratively consider the best preﬁx w 1: m up to length m and extend it by sampling the most probable word given the distribution in Eq.
Revised: We iteratively consider the best preﬁx w 1: m up to length m and extend it by sampling the most probable word given the distribution in Eq.

Original: The process ends when we obtain a well-formed query containing the special end-ofquery token.
Revised: The process ends when we obtain a well-formed query containing the special end-ofquery token.

Original: 3. L EARNING E ND - TO -E ND
Revised: 3. L EARNING E ND - TO -E ND

Original: Neural session relevance framework ranks documents and predicts next query given a query and a set of candidate documents.
Revised: Neural session relevance framework ranks documents and predicts next query given a query and a set of candidate documents.

Original: Therefore, the training objective of M-NSRF consists of two terms.
Revised: Therefore, the training objective of M-NSRF consists of two terms.

Original: The ﬁrst term is the binary cross entropy loss for the document ranker
Revised: The ﬁrst term is the binary cross entropy loss for the document ranker

Original: [Equation]
Revised: [Equation]

Original: where t i and o i represents binary click label and click probability for the i -th document.
Revised: where t i and o i represents binary click label and click probability for the i -th document ( P ( D i | Q t ) deﬁned in Eq.

Original: The second term is the regularized negative log-likelihood loss for the query suggestion model
Revised: The second term is the regularized negative log-likelihood loss for the query suggestion model

Original: [Equation]
Revised: [Equation]

Original: where current query Q n = { w 1 , ..., w q } is of size | q | , Q 1: n − 1 is all the previous queries in the session S , and
Revised: where current query Q n = { w 1 , ..., w q } is of size | q | , Q 1: n − 1 is all the previous queries in the session S , and

Original: [Equation]
Revised: [Equation]

Original: is the regularization term to avoid the distribution of words in Eq.
Revised: is the regularization term to avoid the distribution of words in Eq.

Original: (1) from highly peaky.
Revised: (2) from highly peaky.

Original: λ is a hyper-parameter to tune.
Revised: λ is a hyper-parameter.

Original: For the sake of simplicity,we consider the ﬁnal objective as the summation of L 1 and L 2 .
Revised: For the sake of simplicity, the ﬁnal objective is the summation of L 1 and L 2 .

Original: Note that the document ranker and query recommenders share the same document, query, and session encoders, and the training of M-NSRF can be done in an online manner using the following procedure.
Revised: Note that the document ranker and query recommenders share the same document, query, and session encoders, and the training of M-NSRF can be done in an online manner using the following procedure.

Original: In the forward pass, M-NSRF computes the query and corresponding document encodings, updates session-level recurrent states, click probability for each candidate document and the log-likelihood of each query in the session given the previous ones.
Revised: In the forward pass, M-NSRF computes the query and corresponding document encodings, updates session-level recurrent states, click probability for each candidate document and the log-likelihood of each query in the session given the previous ones.

Original: In the backward pass, the gradients are computed and the parameters are updated based on ADAM update rule (Kingma & Ba, 2014).
Revised: In the backward pass, the gradients are computed and the parameters are updated based on ADAM update rule (Kingma & Ba, 2014).

Original: Details of implementation can be found in Section 4.
Revised: Details of implementation can be found in Section 4.

Original: 3. M ULTITASK L EARNING F RAMEWORK WITH N EURAL IR M ODELS
Revised: 3. M ULTITASK L EARNING F RAMEWORK FOR N EURAL IR M ODELS

Original: The proposed multitask learning framework is general and can be applied to combine different types of document rankers and query recommenders.
Revised: The proposed multitask learning framework is general and can be applied to combine different types of document rankers and query recommenders.

Original: As mentioned previously, our proposed framework is very close to the query suggestion model proposed in (Sordoni et al., 2015) with a multitask learning framework embedded in.
Revised: As mentioned previously, our proposed framework is very close to the query suggestion model proposed in (Sordoni et al., 2015) with a multitask learning framework embedded in.

Original: So, a query suggestion model working in sequence to sequence fashion can be readily extended to our proposed multitask learning architecture.
Revised: So, a query suggestion model working in sequence to sequence fashion can be readily extended to our proposed multitask learning architecture.

Original: On the other hand, most of the neural IR models are built on the notion of comparing query and document representation in latent space to compute the matching degree.
Revised: On the other hand, most of the neural IR models are built on the notion of comparing query and document representation in latent space to compute the matching degree.

Original: Extending such models to our proposed multitask learning framework is straightforward but due to distinctive nature of different neural IR models, we need to decide how to incorporate context-awareness in the ﬁnal architecture.
Revised: Extending such models to our proposed multitask learning framework is straightforward but due to distinctive nature of different neural IR models, we need to decide how to incorporate context-awareness in the ﬁnal architecture.

Original: In the experiment, we take the Match Tensor model (Jaech et al., 2017), a recently proposed neural relevance model as an example and extend it to a Multitask-Match Tensor (M-Match Tensor) model by incorporating the query recommender module to follow our proposed multitask learning framework.
Revised: In the experiment, we take the Match Tensor model (Jaech et al., 2017), a recently proposed neural relevance model as an example and extend it to a Multitask-Match Tensor (M-Match Tensor) model by incorporating the query recommender module to follow our proposed multitask learning framework.

Original: We take the query and document recurrence state produced by bi-LSTM following a linear projection in original Match Tensor model and feed them to the session-level encoder.
Revised: We take the query and document recurrence state produced by bi-LSTM following a linear projection in original Match Tensor model and feed them to the session-level encoder.

Original: In this way, we follow the same relevance computation process as in original Match Tensor model but jointly train M-Match Tensor model on document ranking and query suggestion.
Revised: In this way, we follow the same relevance computation process as in original Match Tensor model but jointly train M-Match Tensor model on document ranking and query suggestion.

Original:  E XPERIMENTS4. D ATA S ETS AND E VALUATION M ETHODOLOGY
Revised:  E XPERIMENTS

Original: We conduct our experiments on the well-known publicly available AOL search log (Pass et al., 2006).
Revised: We conduct our experiments on the well-known publicly available AOL search log (Pass et al., 2006).

Original: The queries in this dataset were sampled between 1 March, 2006 and 31 May, 2006.
Revised: The queries in this dataset were sampled between 1 March, 2006 and 31 May, 2006.

Original: In total there are 16,946,938 queries submitted by 657,426 unique users.
Revised: In total there are 16,946,938 queries submitted by 657,426 unique users.

Original: We remove all nonalphanumeric characters from the queries, apply word segmentation and lowercasing.
Revised: We remove all nonalphanumeric characters from the queries, apply word segmentation and lowercasing.

Original: We follow (Jansen Bernard et al., 2007) to deﬁne the end of a session by a 30 minute window of idle time.
Revised: We follow (Jansen Bernard et al., 2007) to deﬁne the end of a session by a 30 minute window of idle time.

Original: We ﬁltered sessions based on session length (minimum 2, maximum 10).
Revised: We ﬁltered sessions based on session length (minimum 2, maximum 10).

Original: We only keep the most frequent | V | = 100 k words and map all other words to an < unk > token while constructing the vocabulary.
Revised: We only keep the most frequent | V | = 100 k words and map all other words to an < unk > token while constructing the vocabulary.

Original: In total, we get 1,032,459 sessions for training, 129,053 sessions for development and 91,108 sessions for testing.
Revised: In total, we get 1,032,459 sessions for training, 129,053 sessions for development and 91,108 sessions for testing.

Original: In total train, development and test sessions consists of 2,987,486; 287,138 and 259, queries respectively.
Revised: In total train, development and test sessions consists of 2,987,486; 287,138 and 259, queries respectively.

Original: We consider two tasks in multitask learning framework – document ranking and query suggestion.
Revised: We consider two tasks in multitask learning framework – document ranking and query suggestion.

Original: In the document ranking task, our goal is to rank candidate documents’ titles based on their relevance with the query.
Revised: In the document ranking task, our goal is to rank candidate documents’ titles based on their relevance with the query.

Original: We model the document ranking task as a binary classiﬁcation task.
Revised: We model the document ranking task as a binary classiﬁcation task.

Original: Essentially, given a query and document, we predict if the document will be clicked by a user.
Revised: Essentially, given a query and document, we predict if the document will be clicked by a user.

Original: The ﬁnal predictions are evaluated using  ranking metrics.
Revised: The ﬁnal predictions are evaluated using two ranking metrics, mean average precision (MAP) and normalized discounted cumulative gain (NDCG) metric computed at positions one, three and ten.

Original: Since, AOL search log only contains clicks (positive examples), we selected negative examples through random sampling from the top 100 documents retrieved by BM25.
Revised: Since, AOL search log only contains clicks (positive examples), we selected negative examples through random sampling from the top 100 documents retrieved by BM25.

Original: Each query in the test set consists of 50 candidates including the clicked documents.
Revised: Each query in the test set consists ofcandidates including the clicked documents.

Original: However, to reduce the training time and memory use, each query in training and development set only contains 5 candidates.
Revised: However, to reduce the training time and memory use, each query in training and development set only contains 5 candidates.

Original: In the query suggestion task, we aim at predicting the next query in the same session.
Revised: In the query suggestion task, we aim at predicting the next query in the same session.

Original: All evaluation and empirical analysis for document ranking used mean average precision and normalized discounted cumulative gain (NDCG) metric computed at positions one, three and ten.

Original: For query suggestion task, we computed the BLEU scores (Papineni et al., 2002).
Revised: For evaluation, we computed the BLEU scores (Papineni et al., 2002).

Original: 4. B ASELINES AND I MPLEMENTATION D ETAILS
Revised: 4. B ASELINES AND I MPLEMENTATION D ETAILS

Original: Document Ranking Baselines .
Revised: Document Ranking Baselines .

Original: We compare M-NSRF’s performance with word-based baselines and neural ranking baselines.
Revised: We compare M-NSRF and M-Match Tensor with word-based baselines and neural network-based baselines.

Original: Word-based baselines include query likelihood model based on Dirichlet smoothing (QL)
Revised: Word-based baselines include query likelihood model based on Dirichlet smoothing (QL)

Original: (Ponte & Croft, 1998) and BM25 (Robertson et al., 2009).
Revised: (Ponte & Croft, 1998) and BM25 (Robertson et al., 2009).

Original: In addition, following (Mitra et al., 2016), we investigated the ranking performance of a simple word embedding-based model using GloVe word embeddings (Pennington et al., 2014).
Revised: In addition, following (Mitra et al., 2016), we investigated the ranking performance of a simple word embedding-based model using GloVe word embeddings (Pennington et al., 2014).

Original: To compare with neural models, we consider baselines broadly categorized in representation-focused, interaction-focused and a combination of both.
Revised: To compare with neural models, we consider baselines broadly categorized in representation-focused, interactionfocused and a combination of both.

Original: Representation-focused neural baselines include: DSSM (Huang et al., 2013),
Revised: Representation-focused neural baselines include: DSSM (Huang et al CLSM al ARC- I

Original: CLSM (Shen et al., 2014), ARC-I (Hu et al., 2014) and interaction-focused baselines include: ARCII (Hu et al.,2014) and DRMM (Guo et al., 2016a).
Revised: Hu al. interaction-focused baselines ARC-II (Hu et al., DRMM ( et al.

Original: A combination of representationand interaction focused models include: DUET (Mitra et al., 2017) and Match Tensor (Jaech et al., 2017). Details of these models are provided in appendix (see B). We implemented all the baseline models in PyTorch.
Revised: representation

Original: Query Suggestion Baselines .
Revised: Query Suggestion Baselines .

Original: To evaluate performance on query suggestion task, we consider three baseline methods including Seq2seq model proposed by Bahdanau et al.
Revised: To evaluate performance on query suggestion task, we consider three baseline methods including Seq2seq model proposed by Bahdanau et al.

Original: (2014), Seq2seq with global attention mechanism (Luong et al., 2015) and HRED-qs
Revised: (2014), Seq2seq with global attention mechanism (Luong et al., 2015) and HRED-qs

Original: (Sordoni et al., 2015).
Revised: (Sordoni et al., 2015).

Original: Details of these models are provided in appendix (see C).
Revised: Details of these models are provided in appendix (see C).

Original: We implemented all three baselines in PyTorch and optimized using negative log-likelihood loss as in Eq.
Revised: We implemented all three baselines in PyTorch and optimized using negative log-likelihood loss as in Eq.

Original: Implementation Details of M-NSRF .
Revised: Implementation Details of M-NSRF .

Original: The multitask neural session relevance model was trained end-to-end and we used mini-batch SGD with Adam (Kingma & Ba, 2014) for optimization with the two momentum parameters set to 0.9 and 0.999 respectively.
Revised: The multitask neural session relevance model was trained end-to-end and we used mini-batch SGD with Adam (Kingma & Ba, 2014) for optimization with the two momentum parameters set to 0.9 and 0.999 respectively.

Original: We use 300-dimentional word vectors trained with GloVe (Pennington et al., 2014) on 840 billion of tokens to initialize the word embeddings.
Revised: We use 300-dimentional word vectors trained with GloVe (Pennington et al., 2014) on 840 billion of tokens to initialize the word embeddings.

Original: Out-of-vocabulary words were randomly initialized by sampling values from a zeromean unit-variance normal distribution.
Revised: Out-of-vocabulary words were randomly initialized by sampling values from a zeromean unit-variance normal distribution.

Original: All training used a mini-batch size of 32 to ﬁt in single GPU memory.
Revised: All training used a mini-batch size of 32 to ﬁt in single GPU memory.

Original: Learning rate was ﬁxed to 0.001.
Revised: Learning rate was ﬁxed to 0.001.

Original: We used dropout (0.10) (Srivastava et al., 2014) and early stopping with a patience of 5 epochs were used for regularization.
Revised: We used dropout (0.10) (Srivastava et al., 2014) and early stopping with a patience of 5 epochs were used for regularization.

Original: M-NSRF is implemented in PyTorch and it runs on a single GPU (TITAN X) with roughly a runtime of 90 minutes per epoch.
Revised: M-NSRF is implemented in PyTorch and it runs on a single GPU (TITAN X) with roughly a runtime of 90 minutes per epoch.

Original: In general, M-NSRF runs up to 20 epochs and we select the model that achieves the minimum loss on the development set.[Table] (this paper [Table]
Revised: In general, M-NSRF runs up to 20 epochs and we select the model that achieves the minimum loss on the development set.

Original: this paper

Original: [Table]

Original: [Table]

Original:  E VALUATION R ESULTS
Revised: 4. E VALUATION R ESULTS

Original: Document Ranking Accuracy .
Revised: Document Ranking Accuracy .

Original: Table 1 shows the ranking accuracy of M-NSRF and the baseline models.
Revised: Table 1 shows the performance of NSRF, M-NSRF, M-Match Tensor and other baseline models.

Original: M-NSRF signiﬁcantly outperforms all the baselines except  MatchTensor model.
Revised: NSRF signiﬁcantly outperforms all the baselines except the Match

Original: We suspect that because of long and complex architecture to capture immediate and larger contexts by Match Tensor  model makes it more effective than M-NSRF. With multitask learning, the performance of Match Tensor further improves.
Revised: Tensor model. However, the model size of NSRF is much smaller than Match Tensor. With multitask learning, both M-NSRF and M-Match Tensor outperform NSRF and Match Tensor, respectively.

Original: We developed M-Match Tensor model described in 3.4 which slightly improves its’ performance. To avoid complexity in M-Match Tensor model, we did not consider session recurrence in document ranking, rather investigated the utility added by the multitask learning principle. 
Revised: To avoid complexity in M-Match Tensor model, we did not consider session recurrence in document ranking, rather investigated the utility added by the multitask learning principle. To study the advantage of multitask learning on our proposed approach, we trained M-NSRF only on document [Table]

Revised: (this paper)

Revised: [Table]

Original: To study the advantage of multitask learning on our proposed approach, we trained M-NSRF only on document ranking task (noted as NSRF in Table 1) and observed signiﬁcant performance drop which endorses the beneﬁts of multitask learning.
Revised: (this paper [Table] [Table] ranking task (noted as NSRF in Table 1) and observed signiﬁcant performance drop which endorses the beneﬁts of multitask learning.

Original: Previous state-of-the-art techniques DRMM and DUET architecture performed poorly in our experimental settings.
Revised: Previous state-of-the-art techniques DRMM and DUET architecture performed suboptimal in our experimental settings.

Original: We believe because of simple architecture of DRMM with few hundreds parameters, the model fall behind to show competitive performance on the evaluation dataset.
Revised: We believe because of simple architecture of DRMM with few hundreds parameters, the model fall behind to show competitive performance on the evaluation dataset.

Original: On the other hand, we suspect that the use of smaller number of top character n -graphs (in our case, 5000) by DUET architecture limits its’ effectiveness in modeling representation and interaction focused features to compute matching degree between query and document.
Revised: On the other hand, we suspect that the use of smaller number of top character n -graphs (in our case, 5000) by DUET architecture limits its’ effectiveness in modeling representation and interaction focused features to compute matching degree between query and document.

Original: In addition, use of document title while computing relevance between query and document in our experimental setting may limit the performance of DRMM and DUET architecture.
Revised: In addition, use of document title while computing relevance between query and document in our experimental setting may limit the performance of DRMM and DUET architecture.

Original: It is important to note that, some negative results have been reported for document title-based ad hoc retrieval tasks (Guo et al., 2016a), so in our future work, we want to investigate M-NSRF and all baseline model’s performance by leveraging document body.
Revised: It is important to note that, some negative results have been reported for document title-based ad hoc retrieval tasks (Guo et al., 2016a), so in our future work, we want to investigate M-NSRF and all baseline model’s performance by leveraging document body.

Original: Query Suggestion Accuracy Given the all the previous queries in a session, whether M-NSRF or baseline models can predict the user’s next query is evaluated and results are presented in table 2.
Revised: Query Suggestion Accuracy Given the all the previous queries in a session, whether M-NSRF or baseline models can predict the user’s next query is evaluated and results are presented in table 2.

Original: While M-NSRF and HRED-qs considers previous query information from same session, the other two baselines only consider the current query and predicts next one.
Revised: While M-NSRF and HRED-qs considers previous query information from same session, the other two baselines only consider the current query and predicts next one.

Original: Table 2 shows that M-NSRF outperformed all baselines though the performance is still far from good.
Revised: Table 2 shows that M-NSRF outperformed all baselines though the performance is still far from good.

Original: From our observation, we found that Seq2seq with global attention mechanism performed relatively well but the entropy reg- ularization technique helped M-NSRF to achieve little improvement over the model’s performance.
Revised: From our observation, we found that Seq2seq with global attention mechanism performed relatively well but the entropy regularization technique helped M-NSRF to achieve little improvement over the model’s performance.

Original: Though we found multitask learning helped M-NSRF to achieve better performance in document ranking, we could not see much gain on the query suggestion task.
Revised: Though we found multitask learning helped M-NSRF to achieve better performance in document ranking, we could not see much gain on the query suggestion task.

Original: Adding attention over session information and personalization to improve query suggestions are two future directions, we are interested to work on.
Revised: Adding attention over session information and personalization to improve query suggestions are two future directions, we are interested to work on.

Original: Examples of predicted queries by M-NSRF given previous queries from same session is presented in table 3 (more examples are provided inappendix, see D).
Revised: Examples of predicted queries by M-NSRF given previous queries from same session is presented in table 3 (more examples are provided in

Original: [Table]
Revised: [Table]

Original: We conducted experiments to better understand the effectiveness of different components in the MNSRF.
Revised: [Table]

Original: We also analyzed the impact of word embeddings and hidden units dimension in M-NSRF’s performance.
Revised: ) 0.232 ( − 0.336) 0.296 ( − 0.348)

Original: Our ﬁndings are presented in table [Table] [Table] [Table]{+GloVe 6B 100d 0.312 ( − 0.269) 0.241 ( − 0.282) 0.
+}
Revised: {+GloVe 6B 100d 0.312 ( − 0.269) 0.241 ( − 0.282) 0.
+}

Original: [Table] [Table] [Table] − 
Revised: ( − 0.295) 0.306 ( − 0.308)

Original: [Table]
Revised: GloVe 6B 200d 0.378 ( − 0.203) 0.356 ( − 0.167) 0.447 ( − 0.121) 0.498 ( − 0.116)

Original: [Table]
Revised: [Table]

Original: 544 ( −

Original: [Table]
Revised: [Table]

Original: [-) 0.582 ( − 0.032) GloVe 6B 50d 0.247 ( − 0.324) 0.196 ( − 0.329) 0.232 ( − 0.336) 0.296 ( − 0.348) GloVe 6B 100d 0.312 ( − 0.269) 0.241 ( − 0.282) 0.
-]
Revised: [-) 0.582 ( − 0.032) GloVe 6B 50d 0.247 ( − 0.324) 0.196 ( − 0.329) 0.232 ( − 0.336) 0.296 ( − 0.348) GloVe 6B 100d 0.312 ( − 0.269) 0.241 ( − 0.282) 0.
-][Table]

Original: ( − 0.295) 0.306 ( − 0.308)
Revised: [Table]

Original: GloVe 6B 200d 0.378 ( − 0.203) 0.356 ( − 0.167) 0.447 ( − 0.121) 0.498 ( −0.116)
Revised: 507 ( −

Original: [Table]
Revised: [Table]

Original: 0.562 ( − 0.019) 0.507 ( − 0.016) 0.544 ( −0.024) 0.582 ( − 0.032)
Revised: ) 0.544 ( −

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: ( −

Original: [Table]
Revised: [Table]

Original: ) 0.528 ( +
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ) 0.571 ( + 0.003) 0.617 ( + 0.003)
Revised: We conducted experiments to better understand the effectiveness of different components in the MNSRF. We also analyzed the impact of word embeddings and hidden units dimension in M-NSRF’s performance. Our ﬁndings are presented in table 4.

Original: Impact of Different Model Components .
Revised: Impact of Different Model Components .

Original: To study the effect of different model components, we compare the full M-NSRF with several simpler versions of the model.
Revised: To study the effect of different model components, we compare the full M-NSRF with several simpler versions of the model.

Original: At ﬁrst, we turned off training for word embeddings and found a large drop in performance.
Revised: At ﬁrst, we turned off training for word embeddings and found a large drop in performance.

Original: In our training dataset, we have roughly | OOV | = 26 k out-of-vocabulary words and so, training the word embeddings turned out to be very important for our experimental setting.
Revised: In our training dataset, we have roughly | OOV | = 26 k out-of-vocabulary words and so, training the word embeddings turned out to be very important for our experimental setting.

Original: Also, we investigated the role of pre-trained word embeddings (ex., GloVe embeddings) and found signiﬁcant performance drop (27.9% drop in MAP) if we train M-NSRF without any pre-trained word embeddings.
Revised: Also, we investigated the role of pre-trained word embeddings (ex., GloVe embeddings) and found signiﬁcant performance drop (27.9% drop in MAP) if we train M-NSRF without any pre-trained word embeddings.

Original: Hence, we can conclude that use of pre-trained word embeddings and training them further is important to achieve better performance in M-NSRF.
Revised: Hence, we can conclude that use of pre-trained word embeddings and training them further is important to achieve better performance in M-NSRF.

Original: Secondly, we investigated the advantages of using max-pooling over mean-pooling and considering last hidden recurrent state for query and document representation.
Revised: Secondly, we investigated the advantages of using max-pooling over mean-pooling and considering last hidden recurrent state for query and document representation.

Original: We observed that max-pooling and mean-pooling provides almost same performance while biLSTM-last approach lags slightly.
Revised: We observed that max-pooling and mean-pooling provides almost same performance while biLSTM-last approach lags slightly.

Original: To further analyze the features identiﬁed by the query and document encoders using max-pooling technique, we follow the idea of visualization proposed in (Conneau et al., 2017).
Revised: To further analyze the features identiﬁed by the query and document encoders using max-pooling technique, we follow the idea of visualization proposed in (Conneau et al., 2017).

Original: We provide an example in ﬁgure 2 where document 1 is clicked by the user (positive example) and document 2, 3 is retrieved by BM25 and considered as unclicked document (negative examples).
Revised: We provide an example in ﬁgure 2 where document 1 is clicked by the user (positive example) and document 2, 3 is retrieved by BM25 and considered as unclicked document (negative examples).

Original: We observed that the query and document encoders identify distinguishing features (ex., the word priceline in the ﬁrst document’s title is most important) which help to differentiate between clicked and unclicked documents.
Revised: We observed that the query and document encoders identify distinguishing features (ex., the word priceline in the ﬁrst document’s title is most important) which help to differentiate between clicked and unclicked documents.

Original: In another variant of M-NSRF, we did not use session recurrent state while computing relevance score for the candidate documents to examine the inﬂuence of previous information from same session on the ranking performance.
Revised: In another variant of M-NSRF, we did not use session recurrent state while computing relevance score for the candidate documents to examine the inﬂuence of previous information from same session on the ranking performance.

Original: We call this variant of M-NSRF as neural relevance model (M-NRM).
Revised: We call this variant of M-NSRF as neural relevance model (M-NRM).

Original: From table 4 we can see, without session information the performance drops by 2.8% in terms of MAP.
Revised: From table 4 we can see, without session information the performance drops by 2.8% in terms of MAP.

Original: We further investigated and found session information helps particularly for longer sessions (session length > 5 ).
Revised: We further investigated and found session information helps particularly for longer sessions (session length > 5 ).

Original: In our evaluation dataset, we had roughly 5000 sessions of length greater than 5, so the performance difference is not reasonably high as we expected.
Revised: In our evaluation dataset, we had roughly 5000 sessions of length greater than 5, so the performance difference is not reasonably high as we expected.

Original: We will investigate whether directly incorporating click information in the session recurrence helps in improving the performance signiﬁcantly.
Revised: We will investigate whether directly incorporating click information in the session recurrence helps in improving the performance signiﬁcantly.

Original: Impact of Dimensionality .
Revised: Impact of Dimensionality .

Original: We further study the impact of dimensionality of the word embeddings, query, document and session latent vectors.
Revised: We further study the impact of dimensionality of the word embeddings, query, document and session latent vectors.

Original: In M-NSRF, we set the dimension of query, document and session latent vectors as 256, 512 and 1024 respectively.
Revised: In M-NSRF, we set the dimension of query, document and session latent vectors as 256, 512 and 1024 respectively.

Original: As shown in table 4, decreasing the dimensions of latent vectors, drops the performance while increasing the dimensions further, does not affect the performance signiﬁcantly.
Revised: As shown in table 4, decreasing the dimensions of latent vectors, drops the performance while increasing the dimensions further, does not affect the performance signiﬁcantly.

Original: We also experimented with different dimensions of pretrained word embeddings (50d, 100d and 200d GloVe (Pennington et al., 2014) embeddings).
Revised: We also experimented with different dimensions of pretrained word embeddings (50d, 100d and 200d GloVe (Pennington et al., 2014) embeddings).

Original: Word embeddings of different dimensionality provide different granularity of semantic similarity; with lower dimensionality, the similarity between word embeddings might be coarse and thus hard to capture matching between two text sequences.
Revised: Word embeddings of different dimensionality provide different granularity of semantic similarity; with lower dimensionality, the similarity between word embeddings might be coarse and thus hard to capture matching between two text sequences.

Original: In our experiment, we found 300 dimension for embeddings works signiﬁcantly better than other dimensionality.
Revised: In our experiment, we found 300 dimension for embeddings works signiﬁcantly better than other dimensionality.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Existing deep neural models for ad-hoc retrieval often omit session information and are only trained on query-document pairs.
Revised: Existing deep neural models for ad-hoc retrieval often omit session information and are only trained on query-document pairs.

Original: In this work, we propose a context-aware multitask neural session relevance framework which works in a sequence to sequence fashion and show that sharing session-level latent recurrent states across document ranking and query suggestion task beneﬁts each other.
Revised: In this work, we propose a context-aware multitask neural session relevance framework which works in a sequence to sequence fashion and show that sharing session-level latent recurrent states across document ranking and query suggestion task beneﬁts each other.

Original: Our experiments and analysis not only demonstrated the effectiveness of the proposed framework, but also provides useful intuitions about the advantages of multitask learning involving deep neural networks for IR tasks.
Revised: Our experiments and analysis not only demonstrated the effectiveness of the proposed framework, but also provides useful intuitions about the advantages of multitask learning involving deep neural networks for IR tasks.

Original: For future work we would like to leverage document body to train M-NSRF so that we can further explore the potential of the proposed framework on ad-hoc retrieval.
Revised: For future work, we would like to leverage document body to train M-NSRF so that we can further explore the potential of the proposed framework on ad-hoc retrieval.

Original: In addition, a broad research direction would be to ﬁnd ways to summarize individual users’ search log to model long-term search goals to enhance personalized search results and query suggestions
Revised: In addition, a broad research direction would be to ﬁnd ways to summarize individual users’ search log to model long-term search goals to enhance personalized search results and query suggestions

Original: B D OCUMENT R ANKING B ASELINES
Revised: B D OCUMENT R ANKING B ASELINES

Original: Deep semantic similarity model, DSSM (Huang et al., 2013) maps words to letter tri-grams using a word-hashing technique and uses a feed-forward neural network to build representations for both query and document.
Revised: Deep semantic similarity model, DSSM (Huang et al., 2013) maps words to letter tri-grams using a word-hashing technique and uses a feed-forward neural network to build representations for both query and document.

Original: Similarity, convolutional latent semantic model, CLSM (Shen et al., 2014) uses word-hashing technique and uses convolutional neural networks (CNN) to build query and document representations.
Revised: Similarity, convolutional latent semantic model, CLSM (Shen et al., 2014) uses word-hashing technique and uses convolutional neural networks (CNN) to build query and document representations.

Original: To compute relevance between query and document, both DSSM and
Revised: To compute relevance between query and document, both DSSM and

Original: CLSM uses cosine similarity.
Revised: CLSM uses cosine similarity.

Original: ARC-I (Hu et al., 2014) uses CNN to form query and document representations and employs a multi-layer perceptron to compute relevance score.
Revised: ARC-I (Hu et al., 2014) uses CNN to form query and document representations and employs a multi-layer perceptron to compute relevance score.

Original: In our implementation, we used 128 convolution ﬁlters of size 1, 2 and 256 ﬁlters of size 3.
Revised: In our implementation, we used 128 convolution ﬁlters of size 1, 2 and 256 ﬁlters of size 3.

Original: ARC-II (Hu et al., 2014) was proposed by focusing on learning hierarchical matching patterns from local interactions using a CNN.
Revised: ARC-II (Hu et al., 2014) was proposed by focusing on learning hierarchical matching patterns from local interactions using a CNN.

Original: To keep the ARC-II model simple, we use two layers of 2d convolution and max-pooling each and two-layer feed forward neural network to compute relevance score.
Revised: To keep the ARC-II model simple, we use two layers of 2d convolution and max-pooling each and two-layer feed forward neural network to compute relevance score.

Original: DRMM (Guo et al., 2016a) aims to perform term matching over histogram-based features ignoring the actual position of matches.
Revised: DRMM (Guo et al., 2016a) aims to perform term matching over histogram-based features ignoring the actual position of matches.

Original: In DRMM, histogram-based features are computed using exact term matching and pretrained word embeddings based cosine similarities.
Revised: In DRMM, histogram-based features are computed using exact term matching and pretrained word embeddings based cosine similarities.

Original: In principal, the histogram counts the number of word pairs at different similarity levels.
Revised: In principal, the histogram counts the number of word pairs at different similarity levels.

Original: The counts are combined by a feed forward network to produce ﬁnal ranking scores.
Revised: The counts are combined by a feed forward network to produce ﬁnal ranking scores.

Original: The DUET (Mitra et al., 2017) model composed of a local and distributed model where the distributed model projects the query and the document text into an embedding space before matching, while the local model operates over an interaction matrix comparing every query term to every document term.
Revised: The DUET (Mitra et al., 2017) model composed of a local and distributed model where the distributed model projects the query and the document text into an embedding space before matching, while the local model operates over an interaction matrix comparing every query term to every document term.

Original: Similarly, Match Tensor (Jaech et al., 2017) model incorporates both immediate and larger contexts in a given document when comparing document to a query.
Revised: Similarly, Match Tensor (Jaech et al., 2017) model incorporates both immediate and larger contexts in a given document when comparing document to a query.

Original: C Q UERY S UGGESTION B ASELINES
Revised: C Q UERY S UGGESTION B ASELINES

Original: Seq2seq model proposed by Bahdanau et al.
Revised: Seq2seq model proposed by Bahdanau et al.

Original: (2014) is a general neural network architecture that can be applied to the task where both input and output consist of a sequence of tokens.
Revised: (2014) is a general neural network architecture that can be applied to the task where both input and output consist of a sequence of tokens.

Original: This method have been shown successful in machine translation and sequential tagging.
Revised: This method have been shown successful in machine translation and sequential tagging.

Original: Because different input tokens may contribute to each output token differently, attention mechanism which learns a weight between each input-output token pair can further improve the Seq2seq model.
Revised: Because different input tokens may contribute to each output token differently, attention mechanism which learns a weight between each input-output token pair can further improve the Seq2seq model.

Original: In this paper, we consider a seq2seq with global attention method proposed by Luong et al.
Revised: In this paper, we consider a seq2seq with global attention method proposed by Luong et al.

Original: (2015), which is suitable for short text such as web queries.
Revised: (2015), which is suitable for short text such as web queries.

Original: HRED-qs suggested by Sordoni et al.
Revised: HRED-qs suggested by Sordoni et al.

Original: (2015) is very close to our work which proposed to use a hierarhical recurrent encoder-decoder approach by considering session information for context-aware query suggestion.
Revised: (2015) is very close to our work which proposed to use a hierarhical recurrent encoder-decoder approach by considering session information for context-aware query suggestion.

Original: D M ORE E XAMPLES OF Q UERY S UGGESTION BY M-NSRF
Revised: D M ORE E XAMPLES OF Q UERY S UGGESTION BY M-NSRF

Original: Previous session queries discount pet supplies, homes for rent smyrna georgia
Revised: Previous session queries discount pet supplies, homes for rent smyrna georgia

Original: Next user query homes for rent atlanta georgia
Revised: Next user query homes for rent atlanta georgia

Original: Suggested next query pet friendly rentals in georgia
Revised: Suggested next query pet friendly rentals in georgia

Original: Previous session queries language aptitude test, foreign language aptitude test
Revised: Previous session queries language aptitude test, foreign language aptitude test

Original: Next user query american idol
Revised: Next user query american idol

Original: Suggested next query american language association
Revised: Suggested next query american language association

Original: Previous session queries saturday night fever, saturday night fever nj band
Revised: Previous session queries saturday night fever, saturday night fever nj band

Original: Next user query new jersey cover band
Revised: Next user query new jersey cover band

Original: Suggested next query saturday night live
Revised: Suggested next query saturday night live

Original: Previous session queries pregnancy, abortion, abortion clinics
Revised: Previous session queries pregnancy, abortion, abortion clinics

Original: Next user query tampa abortion
Revised: Next user query tampa abortion

Original: Suggested next query abortion clinics in ﬂorida
Revised: Suggested next query abortion clinics in ﬂorida

Original: Previous session queries ncaa basketball, ncaa basketball trees, ncaa mens basketball bracket, sportscenter Next user query mens ncaa basketball odds
Revised: Previous session queries ncaa basketball, ncaa basketball trees, ncaa mens basketball bracket, sportscenter Next user query mens ncaa basketball odds

Original: Suggested next query espn
Revised: Suggested next query espn

Original: Previous session queries childhood autism rating scale, childhood autism rating scale free, autism screening questionnaire Next user query pervasive developmental disorder
Revised: Previous session queries childhood autism rating scale, childhood autism rating scale free, autism screening questionnaire Next user query pervasive developmental disorder

Original: Suggested next query how to do questionnaire
Revised: Suggested next query how to do questionnaire