Original: M ULTI -T ASK L EARNING FOR D OCUMENT R ANKING AND Q UERY S UGGESTION
Revised: M ULTI -T ASK L EARNING FOR D OCUMENT R ANKING AND Q UERY S UGGESTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a multi-task learning framework to jointly learn document ranking and query suggestion for web search.
Revised: We propose a multi-task learning framework to jointly learn document ranking and query suggestion for web search.

Original: It consists of two major components, a document ranker, and a query recommender.
Revised: It consists of two major components, a document ranker, and a query recommender.

Original: Document ranker combines current query and session information and compares the combined representation with document representation to rank the documents.
Revised: Document ranker combines current query and session information and compares the combined representation with document representation to rank the documents.

Original: Query recommender tracks users’ query reformulation sequence considering all previous in-session queries using a sequence to sequence approach.
Revised: Query recommender tracks users’ query reformulation sequence considering all previous in-session queries using a sequence to sequence approach.

Original: As both tasks are driven by the users’ underlying search intent, we perform joint learning of these two components through session recurrence, which encodes search context and intent.
Revised: As both tasks are driven by the users’ underlying search intent, we perform joint learning of these two components through session recurrence, which encodes search context and intent.

Original: Extensive comparisons against state-of-the-art document ranking and query suggestion algorithms are performed on the public AOL search log, and the promising results endorse the effectiveness of the joint learning framework.
Revised: Extensive comparisons against state-of-the-art document ranking and query suggestion algorithms are performed on the public AOL search log, and the promising results endorse the effectiveness of the joint learning framework.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Understanding users’ information need is the key to optimize a search engine for providing relevant search results.
Revised: Understanding users’ information need is the key to optimize a search engine for providing relevant search results.

Original: Search engine logs have been extensively used to mine users’ search intent, reﬂected in their search result click preferences and query reformulations (Baeza-Yates et al., 2004; Croft et al., 2010).
Revised: Search engine logs have been extensively used to mine users’ search intent, reﬂected in their search result click preferences and query reformulations (Baeza-Yates et al., 2004; Croft et al., 2010).

Original: In particular, user query logs are partitioned into search sessions, i.e., sequences of queries and clicks issued by the same user and within a short time interval.
Revised: In particular, user query logs are partitioned into search sessions, i.e., sequences of queries and clicks issued by the same user and within a short time interval.

Original: Search sessions provide useful contextual information about user intent and help to narrow down ambiguity while ranking documents for the current query and predicting next query that users will submit, a.k.a.
Revised: Search sessions provide useful contextual information about user intent and help to narrow down ambiguity while ranking documents for the current query and predicting next query that users will submit, a.k.a.

Original: contextawareness (Jiang et al., 2014).
Revised: contextawareness (Jiang et al., 2014).

Original: Since both a user’s click behavior and query reformulation are driven by the underlying search intent, we argue that jointly modeling both tasks can beneﬁt each other.
Revised: Since both a user’s click behavior and query reformulation are driven by the underlying search intent, we argue that jointly modeling both tasks can beneﬁt each other.

Original: In this work, we propose a joint learning framework, called multi-task neural session relevance framework (M-NSRF), to predict users’ result clicks and future queries in search sessions.
Revised: In this work, we propose a joint learning framework, called multi-task neural session relevance framework (M-NSRF), to predict users’ result clicks and future queries in search sessions.

Original: We model search context within a session via a recurrent latent state in a deep neural network (Collobert & Weston, 2008; Liu et al., 2015), which governs the generation of result clicks in the current query and formation of next query.
Revised: We model search context within a session via a recurrent latent state in a deep neural network (Collobert & Weston, 2008; Liu et al., 2015), which governs the generation of result clicks in the current query and formation of next query.

Original: By sharing the latent states across the tasks of document ranking and query suggestion, we learn the representations of queries, documents and user intent carrying over the whole session jointly, i.e., multi-task learning.
Revised: By sharing the latent states across the tasks of document ranking and query suggestion, we learn the representations of queries, documents and user intent carrying over the whole session jointly, i.e., multi-task learning.

Original: Given the fact that web search queries are usually very short and the number of clicks in each query is also very few, our joint learning framework greatly reduces the sample complexity in learning each individual task and effectively improves generalization of both learnt models.
Revised: This multi-task learning framework is ﬂexible and can be incorporated with existing approaches for representing query and documents.

Original: The general workﬂow of M-NSRF is illustrated in Figure 1.
Revised: The general workﬂow of M-NSRF is illustrated in Figure 1.

Original: Given a sequence of queries from the same search session, “cheap furniture” and “craig list virginia” , M-NSRF is trained to predict the result clicks under the current query, and the next query, “cheap furniture for sale.”
Revised: Given a sequence of queries from the same search session, “cheap furniture” and “craig list virginia” , M-NSRF is trained to predict the result clicks under the current query, and the next query, “cheap furniture for sale.”

Original: It is evident that in this search session the user kept reformulating the queries because his/her information need has not been met by the result clicks in the previously submitted queries, which is reﬂected in the added and removed query terms.
Revised: It is evident that in this search session the user kept reformulating the queries because his/her information need has not been met by the result clicks in the previously submitted queries, which is reﬂected in the added and removed query terms.

Original: And such revisions suggest what he/
Revised: And such revisions suggest what he/

Original: she might want to click next.
Revised: she might want to click next.

Original: As a result, the modeling of result clicks and query reformulations mutually reinforce each other in revealing users’ underlying search intent.
Revised: As a result, the modeling of result clicks and query reformulations mutually reinforce each other in revealing users’ underlying search intent.

Original: In M-NSRF, we model user search intent as a sessionlevel recurrent state, the learning of which is aided by both document ranking and query prediction tasks.
Revised: In M-NSRF, we model user search intent as a session-level recurrent state, the learning of which is aided by both document ranking and query prediction tasks.

Original: Our proposed framework is not restricted to any speciﬁc architecture to represent queries and documents and can be trained end-to-end on search sessions.

Original: We evaluate the effectiveness of the proposed framework using the publicly available AOL search log and compare with several well-known classical retrieval models, as well as various neural retrieval models speciﬁcally designed for ad-hoc retrieval.
Revised: We evaluate the effectiveness of the proposed framework using the publicly available AOL search log and compare with several well-known classical retrieval models, as well as various neural retrieval models speciﬁcally designed for ad-hoc retrieval.

Original: We also compare M-NSRF with several baseline models for the  prediction task empirical forms
Revised: We also compare M-NSRF with several baseline models for the query prediction task.

Revised: The empirical results show that by leveraging infor- mation in proposed approach

Original: To summarize the key contributions of this work include:
Revised: To summarize the key contributions of this work include:

Original: We propose a novel multi-task neural session relevance model that is jointly trained on document ranking and query suggestion tasks by utilizing in-session queries and clicks in a holistic way.
Revised: We propose a novel multi-task neural session relevance model that is jointly trained on document ranking and query suggestion tasks by utilizing in-session queries and clicks in a holistic way.

Original: We provide detailed experiment analysis, and will release the implementation and the data processing tool to facilitate future research.
Revised: We provide detailed experiment analysis, and will release the implementation and the data processing tool to facilitate future research.

Original:  R ELATED W ORK AND B ACKGROUND
Revised:  R ELATED W ORK AND B ACKGROUND

Original: Ad-hoc Retrieval.
Revised: Ad-hoc Retrieval.

Original: Traditional retrieval models such as query likelihood (Ponte & Croft, 1998) and
Revised: Traditional retrieval models such as query likelihood (Ponte & Croft, 1998) and

Original: BM25 (Robertson et al., 2009) are based on exact matching of query and document words with a variety of smoothing, weighting and normalization techniques.
Revised: BM25 (Robertson et al., 2009) are based on exact matching of query and document words with a variety of smoothing, weighting and normalization techniques.

Original: Recently, deep neural network based approaches demonstrate strong advance in ad-hoc retrieval.
Revised: Recently, deep neural network based approaches demonstrate strong advance in ad-hoc retrieval.

Original: Existing neural ranking models fall into two categories: representation focused (Huang et al., 2013; Gao et al., 2014) and interaction focused (Guo et al., 2016b).
Revised: Existing neural ranking models fall into two categories: representation focused (Huang et al., 2013; Gao et al., 2014) and interaction focused (Guo et al., 2016b).

Original: The earlier focus of neural ranking models was mainly on representation based models (Hu et al., 2014;
Revised: The earlier focus of neural ranking models was mainly on representation based models (Hu et al., 2014;

Original: Shen et al., 2014), in which the query and documents are ﬁrst embedded into continuous vectors, and the ranking is calculated based on their embeddings’ similarity.
Revised: Shen et al., 2014), in which the query and documents are ﬁrst embedded into continuous vectors and the ranking is calculated based on their embeddings’ similarity.

Original: The interaction focused neural models (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016a), on the other hand, learn query-document matching patterns from word-level interactions.
Revised: The interaction focused neural models (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016a), on the other hand, learn query-document matching patterns from word-level interactions.

Original: Both the interaction and representation focused models can be combined for further improvements (Mitra et al., 2017).
Revised: Both the interaction and representation focused models can be combined for further improvements (Mitra et al., 2017).

Original: Similarly, Jaech et al.
Revised: Similarly, Jaech et al.

Original: (2017) captures both local relevance matching and global topicality signals when computing relevance of a document to a query.
Revised: (2017) captures both local relevance matching and global topicality signals when computing relevance of a document to a query.

Original: Our work falls into the representation focused approach to form query and document representations and jointly models the two tasks through session representation learning.
Revised: Our work falls into the representation focused approach to form query and document representations and jointly models the two tasks through session representation learning.

Original: Query Suggestion.
Revised: Query Suggestion.

Original: In general, query suggestion algorithms deﬁne various distance metrics between queries to ﬁnd the most similar ones as suggestions for one another (Wen et al., 2001; Baeza-Yates et al., 2004).
Revised: In general, query suggestion algorithms deﬁne various distance metrics between queries to ﬁnd the most similar ones as suggestions for one another (Wen et al., 2001; Baeza-Yates et al., 2004).

Original: The closest work to ours is the end-to-end hierarchical recurrent encoder-decoder architecture (HRED-qs)
Revised: The closest work to ours is the end-to-end hierarchical recurrent encoder-decoder architecture (HRED-qs)

Original: (Sordoni et al., 2015), which ranks candidates for context-aware query suggestion.
Revised: (Sordoni et al., 2015), which ranks candidates for context-aware query suggestion.

Original: Our proposed framework differs from HRED-qs as it integrates another important type of user search behavior, i.e., result clicks, into the joint modeling, which provides additional contextual information for query suggestion.
Revised: Our proposed framework differs from HRED-qs as it integrates another important type of user search behavior, i.e., result clicks, into the joint modeling, which provides additional contextual information for query suggestion.

Original: Mitra & Craswell (2015) proposed a candidate generation approach for rare preﬁxes using frequently observed query sufﬁxes and suggested a neural model to generate ranking features along with n-gram features.
Revised: Mitra & Craswell (2015) proposed a candidate generation approach for rare preﬁxes using frequently observed query sufﬁxes and suggested a neural model to generate ranking features along with n-gram features.

Original: Similarly, a large pool of previous works investigated task or context-aware approaches (Cao et al., 2008;
Revised: Similarly, a large pool of previous works investigated task or context-aware approaches (Cao et al., 2008;

Original: He et al., 2009; Feild & Allan, 2013;
Revised: He et al., 2009; Feild & Allan, 2013;

Original: Chen et al., 2017; Garigliotti & Balog, 2017) for query suggestion.
Revised: Chen et al., 2017; Garigliotti & Balog, 2017) for query suggestion.

Original: Multi-task Learning.
Revised: Multi-task Learning.

Original: The goal of multi-task learning is to improve generalization on the target task by leveraging the domain-speciﬁc information contained in the training signals of related tasks (Caruana, 1998).
Revised: The goal of multi-task learning is to improve generalization on the target task by leveraging the domain-speciﬁc information contained in the training signals of related tasks (Caruana, 1998).

Original: Multi-task learning in combination with deep neural networks has been successfully used in many application scenarios, including natural language processing (Collobert & Weston, 2008; Liu et al., 2016a;b; Peng & Dredze, 2017; Peng et al., 2017), speech recognition (Deng et al., 2013; Thanda & Venkatesan, 2017) and computer vision (Girshick, 2015).
Revised: Multi-task learning in combination with deep neural networks has been successfully used in many application scenarios, including natural language processing (Collobert & Weston, 2008; Liu et al., 2016a;b; Peng & Dredze, 2017; Peng et al., 2017), speech recognition (Deng et al., 2013; Thanda & Venkatesan, 2017) and computer vision (Girshick, 2015).

Original: However, it has been less explored in the information retrieval domain.
Revised: However, it has been less explored in the information retrieval domain.

Original: Liu et al. (2015) proposed a multi-task deep neural approach to combine query classiﬁcation and document ranking, and reported improvement on both the tasks.
Revised: Liu et al. (2015) proposed a multi-task deep neural approach to combine query classiﬁcation and document ranking, and reported improvement on both the tasks.

Original: Bai et al.
Revised: Bai et al.

Original: (2009) used multi-task learning in learning to rank for web search.
Revised: (2009) used multi-task learning in learning to rank for web search.

Original: We propose to jointly learn document ranking and query suggestion via multi-task learning to better capture latent intent embedded in users’ search behaviors.
Revised: We propose to jointly learn document ranking and query suggestion via multi-task learning to better capture latent intent embedded in users’ search behaviors.

Original:  M ULTI - TASK N EURAL S ESSION R ELEVANCE F RAMEWORK
Revised:  M ULTI - TASK N EURAL S ESSION R ELEVANCE F RAMEWORK

Original: We deﬁne a session to be a sequence of queries, Q = { Q 1 , ..., Q n } , submitted by a user in a short time interval in a chronological order to satisfy a speciﬁc search intent.
Revised: We deﬁne a session to be a sequence of queries, Q = { Q 1 , . . . , Q n } , submitted by a user in a short time interval in a chronological order to satisfy a speciﬁc search intent.

Original: Every query Q i in a session is associated with a set of related documents D
Revised: Every query Q i in a session is associated with a set of related documents D

Original: = { D 1 , ..., D m } which need to be ranked by its relevance to the query, and o = { o 1 , ..., o m } is the set of relevance labels for each document in D .
Revised: = { D 1 , . . . D m } which need to be ranked by its relevance to the query, and o = { o 1 , . . , o m } is the set of relevance labels for each document in D .

Original: In typical search engine logs, o i is usually approximated via user clicks, e.g.,clicked documents’ o i = 1 , otherwise 0.
Revised: In typical search engine logs, o j is usually approximated via user clicks, e.g., o j = 1 if and only if the document D j is clicked.

Original: A query Q i and a document D j consist of a sequence of words, i.e., Q i = { w 1 i , ..., w qi } and D j = { w 1 j , ..., w dj } , where q = | Q i | and d = | D j | are the query and document lengths respectively.
Revised: A query Q i and a document D j consist of a sequence of words, i.e., Q i = { w 1 i , . . . , w qi } and D j = { w 1 j , . . . , w dj } , where q = | Q i | and d = | D j | are the query and document lengths respectively.

Original: V is the size of vocabulary constructed over queries and relevant documents.
Revised: V is the size of vocabulary constructed over queries and relevant documents.

Original: The document ranking refers to the task of ordering the retrieved results with respect to their relevance to the given query under the users’ search intent.
Revised: The document ranking refers to the task of ordering the retrieved results with respect to their relevance to the given query under the users’ search intent.

Original: Accurate modeling of relevance between a document and a query is the key in this task.
Revised: Accurate modeling of relevance between a document and a query is the key in this task.

Original: In this work, we model ranking of related documents as a pointwise classiﬁcation problem where the goal is to predict the relevance label between a query and a document.
Revised: In this work, we model ranking of related documents as a pointwise classiﬁcation problem where the goal is to predict the relevance label between a query and a document.

Original: But our developed solution can be extended to pairwise or list-wise ranking models (Liu et al., 2009).
Revised: But our developed solution can be extended to pairwise or list-wise ranking models (Liu et al., 2009).

Original: On the other hand, query suggestion refers to the task of predicting next query
Revised: On the other hand, query suggestion refers to the task of predicting next query

Original: Q t based on previous queries Q 1 . . .
Revised: Q i based on previous queries Q 1 . . .

Original: Q t − 1 by users in the same session, so as to help them explore the search space.
Revised: Q i − 1 by users in the same session, so as to help them explore the search space.

Original: The key challenge is to maintain the semantic consistency between the suggested queries and users’ original queries, with respect to their search intent.
Revised: The key challenge is to maintain the semantic consistency between the suggested queries and users’ original queries, with respect to their search intent.

Original: Traditional IR approaches consider document ranking and query suggestion tasks separately (Huang et al., 2013;
Revised: Traditional IR approaches consider document ranking and query suggestion tasks separately (Huang et al., 2013;

Original: Shen et al., 2014;
Revised: Shen et al., 2014;

Original: Sordoni et al., 2015), although both the tasks are driven by users’ underlying search intent.
Revised: Sordoni et al., 2015), although both the tasks are driven by users’ underlying search intent.

Original: In contrast, M-NSRF is designed to jointly learn a document ranker and a query recommender by modeling shared search context embedding in a session.
Revised: In contrast, M-NSRF is designed to jointly learn a document ranker and a query recommender by modeling shared search context embedding in a session.

Original: Based on the latent session state inferred from all the previous queries and clicks in the same session, the document ranker is trained to predict user clicks from the candidate documents for the current query and the query recommender is trained in a sequence to sequence fashion (Sutskever et al., 2014) to predict the user’s next query.
Revised: Based on the latent session state inferred from all the previous queries and clicks in the same session, the document ranker is trained to predict user clicks from the candidate documents for the current query and the query recommender is trained in a sequence to sequence fashion (Sutskever et al., 2014) to predict the user’s next query.

Original: The process is repeated sequentially for all the queries in the same session.
Revised: The process is repeated sequentially for all the queries in the same session.

Original: The detailed architecture of the proposed multi-task neural session relevance framework (M-NSRF) is provided in Appendix A.
Revised: The detailed architecture of the proposed multi-task neural session relevance framework (M-NSRF) is provided in Appendix A.

Original: In the following, we discuss each component of M-NSRF.
Revised: In the following, we discuss each component of M-NSRF.

Original: 3. D OCUMENT R ANKER
Revised: 3. D OCUMENT R ANKER

Original: Ranking the retrieved documents for an input query requires encoding the query and documents into a shared representation space.
Revised: Ranking the retrieved documents for an input query requires encoding the query and documents into a shared representation space.

Original: In addition to the search intent carried by the current query, search context, which is reﬂected in the previously submitted queries in the same session, should also be accounted in ranking the documents.
Revised: In addition to the search intent carried by the current query, search context, which is reﬂected in the previously submitted queries in the same session, should also be accounted in ranking the documents.

Original: In M-NSRF, we model the latent user search intent in a sequence of queries via a series of session recurrent states.
Revised: In M-NSRF, we model the latent user search intent in a sequence of queries via a series of session recurrent states.

Original: As a result, the document ranker component in M-NSRF consists of a query encoder, a document encoder, a session encoder and a ranker.
Revised: As a result, the document ranker component in M-NSRF consists of a query encoder, a document encoder, a session encoder and a ranker.

Original: The ranker sub-component combines the latent representations of query and session and match them with document representations to generate the ranking score of documents, based on which the documents are ordered.
Revised: The ranker sub-component combines the latent representations of query and session and match them with document representations to generate the ranking score of documents, based on which the documents are ordered.

Original: The technical details of each constituent element are given as follows.
Revised: The technical details of each constituent element are given as follows.

Original: Query Encoder .
Revised: Query Encoder .

Original: The query encoder encodes a query into a vector.
Revised: The query encoder encodes a query into a vector.

Original: To encode a sequence of words, various approaches have been studied.
Revised: To encode a sequence of words, various approaches have been studied.

Original: We follow (Conneau et al., 2017) to adopt a bidirectional
Revised: We follow (Conneau et al., 2017) to adopt a bidirectional

Original: LSTM with max pooling (BiLSTM-max), due to its superior practical performance.
Revised: LSTM with max pooling (BiLSTM-max), due to its superior practical performance.

Original: Considering query as a sequence of words Q i = { w [-
-] i 
Revised: Considering query as a sequence of words Q i = { w [-
-]1 i , . . .

Original: ..., w qi } , the encoder composed of forward and backward
Revised: , w qi } , the encoder composed of forward and backward

Original: LSTM reads the sequence in two opposite directions,
Revised: LSTM reads the sequence in two opposite directions,

Original: [Equation]
Revised: [Equation]

Original: where h t ∈ R 2 d is the query-level recurrent state, d is the dimensionality of the LSTM hidden unit initialized to a zero vector.
Revised: where h t ∈ R 2 d is the query-level recurrent state, d is the dimensionality of the LSTM hidden unit initialized to a zero vector.

Original: To form a ﬁxed-size vector representation of variable length queries, maximum value is selected over each dimension of the hidden units,
Revised: To form a ﬁxed-size vector representation of variable length queries, maximum value is selected over each dimension of the hidden units,

Original: [Equation]
Revised: [Equation]

Original: where Q i,k is the k -th element of the latent vector Q i .
Revised: where Q i,k is the k -th element of the latent vector Q i .

Original: Document Encoder .
Revised: Document Encoder .

Original: The goal of the document encoder is to encode documents into continuous vectors.
Revised: The goal of the document encoder is to encode documents into continuous vectors.

Original: We use the same BiLSTM-max technique that is utilized in encoding queries as the document encoder.
Revised: We use the same BiLSTM-max technique that is utilized in encoding queries as the document encoder.

Original: The only difference is in the dimensionality of the LSTM hidden units.
Revised: The only difference is in the dimensionality of the LSTM hidden units.

Original: In general, because a document (body or title) is longer than a query, we consider dense vector of a larger size as the continuous representations of the documents.
Revised: In general, because a document (body or title) is longer than a query, we consider dense vector of a larger size as the continuous representations of the documents.

Original: Session Encoder .
Revised: Session Encoder .

Original: The session encoder generates a representation to encode the queries that the system has received so far.
Revised: The session encoder generates a representation to encode the queries that the system has received so far.

Original: Unlike query and document encoders that can read the complete encoded queries and documents, the session encoder does not have information from the future queries.
Revised: Unlike query and document encoders that can read the complete encoded queries and documents, the session encoder does not have information from the future queries.

Original: Therefore, we use a unidirectional LSTM (Hochreiter & Schmidhuber, 1997) for session encoding.
Revised: Therefore, we use a unidirectional LSTM (Hochreiter & Schmidhuber, 1997) for session encoding.

Original: The session encoder takes the sequence of query representations Q 1 , ..., Q n as input and computes the sequence of session-level recurrent states.
Revised: The session encoder takes the sequence of query representations Q 1 , ..., Q n as input and computes the sequence of session-level recurrent states.

Original: [Equation]
Revised: [Equation]

Original: where S t ∈ R
Revised: where S i ∈ R

Original: d is the session-level recurrent state initialized to a zero vector.
Revised: d is the session-level recurrent state initialized to a zero vector.

Original: As a result, each query in the session has its session-level recurrent state, summarizing the user’s information need that has been processed up to query t .
Revised: As a result, each query in the session has its session-level recurrent state, summarizing the user’s information need that has been processed up to query Q i .

Original: Ranker .
Revised: Ranker .

Original: We ﬁrst concatenate the current query representation Q t with previous session-level recurrent state S t − 1 via a non-linear transformation.
Revised: We ﬁrst concatenate the current query representation Q i with previous session-level recurrent state S i − 1 via a non-linear transformation.

Original: This combined representation reﬂects the search intent reﬂected in the current query and the past ones in the same session.
Revised: This combined representation reﬂects the search intent reﬂected in the current query and the past ones in the same session.

Original: Then we compute the ranking score of document D i under the query as its probability of being relevant via a sigmoid function (with binary relevance labels),
Revised: Then we compute the ranking score of document D j under the query as its probability of being relevant via a sigmoid function (with binary relevance labels),

Original: [Equation]
Revised: [Equation]

Original: where W r ∈ R ( d q + d s ) × d d , b r ∈ R d d , and d q , d s and d d are the dimensionality of the query encoder, session encoder and document encoder hidden units and σ is the sigmoid function.
Revised: where W r ∈ R ( d q + d s ) × d d , b r ∈ R d d , and d q , d s and d d are the dimensionality of the query encoder, session encoder and document encoder hidden units and σ is the sigmoid function.

Original: A list of retrieved documents can therefore be ordered by this ranking score.
Revised: A list of retrieved documents can therefore be ordered by this ranking score.

Original: 3. Q UERY R ECOMMENDER
Revised: 3. Q UERY R ECOMMENDER

Original: Query recommender suggests related queries to users by inferring the underlying search intent through utilizing in-session previous queries embedded in latent vectors.
Revised: Query recommender suggests related queries to users by inferring the underlying search intent through utilizing in-session previous queries embedded in latent vectors.

Original: Following Sutskever et al.
Revised: Following Sutskever et al.

Original: (2014) and Bahdanau et al.
Revised: (2014) and Bahdanau et al.

Original: , the query recommender in M-NSRF predicts users’ next query in a sequence to sequence manner.
Revised: , the query recommender in M-NSRF predicts users’ next query in a sequence to sequence manner.

Original: Basically the query recommender module estimates the probability of the next query Q t = { w 1  ,..., w q } , given all the previous queries up to position t − 1 in a session as follows,
Revised: Basically the query recommender module estimates the probability of the next query Q i = { w 1 i , . . . , w qi } , given all the previous queries up to position i − 1 in a session as follows,

Original: [Equation]
Revised: [Equation]

Original: We use LSTM as a basic building block for the query recommender.
Revised: We use LSTM as a basic building block for the query recommender.

Original: Information about all the previous queries represented through a session vector S t is passed to the query recommender.
Revised: Information about all the previous queries represented through a session vector S i is passed to the query recommender.

Original: To this end, the recurrent state of the query recommender is initialized with a non-linear transformation of S t , h 0 =
Revised: To this end, the recurrent state of the query recommender is initialized with a non-linear transformation of S i , h 0 =

Original: tanh( W q S t + b q ) , where h 0 ∈ R
Revised: tanh( W q S i + b q ) , where h 0 ∈ R

Original: d is the initial recurrent state.
Revised: d is the initial recurrent state.

Original: Then the query recommender’s recurrence is computed by h t =
Revised: Then the query recommender’s recurrence is computed by h t =

Original: LSTM t ( h t − 1 , w t − 1 ) , where h t − 1 is the previous hidden state, w t − 1  is the previous query term.
Revised: LSTM t ( h t − 1 , w t i ) , where h t − 1 is the previous hidden state, w t − 1 i is the previous query term.

Original: Finally, each recurrent state is mapped to a probability distribution over the vocabulary of size
Revised: Finally, each recurrent state is mapped to a probability distribution over the vocabulary of size

Original: V using a combination of linear transformation and  softmax function.
Revised: V using a combination of linear transformation and the softmax function.

Original: Word with the highest probability is chosen as the next word in sequence.
Revised: Word with the highest probability is chosen as the next word in sequence.

Original: [Equation]
Revised: [Equation]

Original: where w t = arg max w P ( w | w 1: t − 1 , Q 1: n − 1 ) and g ( z ) j = e zj (cid:80) Kk =1 e zk , j = 1 , ..., K . This formulation has shown to be beneﬁcial for language modelling tasks (Mikolov et al., 2010).
Revised: where g is the softmax function that outputs a vector to represent the distribution of next words, where the probability assigned to the j -th element is deﬁned as g ( z ) j = [Equation]

Original: Query Suggestion .
Revised: Query Suggestion .

Original: We consider the query suggestion task as a sequence inference problem (Sordoni et al., 2015).
Revised: In the decoding phase, similar to (Sordoni et al., 2015), we use greedy decoding algorithm for suggesting next query.

Original: Given a sequence of queries up to position t − 1 , a suggested query Q t is:
Revised: Given a sequence of queries up to position i − 1 , a suggested query Q i is:

Original: [Equation]
Revised: [Equation]

Original: where Q is the space of all possible queries.
Revised: where Q is the space of all possible queries.

Original: To generate  query suggestions of variable lengths, we use standard word-level decoding techniques (Cho et al., 2014).
Revised: To generate Q ∗ = { w 1 , . . . , w q } , we use a greedy approach like in (Sordoni et al., 2015) where w t = arg max w P ( w | w 1: t − 1 , Q 1: i − 1 ) To provide query suggestions of variable lengths, we use standard word-level decoding techniques.

Original: We iteratively consider the best preﬁx w 1: m up to length m and extend it by sampling the most probable word given the distribution in Eq.
Revised: We iteratively consider the best preﬁx w 1: t up to length t and extend it by sampling the most probable word given the distribution in Eq.

Original: The process ends when we obtain a well-formed query containing the special end-ofquery token.
Revised: The process ends when we obtain a well-formed query containing the special end-of-query token.

Original: 3. L EARNING E ND - TO -E ND
Revised: 3. L EARNING E ND - TO -E ND

Original: Within a session, M-NSRF ranks documents in a set of candidates and predicts next query given the current query sequence.
Revised: Within a session, M-NSRF ranks documents in a set of candidates and predicts next query given the current query sequence.

Original: Therefore, the training objective of M-NSRF consists of two terms.
Revised: Therefore, the training objective of M-NSRF consists of two terms.

Original: The ﬁrst term is the binary cross entropy loss from the document ranker,
Revised: The ﬁrst term is the binary cross entropy loss from the document ranker,

Original: [Equation]
Revised: [Equation]

Original:  wheret i and o i represents binary click labeland click probability forthe i -th document, i.e., P ( D i | Q t ) deﬁned in Eq.
Revised: )) where o j represents binary click label for D j .

Original: The second term is the regularized negative log-likelihood loss from the query suggestion model,
Revised: The second term is the regularized negative loglikelihood loss from the query suggestion model,

Original: [Equation]
Revised: [Equation]

Original: where L R ≡ − λ (cid:80) w ∈ V P ( w | w 1: i − 1  , Q 1: t − 1 ) log P ( w | w 1: i − 1  , Q 1: t − 1 ) is the regularization term to avoid the distribution of words in Eq.
Revised: where L R ≡ − λ (cid:80) w ∈ V P ( w | w 1: t − 1 i , Q 1: i − 1 ) log P ( w | w 1: t − 1 i , Q 1: i − 1 ) is the regularization term to avoid the distribution of words in Eq.

Original: (2) from being highly skewed, and λ is a hyper-parameter to control the regularization term.
Revised: (2) from being highly skewed, and λ is a hyper-parameter to control the regularization term.

Original: The ﬁnal objective is the summation of L 1 and L 2 over all the queries.
Revised: The ﬁnal objective is the summation of L 1 and L 2 over all the queries.

Original: Note that the document ranker and query recommender share the same document, query, and session encoders, and the training of M-NSRF can be done in an online manner using the following procedure.
Revised: Note that the document ranker and query recommender share the same document, query, and session encoders, and the training of M-NSRF can be done in an online manner using the following procedure.

Original: In the forward pass, M-NSRF computes the query and corresponding document encodings, updates session-level recurrent states, click probability for each candidate document and the loglikelihood of each query in the session given the previous ones.
Revised: In the forward pass, M-NSRF computes the query and corresponding document encodings, updates session-level recurrent states, click probability for each candidate document and the log-likelihood of each query in the session given the previous ones.

Original: In the backward pass, the gradients are computed and the parameters are updated based on  ADAM update rule (Kingma & Ba, 2014).
Revised: In the backward pass, the gradients are computed and the parameters are updated based on the ADAM update rule (Kingma & Ba, 2014).

Original: Details of implementation can be found in Section 4.
Revised: Details of implementation can be found in Section 4.

Original: 3. G ENERALIZING M-NSRF FOR N EURAL IR M ODELS
Revised: G ENERALIZING M-NSRF FOR N EURAL IR M ODELS

Original: The proposed multi-task learning framework is general and can be applied to combine different types of document rankers and query recommenders.
Revised: The proposed multi-task learning framework is general and the query encoder, document encoder, document ranker, and query recommender can be replaced by other designed architecture.

Original: For example, as mentioned previously, our query suggestion component is close to the model proposed in (Sordoni et al., 2015) with a multitask learning framework embedded in.

Original: Hence, any neural query suggestion model working in the sequence to sequence fashion can be readily extended to our proposed multi-task learning architecture.
Revised: In general, any neural query suggestion model working in the sequence to sequence fashion can be readily incorporated with the proposed multi-task learning framework.

Original: On the other hand, mostof the neural IR models are built on the notion of learning query and document representations in a latent space for relevance modeling.
Revised: Similarly, most neural IR models that built on the notion of learning query and document representations in a latent space for relevance modeling can be Incorporated in our framework as well.

Original: Incorporating such models into our proposed multi-task learning framework is straightforward; but due to distinctive nature of different neural IR models, careful study is required while adding context-awareness into the ﬁnal architecture.
Revised: However, due to distinctive nature of different neural IR models, careful study is required while adding context-awareness into the ﬁnal architecture.

Original: In this paper, we take the Match-Tensor model (Jaech et al., 2017), a recently proposed neural relevance model for document ranking as an example and extend it to a multi-task Match-Tensor (M-Match-Tensor) model by incorporating the query recommender component within our proposed multi-task learning framework.
Revised: In this paper, we take the Match-Tensor model (Jaech et al., 2017), a recently proposed neural relevance model for document ranking as an example and extend it to a multi-task Match-Tensor (M-Match-Tensor) model by incorporating the query recommender component within our proposed multi-task learning framework.

Revised: Different from NSRF, which embeds queries and documents into vectors, Match-Tensor learns a contextual representation for each word in the queries and documents.

Revised: Therefore, documents and queries are represented as matrices.

Original: The document ranker in Match-Tensor works as follows,
Revised: The document ranker in Match-Tensor then computes the relevant score based on the following formulation:

Original: [Equation]
Revised: [Equation]

Original: where C represents a sequence of convolutional operation (Lawrence et al., 1997) and max-pooling on the product of query and document vectors.
Revised: where C represents a sequence of convolutional operation (Lawrence et al., 1997) and max-pooling on the product of query and document vectors.

Original: The detailed architecture of the M-Match-Tensor model (M-Match-Tensor) is provided in Appendix B.
Revised: The detailed architecture of the M-Match-Tensor model (M-Match-Tensor) is provided in Appendix B.

Original: To keep M-Match-Tensor model simple, we do not involvesession recurrence in the document ranker component. We bring the query-level recurrent states produced by bi-LSTM following a linear projection into original Match-Tensor model and feed them to the session-level encoder.
Revised: To incorporate the match-tensor in our mulittask learning framework, we can replace the document encoder, query encoder, and the document ranker in M-NSRF with the ones speciﬁed in the Match-Tensor model. However, as the computations involve in the document ranker is substantial, we do not increase the computational complexity further by adding session recurrence in the document ranking. The query decoding component in

Original:  Inthis way, we follow the same relevance computation process as in the original Match-Tensor model but jointly train M-Match-Tensor model on document ranking and query suggestion.
Revised: M-Match-Tensor is identical to M-NSRF. In the experiment, we will show that multi-task MatchTensor achieves better performance than Match-Tensor in document ranking task.

Original:  E XPERIMENTS 4. D ATA S ETS AND E VALUATION M ETHODOLOGY
Revised:  E XPERIMENTS 4. D ATA S ETS AND E VALUATION M ETHODOLOGY

Original: We conduct our experiments on the publicly available AOL search log (Pass et al., 2006).
Revised: We conduct our experiments on the publicly available AOL search log (Pass et al., 2006).

Original: The queries in this dataset were sampled between 1 March, 2006 and 31 May, 2006.
Revised: The queries in this dataset were sampled between 1 March, 2006 and 31 May, 2006.

Original: In total there are 16,946,938 queries submitted by 657,426 unique users.
Revised: In total there are 16,946,938 queries submitted by 657,426 unique users.

Original: We removed all non-alphanumeric characters from the queries, applied word segmentation and lowercasing.
Revised: We removed all non-alphanumeric characters from the queries, applied word segmentation and lowercasing.

Original: We followed (Jansen Bernard et al., 2007) to deﬁne a session by a 30-minute window of inactive time, and ﬁltered sessions by their lengths (minimum 2, maximum 10).
Revised: We followed (Jansen Bernard et al., 2007) to deﬁne a session by a 30-minute window of inactive time, and ﬁltered sessions by their lengths (minimum 2, maximum 10).

Original: We only kept the most frequent | V | = 100 k words and mapped all other words to an < unk > token when constructing the vocabulary.
Revised: We only kept the most frequent | V | = 100 k words and mapped all other words to an < unk > token when constructing the vocabulary.

Original: We randomly selected 1,032,459 sessions for training, 129,053 sessions for development and 91,108 sessions for testing, with no overlapping.
Revised: We randomly selected 1,032,459 sessions for training, 129,053 sessions for development and 91,108 sessions for testing, with no overlapping.

Original: In total, there are 2,987,486 queries for training, 287,138 for development, and 259,117 for testing.
Revised: In total, there are 2,987,486 queries for training, 287,138 for development, and 259,117 for testing.

Original: The average length of the queries and documents (only the title ﬁeld) are 3. and 6.77 respectively.
Revised: The average length of the queries and documents (only the title ﬁeld) are 3. and 6.77 respectively.

Original: In our experiments, we set maximum allowable length of query and document to 10 and 20 respectively.
Revised: In our experiments, we set maximum allowable length of query and document to 10 and 20 respectively.

Original: In the document ranking task, we need to rank the most relevant (e.g., most clickable) document on top.
Revised: In the document ranking task, we need to rank the most relevant (e.g., most clickable) document on top.

Original: We used three standard ranking metrics, mean average precision (MAP), mean reciprocal rank (MRR) and normalized discounted cumulative gain (NDCG) metric computed at positions one, three, ﬁve and ten, to measure the performance.
Revised: We used three standard ranking metrics, mean average precision (MAP), mean reciprocal rank (MRR) and normalized discounted cumulative gain (NDCG) metric computed at positions one, three, ﬁve and ten, to measure the performance.

Original: Since AOL search log only contains clicked documents, we constructed the ranking candidates by the top ranked documents by BM25 (Robertson et al., 2009).
Revised: Since AOL search log only contains clicked documents, we constructed the ranking candidates by the top ranked documents by BM25 (Robertson et al., 2009).

Original: Each query in the test set consists of 50 candidate documents including the clicked ones.
Revised: Each query in the test set consists of 50 candidate documents including the clicked ones.

Original: However, to reduce the training time and memory use, each query in training and development set only contains 5 candidates.
Revised: However, to reduce the training time and memory use, each query in training and development set only contains 5 candidates.

Original: In the query suggestion task, we need to suggest the most semantically related query to users.
Revised: In the query suggestion task, we need to suggest the most semantically related query to users.

Original: As we do not have user feedback on the suggested queries, we treated the users’ next submitted query as ground-truth (Sordoni et al., 2015), and used the BLEU scores (Papineni et al., 2002) as the evaluation metric, which is a popularly used metric in machine translation and ext generation tasks.
Revised: As we do not have user feedback on the suggested queries, we treated the users’ next submitted query as ground-truth (Sordoni et al., 2015), and used the BLEU scores (Papineni et al., 2002) as the evaluation metric, which is a popularly used metric in machine translation and ext generation tasks.

Original: In addition, following (Santos et al., 2013; Sordoni et al., 2015), we evaluated the query suggestion quality by mean reciprocal rank (MRR), i.e., to test if the algorithm can rank the users’ next query on top of its recommendation list.
Revised: In addition, following (Santos et al., 2013; Sordoni et al., 2015), we evaluated the query suggestion quality by mean reciprocal rank (MRR), i.e., to test if the algorithm can rank the users’ next query on top of its recommendation list.

Original: In this set of experiments, given a set of candidate queries, query suggestion models give a likelihood score of generating the candidates.
Revised: In this set of experiments, given a set of candidate queries, query suggestion models give a likelihood score of generating the candidates.

Original: We follow Sordoni et al.
Revised: We follow Sordoni et al.

Original: (2015) for dataset split and generate candidates using a co-occurrence based suggestion model.
Revised: (2015) for dataset split and generate candidates using a co-occurrence based suggestion model.

Original: Like
Revised: Like

Original: Sordoni et al.
Revised: Sordoni et al.

Original: , we give the anchor queries (second last query of a session) as an input to the query suggestion model and evaluates the rank of the next query among the candidates.
Revised: , we give the anchor queries (second last query of a session) as an input to the query suggestion model and evaluates the rank of the next query among the candidates.

Original: 4. B ASELINES AND I MPLEMENTATION D ETAILS
Revised: 4. B ASELINES AND I MPLEMENTATION D ETAILS

Original: Document Ranking Baselines .
Revised: Document Ranking Baselines .

Original: We compared M-NSRF and M-Match Tensor with word-based baselines and neural network-based baselines.
Revised: We compared M-NSRF and M-Match Tensor with word-based baselines and neural network-based baselines.

Original: Word-based baselines include query likelihood model 
Revised: Word-based baselines include query likelihood model based on Dirichlet smoothing (QL)

Revised: (Ponte & Croft, 1998) and BM25 (Robertson et al., 2009).

Revised: In addition, following (Mitra et al., 2016), we investigated the ranking performance of a simple word

Original: [Table]
Revised: [Table]

Original: based on Dirichlet smoothing (QL) (Ponte & Croft, 1998) and BM25 (Robertson et al., 2009).
Revised: embedding

Original: In addition, following (Mitra et al., 2016), we investigated the ranking performance of a simple word embedding-based model using GloVe word embeddings (Pennington et al., 2014).
Revised: [Table] based model using GloVe word embeddings [Table]

Original:  To compare with neural models, we consider baselines broadly categorized in representation-focused, interactionfocused and a combination of both.
Revised: Pennington et [Table] To compare with neural models, we consider baselines broadly categorized in representation-focused, interactionfocused and a combination of both.

Original: Representation-focused neural baselines include: DSSM (Huang et al., 2013), CLSM (Shen et al., 2014), ARC-I (Hu et al., 2014) and interaction-focused baselines include: ARC-II (Hu et al., 2014) and DRMM (Guo et al., 2016a).
Revised: Representation-focused neural baselines include: DSSM (Huang et al., 2013), CLSM (Shen et al., 2014), ARC-I (Hu et al., 2014) and interaction-focused baselines include: ARC-II (Hu et al., 2014) and DRMM (Guo et al., 2016a).

Original: A combination of representation and interaction focused models include: DUET (Mitra et al., 2017) and Match Tensor (Jaech et al., 2017).
Revised: A combination of representation and interaction focused models include: DUET (Mitra et al., 2017) and Match Tensor (Jaech et al., 2017).

Original: Details of these models are provided in Appendix C. We implemented all the baseline models in PyTorch.
Revised: Details of these models are provided in Appendix C. We implemented all the baseline models in PyTorch.

Original: Query Suggestion Baselines .
Revised: Query Suggestion Baselines .

Original: To evaluate the performance on query suggestion task, we consider three baseline methods including Seq2seq model proposed by Bahdanau et al.
Revised: To evaluate the performance on query suggestion task, we consider three baseline methods including Seq2seq model proposed by Bahdanau et al.

Original: (2015), Seq2seq with global attention mechanism (Luong et al., 2015) and HRED-qs
Revised: (2015), Seq2seq with global attention mechanism (Luong et al., 2015) and HRED-qs

Original: (Sordoni et al., 2015).
Revised: (Sordoni et al., 2015).

Original: Details of these models are provided in appendix D. We implemented all three baselines in PyTorch and optimized using negative log-likelihood loss as in Eq.
Revised: Details of these models are provided in appendix D. We implemented all three baselines in PyTorch and optimized using negative log-likelihood loss as in Eq.

Original: Implementation Details of M-NSRF .
Revised: Implementation Details of M-NSRF .

Original: The model was trained end-to-end and we used mini-batch
Revised: The model was trained end-to-end and we used mini-batch

Original: SGD with Adam (Kingma & Ba, 2014) for optimization.
Revised: SGD with Adam (Kingma & Ba, 2014) for optimization.

Original: with the two momentum parameters set to 0.9 and 0.999 respectively.
Revised: with the two momentum parameters set to 0.9 and 0.999 respectively.

Original: We use 300-dimensional word vectors trained with GloVe (Pennington et al., 2014) on 840 billion of tokens to initialize the word embeddings.
Revised: We use 300-dimensional word vectors trained with GloVe (Pennington et al., 2014) on 840 billion of tokens to initialize the word embeddings.

Original: Out-of-vocabulary words were randomly initialized by sampling values from a zero-mean unit-variance normal distribution.
Revised: Out-of-vocabulary words were randomly initialized by sampling values from a zero-mean unit-variance normal distribution.

Original: All training used a mini-batch size of 32 to ﬁt in single GPU memory.
Revised: All training used a mini-batch size of 32 to ﬁt in single GPU memory.

Original: Learning rate was ﬁxed to 0.001.
Revised: Learning rate was ﬁxed to 0.001.

Original: We used dropout (0.20) (Srivastava et al., 2014) and early stopping with a patience of 5 epochs for regularization.
Revised: We used dropout (0.20) (Srivastava et al., 2014) and early stopping with a patience of 5 epochs for regularization.

Original: We set λ = 0.1 for entropy regularization in Eq.
Revised: We set λ = 0.1 for entropy regularization in Eq.

Original: M-NSRF is implemented in
Revised: M-NSRF is implemented in

Original: PyTorch and it runs on a single GPU (TITAN X) with roughly a runtime of 90 minutes per epoch.
Revised: PyTorch and it runs on a single GPU (TITAN X) with roughly a runtime of 90 minutes per epoch.

Original: In general, M-NSRF runs up to 20 epochs and we select the model that achieves the minimum loss on the development set.
Revised: In general, M-NSRF runs up to 20 epochs and we select the model that achieves the minimum loss on the development set.

Original: 4. E VALUATION R ESULTS
Revised: 4. E VALUATION R ESULTS

Original: Document Ranking Quality .
Revised: Document Ranking Quality .

Original: Table 1 shows the performance of NSRF, M-NSRF, M-Match-Tensor and other baseline models.
Revised: Table 1 shows the performance of NSRF, M-NSRF, M-Match-Tensor and other baseline models.

Original: NSRF signiﬁcantly outperforms all the baselines except the MatchTensor model, and the model size of NSRF is much smaller than Match-Tensor.
Revised: NSRF signiﬁcantly outperforms all the baselines except the MatchTensor model, and the model size of NSRF is much smaller than Match-Tensor.

Original: With multi-task learning, both M-NSRF and M-Match-Tensor outperform NSRF and Match-Tensor, respectively.
Revised: With multi-task learning, both M-NSRF and M-Match-Tensor outperform NSRF and Match-Tensor, respectively.

Original: To study the advantage of multi-task learning on our proposed approach, we trained M-NSRF only on document ranking task (noted as NSRF in Table 1) and observed signiﬁcant performance drop which endorses the mutual beneﬁt of joint learning of these two tasks via multi-task learning.
Revised: To study the advantage of multi-task learning on our proposed approach, we trained M-NSRF only on document ranking task (noted as NSRF in Table 1) and observed signiﬁcant performance drop which endorses the mutual beneﬁt of joint learning of these two tasks via multi-task learning.

Original: Existing neural ranking models, like DRMM and DUET architecture, achieved sub-optimal performance in our experiments.
Revised: Existing neural ranking models, like DRMM and DUET architecture, achieved sub-optimal performance in our experiments.

Original: We believe because of the simple architecture of DRMM with few hundreds of parameters, the model fell behind to show competitive performance on the evaluation 
Revised: We believe because of the simple architecture of DRMM with few hundreds of parameters, the model fell behind to show competitive performance on the evaluation dataset.

Original: [Table]
Revised: On the other hand, we believe that the use of smaller number of top character n -graphs (in our case, 5000) by the DUET architecture limits its effectiveness in modeling representation and

Original: [Table]
Revised: [Table]

Original: dataset.

Original: On the other hand

Original: [Table]
Revised: [Table]

Original: we believe that the use of smaller number of top character n -graphs (in our case, 5000) by the DUET architecture limits its effectiveness in modeling representation and interaction focused features to compute matching quality between query and document.
Revised: interaction focused features to compute matching quality between query and document.

Original: We have to note that, some negative results have been reported for document title-based ad-hoc retrieval tasks (Guo et al., 2016a); and thus in our future work, we plan to investigate M-NSRF and all baseline models’ performance with document body content considered [Table]
Revised: We have to note that, some negative results have been reported for document title-based ad-hoc retrieval tasks (Guo et al., 2016a); and thus in our future work, we plan to investigate M-NSRF and all baseline models’ performance with document body content considered.

Original: Query Suggestion Accuracy .
Revised: Query Suggestion Accuracy .

Original: Examples of the predicted queries by M-NSRF given preceding queries from the same session is presented in Table 2 (more examples are provided in Appendix
Revised: Examples of the predicted queries by M-NSRF given preceding queries from the same session is presented in Table 2 (more examples are provided in Appendix

Original: E).
Revised: E).

Original: The quantitative comparison results in this task between our proposed framework and baseline models are presented in Table 3.
Revised: The quantitative comparison results in this task between our proposed framework and baseline models are presented in Table 3.

Original: While M-NSRF and HRED-qs consider information from all preceding queries in the same session, the other two baselines only consider the current query to predict the next one.
Revised: While M-NSRF and HRED-qs consider information from all preceding queries in the same session, the other two baselines only consider the current query to predict the next one.

Original: Table 3 shows that M-NSRF outperformed seq2seq and HRED-qs baselines in all measured BLUE scores, and MRR; but the Seq2seq with attention baseline performed better than M-NSRF in terms of BLEU-3, BLEU-4 and MRR 2 .
Revised: Table 3 shows that M-NSRF outperformed seq2seq and HRED-qs baselines in all measured BLUE scores, and MRR; but the Seq2seq with attention baseline performed better than M-NSRF in terms of BLEU-3, BLEU-4 and MRR 2 .

Original: We investigated the advantage of attention mechanism in this particular task and found it performs well when there is a considerable overlap between the input and output queries.
Revised: We investigated the advantage of attention mechanism in this particular task and found it performs well when there is a considerable overlap between the input and output queries.

Original: Speciﬁcally, in more than 25% of the test sessions, the input queries and their next queries are exactly the same; and the attention mechanism encourages the model to repeat words from the input query.
Revised: Speciﬁcally, in more than 25% of the test sessions, the input queries and their next queries are exactly the same; and the attention mechanism encourages the model to repeat words from the input query.

Original: When we restrict the experiment on test sessions that have no overlap between the input and its next query, the
Revised: When we restrict the experiment on test sessions that have no overlap between the input and its next query, the

Original: Seq2seq with attention baseline encountered signiﬁcant performance drop, while M-NSRF provided improved performance compared to all baselines.
Revised: Seq2seq with attention baseline encountered signiﬁcant performance drop, while M-NSRF provided improved performance compared to all baselines (approx.

Revised: 5%, 10% and 1% improvement over

Revised: Seq2seq, Seq2seq with attention and HRED-qs model).

Original: The major reason for this improvement is that our model leverages the global session information and is therefore not restricted by the input query and is able to generate related but totally new queries (as shown in in Appendix E).
Revised: The major reason for this improvement is that our model leverages the global session information and is therefore not restricted by the input query and is able to generate related but totally new queries (as shown in in Appendix E).

Original: We also observed that entropy-based regularization helped M-NSRF in predicting the next query, as shown in Table 3 that M-NSRF without regularization achieved worse performance in all measured
Revised: We also observed that entropy-based regularization helped M-NSRF in predicting the next query, as shown in Table 3 that M-NSRF without regularization achieved worse performance in all measured

Original: BLUE scores.
Revised: BLUE scores.

Original: We further investigated the utility of regularization for the baselines.
Revised: We further investigated the utility of regularization for the baselines.

Original: We found signiﬁcant ( ∼ 1.2%) improvement for the HRED-qs, but the performance improvement for the seq2seq and seq2seq with attention mechanism is rather marginal ( ∼ 0.2%).
Revised: We found signiﬁcant ( ∼ 1.2%) improvement for the HRED-qs, but the performance improvement for the seq2seq and seq2seq with attention mechanism is rather marginal ( ∼ 0.2%).

Original: However, the regularization technique does not introduce any signiﬁcant difference in the MRR metric.
Revised: However, the regularization technique does not introduce any signiﬁcant difference in the MRR metric.

Original: Comparing to HRED-qs, we conclude multi-task learning also helps M-NSRF achieve improved performance in query suggestion task, as the baseline employs a very similar model as ours for query [Table] [Table]
Revised: Comparing to HRED-qs, we conclude multi-task learning also helps M-NSRF achieve improved performance in query suggestion task, as the baseline employs a very similar model as ours for query suggestion.

Original: [Table] [Table] [Table] [Table]
Revised: The comparison with Seq2seq with attention suggests adding attention over session

Original: [Table]
Revised: [Table]

Original: ) suggestion. The comparison with Seq2seq with attention suggests adding attention over session information is a promising direction to further improve query suggestions quality as it emphasizes the information carried by the immediately previous query.
Revised: information is a promising direction to further improve query suggestions quality as it emphasizes the information carried by the immediately previous query.

Original: We will pursue this as our future work.
Revised: We will pursue this as our future work.

Original: 4. A BLATION S TUDY ON M-NSRF
Revised: 4. A BLATION S TUDY ON M-NSRF

Original: We conducted experiments to better understand the effectiveness of different components in the MNSRF.
Revised: We conducted experiments to better understand the effectiveness of different components in the MNSRF.

Original: We also analyzed the impact of word embeddings and hidden units dimension in M-NSRF’s performance.
Revised: We also analyzed the impact of word embeddings and hidden units dimension in M-NSRF’s performance.

Original: Our ﬁndings are presented in Table 4.
Revised: Our ﬁndings are presented in Table 4.

Original: Impact of Different Model Components .
Revised: Impact of Different Model Components .

Original: To study the effect of different model components, we compared the full M-NSRF with several simpler versions of the model.
Revised: To study the effect of different model components, we compared the full M-NSRF with several simpler versions of the model.

Original: At ﬁrst, we turned off training for word embeddings and found a signiﬁcant decrease in performance.
Revised: At ﬁrst, we turned off training for word embeddings and found a signiﬁcant decrease in performance.

Original: In our training dataset, we have roughly | OOV | = 26 k out-of-vocabulary words and thus, training the word embeddings turned out to be very important on this data set.
Revised: In our training dataset, we have roughly | OOV | = 26 k out-of-vocabulary words and thus, training the word embeddings turned out to be very important on this data set.

Original: Also, we investigated the role of pre-trained word embeddings (ex., GloVe embeddings) and found signiﬁcant performance decrease (27.9% drop in
Revised: Also, we investigated the role of pre-trained word embeddings (ex., GloVe embeddings) and found signiﬁcant performance decrease (27.9% drop in

Original: MAP) if we train M-NSRF without any pre-trained word embeddings.
Revised: MAP) if we train M-NSRF without any pre-trained word embeddings.

Original: Hence, we can conclude that the use of pre-trained word embeddings and training them further is important to achieve better performance in M-NSRF.
Revised: Hence, we can conclude that the use of pre-trained word embeddings and training them further is important to achieve better performance in M-NSRF.

Original: Second, we investigated the advantages of using max-pooling over mean-pooling and considering the last hidden recurrent state for query and document representation.
Revised: Second, we investigated the advantages of using max-pooling over mean-pooling and considering the last hidden recurrent state for query and document representation.

Original: We observed that max-pooling and mean-pooling provide almost the same performance while biLSTM-last approach lags slightly.
Revised: We observed that max-pooling and mean-pooling provide almost the same performance while biLSTM-last approach lags slightly.

Original: To further analyze the features identiﬁed by the query and document encoders using the max-pooling technique, we followed the idea of visualization proposed in (Conneau et al., 2017).
Revised: To further analyze the features identiﬁed by the query and document encoders using the max-pooling technique, we followed the idea of visualization proposed in (Conneau et al., 2017).

Original: We provide an example in Figure 2 where document 1 is clicked by the user (a positive example) and document 2, 3 is retrieved by BM25 but not clicked (negative examples).
Revised: We provide an example in Figure 2 where document 1 is clicked by the user (a positive example) and document 2, 3 is retrieved by BM25 but not clicked (negative examples).

Original: We observed that the query and document encoders identiﬁed distinct features (e.g., the word priceline in the ﬁrst document’s title is most important) which help differentiate between clicked and unclicked documents.
Revised: We observed that the query and document encoders identiﬁed distinct features (e.g., the word priceline in the ﬁrst document’s title is most important) which help differentiate between clicked and unclicked documents.

Original: In another variant of M-NSRF, we did not use the session recurrent state when computing relevance score for the candidate documents to examine the inﬂuence of previous queries from the same session on the ranking performance.
Revised: In another variant of M-NSRF, we did not use the session recurrent state when computing relevance score for the candidate documents to examine the inﬂuence of previous queries from the same session on the ranking performance.

Original: We refer to this variant of M-NSRF as multi-task neural relevance model (M-NRF).
Revised: We refer to this variant of M-NSRF as multi-task neural relevance model (M-NRF).

Original: From Table 4 we can ﬁnd that without session information the performance drops by 2.8% in terms of MAP.
Revised: From Table 4 we can ﬁnd that without session information the performance drops by 2.8% in terms of MAP.

Original: We further investigated and found the session information helps particularly for longer sessions (session length > 5 ).
Revised: We further investigated and found the session information helps particularly for longer sessions (session length > 5 ).

Original: In our evaluation dataset, we have roughlysessions of length greater than 5 (91 thousands in total).
Revised: In our evaluation dataset, we have roughlysessions of length greater than 5 (91 thousands in total).

Original: Impact of Dimensionality .
Revised: Impact of Dimensionality .

Original: We further study the impact of dimensionality of the word embeddings, query, document and session latent vectors.
Revised: We further study the impact of dimensionality of the word embeddings, query, document and session latent vectors.

Original: In M-NSRF, we set the dimension of query, document and session latent vectors to 256, 512 and 1024 respectively.
Revised: In M-NSRF, we set the dimension of query, document and session latent vectors to 256, 512 and 1024 respectively.

Original: As shown in Table 4, decreasing the dimensions of latent vectors, decreases the performance; while increasing the dimensions further, does not affect the performance signiﬁcantly.
Revised: As shown in Table 4, decreasing the dimensions of latent vectors, decreases the performance; while increasing the dimensions further, does not affect the performance signiﬁcantly.

Original: We also experimented with different dimensions of pre-trained word embeddings (50d, 100d and 200d GloVe (Pennington et al., 2014) embeddings).
Revised: We also experimented with different dimensions of pre-trained word embeddings (50d, 100d and 200d GloVe (Pennington et al., 2014) embeddings).

Original: Word embeddings of different dimensions provide different granularity of semantic similarity; with lower dimensionality, the similarity between word embeddings might be coarse and thus hard to capture matching between two text sequences.
Revised: Word embeddings of different dimensions provide different granularity of semantic similarity; with lower dimensionality, the similarity between word embeddings might be coarse and thus hard to capture matching between two text sequences.

Original: In our experiment, we found 300 dimension for embeddings works signiﬁcantly better than other dimensionality on this data set.
Revised: In our experiment, we found 300 dimension for embeddings works signiﬁcantly better than other dimensionality on this data set.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Existing deep neural models for ad-hoc retrieval often omit session information and are only trained on individual query-document pairs.
Revised: Existing deep neural models for ad-hoc retrieval often omit session information and are only trained on individual query-document pairs.

Original: In this work, we propose a context-aware multi-task neural session relevance framework which works in a sequence to sequence fashion and show that sharing session-level latent recurrent states across document ranking and query suggestion tasks beneﬁts each other.
Revised: In this work, we propose a context-aware multi-task neural session relevance framework which works in a sequence to sequence fashion and show that sharing session-level latent recurrent states across document ranking and query suggestion tasks beneﬁts each other.

Original: Our experiments and analysis not only demonstrate the effectiveness of the proposed framework, but also provide useful intuitions about the advantages of multi-task learning involving deep neural networks for two different information retrieval tasks
Revised: Our experiments and analysis not only demonstrate the effectiveness of the proposed framework, but also provide useful intuitions about the advantages of multi-task learning involving deep neural networks for two different information retrieval tasks.

Original: As our future work we would like to leverage the content from document body and click sequence to update M-NSRF (especially the session-level recurrent states) so that we can further explore the potential of the proposed framework for ad-hoc retrieval.
Revised: As our future work we would like to leverage the content from document body and click sequence to update M-NSRF (especially the session-level recurrent states) so that we can further explore the potential of the proposed framework for ad-hoc retrieval.

Original: As attention mechanism shows promise in improving query suggestion performance, we also also explore it in our multi-task learning setting.
Revised: As attention mechanism shows promise in improving query suggestion performance, we also also explore it in our multi-task learning setting.

Original: In addition, a broad research direction is to go beyond session boundaries to model users’ long-term search goals to enhance personalized search results and query suggestions
Revised: In addition a broad research direction is to go beyond session boundaries to model users’ long-term search goals to enhance personalized search results and query suggestions.

Original: B M ULTITASK M ATCH -T ENSOR A RCHITECTURE
Revised: B M ULTITASK M ATCH -T ENSOR A RCHITECTURE

Original: C D OCUMENT R ANKING B ASELINES
Revised: C D OCUMENT R ANKING B ASELINES

Original: Deep semantic similarity model, DSSM (Huang et al., 2013) maps words to letter tri-grams using a word-hashing technique and uses a feed-forward neural network to build representations for both query and document.
Revised: Deep semantic similarity model, DSSM (Huang et al., 2013) maps words to letter tri-grams using a word-hashing technique and uses a feed-forward neural network to build representations for both query and document.

Original: Similarity, convolutional latent semantic model, CLSM (Shen et al., 2014) uses word-hashing technique and uses convolutional neural networks (CNN) to build query and document representations.
Revised: Similarity, convolutional latent semantic model, CLSM (Shen et al., 2014) uses word-hashing technique and uses convolutional neural networks (CNN) to build query and document representations.

Original: To compute relevance between query and document, both DSSM and
Revised: To compute relevance between query and document, both DSSM and

Original: CLSM uses cosine similarity.
Revised: CLSM uses cosine similarity.

Original: ARC-I (Hu et al., 2014) uses CNN to form query and document representations and employs a multi-layer perceptron to compute relevance score.
Revised: ARC-I (Hu et al., 2014) uses CNN to form query and document representations and employs a multi-layer perceptron to compute relevance score.

Original: In our implementation, we used 128 convolution ﬁlters of size 1, 2 and 256 ﬁlters of size 3.
Revised: In our implementation, we used 128 convolution ﬁlters of size 1, 2 and 256 ﬁlters of size 3.

Original: ARC-II (Hu et al., 2014) was proposed by focusing on learning hierarchical matching patterns from local interactions using a CNN.
Revised: ARC-II (Hu et al., 2014) was proposed by focusing on learning hierarchical matching patterns from local interactions using a CNN.

Original: To keep the ARC-II model simple, we use two layers of 2d convolution and max-pooling each and two-layer feed forward neural network to compute relevance score.
Revised: To keep the ARC-II model simple, we use two layers of 2d convolution and max-pooling each and two-layer feed forward neural network to compute relevance score.

Original: DRMM (Guo et al., 2016a) aims to perform term matching over histogram-based features ignoring the actual position of matches.
Revised: DRMM (Guo et al., 2016a) aims to perform term matching over histogram-based features ignoring the actual position of matches.

Original: In DRMM, histogram-based features are computed using exact term matching and pretrained word embeddings based cosine similarities.
Revised: In DRMM, histogram-based features are computed using exact term matching and pretrained word embeddings based cosine similarities.

Original: In principal, the histogram counts the number of word pairs at different similarity levels.
Revised: In principal, the histogram counts the number of word pairs at different similarity levels.

Original: The counts are combined by a feed forward network to produce ﬁnal ranking scores.
Revised: The counts are combined by a feed forward network to produce ﬁnal ranking scores.

Original: The DUET (Mitra et al., 2017) model composed of a local and distributed model where the distributed model projects the query and the document text into an embedding space before matching, while the local model operates over an interaction matrix comparing every query term to every document term.
Revised: The DUET (Mitra et al., 2017) model composed of a local and distributed model where the distributed model projects the query and the document text into an embedding space before matching, while the local model operates over an interaction matrix comparing every query term to every document term.

Original: Similarly, Match-Tensor (Jaech et al., 2017) model incorporates both immediate and larger contexts in a given document when comparing document to a query.
Revised: Similarly, Match-Tensor (Jaech et al., 2017) model incorporates both immediate and larger contexts in a given document when comparing document to a query.

Original: D Q UERY S UGGESTION B ASELINES
Revised: D Q UERY S UGGESTION B ASELINES

Original: Seq2seq model proposed by Bahdanau et al.
Revised: Seq2seq model proposed by Bahdanau et al.

Original: (2015) is a general neural network architecture that can be applied to the task where both input and output consist of a sequence of tokens.
Revised: (2015) is a general neural network architecture that can be applied to the task where both input and output consist of a sequence of tokens.

Original: This method have been shown successful in machine translation and sequential tagging.
Revised: This method have been shown successful in machine translation and sequential tagging.

Original: Because different input tokens may contribute to each output token differently, attention mechanism which learns a weight between each input-output token pair can further improve the Seq2seq model.
Revised: Because different input tokens may contribute to each output token differently, attention mechanism which learns a weight between each input-output token pair can further improve the Seq2seq model.

Original: In this paper, we consider a seq2seq with global attention method proposed by Luong et al.
Revised: In this paper, we consider a Seq2seq with global attention method proposed by Luong et al.

Original: (2015), which is suitable for short text such as web queries.
Revised: (2015), which is suitable for short text such as web queries.

Original: HRED-qs suggested by Sordoni et al.
Revised: HRED-qs suggested by Sordoni et al.

Original: (2015) is very close to our work which proposed to use a hierarhical recurrent encoder-decoder approach by considering session information for context-aware query suggestion.
Revised: (2015) is very close to our work which proposed to use a hierarhical recurrent encoder-decoder approach by considering session information for context-aware query suggestion.

Original: E M ORE E XAMPLES OF Q UERY S UGGESTION BY M-NSRF
Revised: E M ORE E XAMPLES OF Q UERY S UGGESTION BY M-NSRF

Original: Previous session queries discount pet supplies, homes for rent smyrna georgia
Revised: Previous session queries discount pet supplies, homes for rent smyrna georgia

Original: Next user query homes for rent atlanta georgia
Revised: Next user query homes for rent atlanta georgia

Original: Suggested next query pet friendly rentals in georgia
Revised: Suggested next query pet friendly rentals in georgia

Original: Previous session queries language aptitude test, foreign language aptitude test
Revised: Previous session queries language aptitude test, foreign language aptitude test

Original: Next user query american idol
Revised: Next user query american idol

Original: Suggested next query american language association
Revised: Suggested next query american language association

Original: Previous session queries saturday night fever, saturday night fever nj band
Revised: Previous session queries saturday night fever, saturday night fever nj band

Original: Next user query new jersey cover band
Revised: Next user query new jersey cover band

Original: Suggested next query saturday night live
Revised: Suggested next query saturday night live

Original: Previous session queries pregnancy, abortion, abortion clinics
Revised: Previous session queries pregnancy, abortion, abortion clinics

Original: Next user query tampa abortion
Revised: Next user query tampa abortion

Original: Suggested next query abortion clinics in ﬂorida
Revised: Suggested next query abortion clinics in ﬂorida

Original: Previous session queries ncaa basketball, ncaa basketball trees, ncaa mens basketball bracket, sportscenter Next user query mens ncaa basketball odds
Revised: Previous session queries ncaa basketball, ncaa basketball trees, ncaa mens basketball bracket, sportscenter Next user query mens ncaa basketball odds

Original: Suggested next query espn
Revised: Suggested next query espn

Original: Previous session queries childhood autism rating scale, childhood autism rating scale free, autism screening questionnaire Next user query pervasive developmental disorder
Revised: Previous session queries childhood autism rating scale, childhood autism rating scale free, autism screening questionnaire Next user query pervasive developmental disorder

Original: Suggested next query how to do questionnaire
Revised: Suggested next query how to do questionnaire