Incorporating Behavioral Hypotheses for Query Generation (2020)

Abstract: Generative neural networks have been shown effective on query suggestion. Commonly posed as a conditional generation problem, the task aims to leverage earlier inputs from users in a search session to predict queries that they will likely issue at a later time. User inputs come in various forms such as querying and clicking, each of which can imply different semantic signals channeled through the corresponding behavioral patterns. This paper induces these behavioral biases as hypotheses for query generation, where a generic encoder-decoder Transformer framework is presented to aggregate arbitrary hypotheses of choice. Our experimental results show that the proposed approach leads to significant improvements on top-k word error rate and Bert F1 Score compared to a recent BART model.

## introduction query suggestion is key to the usability of a search engine in the way it helps users formulating more effective queries or exploring related search needs. prior work tackles this problem by employing primarily two strategies. the first one is based on a discriminative characterization, with candidate queries drawn from production logs ranked to align with what users may most likely issue next. although effective @xcite , this strategy is inherently restricted by what is available in the logs, which in turn can penalize tail queries @xcite . in this work, we pursue the second strategy where query suggestion is cast as a natural language generation problem, aiming at producing effective continuations of the user's intent by using generative modeling. for query generation, prior research has focused mostly on extending standard seq2seq models where the input is a concatenation of earlier queries a user has submitted in a session @xcite , 2015; @xcite . however, literature often leaves out the influence of clickthrough actions (i.e., red blocks in figure 1 ), which we argue should be taken into account in the generative process as they could be surrogates of the user's implicit search intent @xcite . users may exhibit diverse behaviors such as consecutively issuing queries without further engagement, or following up a single query with extensive clickthrough actions. these vastly different patterns are indicative of information pieces that the users find most relevant, which we conjecture can help producing suggestions better aligned to the user needs. we present an encoder-decoder transformer model for the generative task that includes these patterns that we called behavioral hypotheses. one challenge that arises with transformers is that they make minimal assumptions about input (i.e., a single string of tokens), making it non-trivial to add multiple hypotheses directly. to address this issue, we propose a generic approach that leverages tokenwise attention to aggregate multiple behavior hypotheses encoded by a shared transformer encoder bart @xcite . the resulting end-to-end model can capture the underlying userinduced belief while maintaining the same order of complexity as the original bart. for evaluation, we conduct experiments by sampling over 600k search sessions from a major commercial job search engine in australia. with evaluation metrics including word error rate and bertscore @xcite , we show that the approach outperforms prior competitive baselines and a recent transformer model bart, suggesting attending to behavioral patterns is crucial to reflect users' intent. ## related work generative approaches have been studied extensively in machine translation @xcite , dialogue systems @xcite , and many other related areas @xcite . the methodology was first applied to the query domain in @xcite . query suggestion has traditionally been a web search usability task. ranking based approaches that leverage query co-occurrence and discriminative modeling are known to be most effective @xcite , but also likely to suffer from the lack of appropriate candidates for rarely seen queries. some recent work sought to characterize the generative nature @xcite of this process. the hierarchical formulation of sequence-to-sequence model @xcite can effectively capture the query transitions, but does not offer a mechanism to incorporate implicit user signals @xcite . our approach combines heterogeneous behavioral hypotheses by leveraging large-scale encoders and cross-structure attentions. apart from similar attempts regarding encoding multiple sentences @xcite , our work in a generative setting tackles a different problem of decoding over a meshed representation originated from multiple sources. ## approach let @xmath0, q 2 , . . . , q n ) represent a sequence of queries submitted by a user in a consecutive fashion, where each q i comprises a sequence of terms. each q i can lead to a succession of follow-up user interactions, among which we are mostly interested in textual matching cues c i that enticed clicking on some underlying documents. the full list of such matching cues are denoted as @xmath1, c 2 , . . . , c n ), in which each c i is a set of text excerpts t i1 , . . . , t im , such as document title or metadata, displayed in response to q i . given such a search context (q, c), the query generation task aims to create a candidate query q n+1 that the user is most likely to follow up with. the overall process is depicted in figure 1 . behavioral hypotheses. we conjecture that, when making a new query, the assumed user takes inspiration from his/her preceding search context, following some behavioral hypotheses formed by preceding queries or matching cues. in this paper we seek to characterize this influence to formulated queries as follows: each of these definitions loosely specifies a generative story behind the process: influence may come directly from preceding queries (k 1 ), preceding matching cues (k 2 ), interacted queries and the respective matching cues (k 3 ) , or the most recently submitted query and observed cues (k 4 ). ## experimental setup our main testbed was a sample of session logs from the seek job search engine 1 . this task domain is known for its characteristic query topics, surrounding role titles, skills, and entities such as company names or geo-locations, and distinctively different user behaviors to general web search. this dataset is preferred over the aol logs for the availability 1 https://www.seek.com.au of clicked document texts, but our approach should be equally applicable to other search domains. we collected textual queries q i and the titles of documents that were clicked on in response to q i as c i . all search sessions were anonymized to ensure that the query and click information cannot be linked back to individual users. session boundaries were determined by an inactivity of 30 minutes or more between two consecutive actions. in each session the latest query was held out as the ground truth. training sessions (500k) were initially gathered from a two-week span starting from oct 1, 2019, and out of the same period a separate split was selected as a dev set (1k); then, the latter two weeks from the same month were sampled to form the test sessions (100k). about 15% of the collected sessions were found to exceed the maximum sequence length of the bart encoder, and were removed from the experiment to avoid inadvertently favoring the proposed approach for that the baseline may only see truncated input. standard preprocessing steps were performed to remove noisy queries that occurred 10 times or less across the periods, and singleton sessions that contain only one query. in our experiments we compare the following approaches: seq2seq+attn a standard sequence-tosequence model using a two-layer bidirectional gru @xcite as the encoder and a uni-directional attentive gru as the decoder @xcite . our implementation used 1,000 hidden dimensions and the same byte-pair encodings as other methods. ## results quality of generated queries. we present the effectiveness scores of different generation models in further comparisons with non-generative mps also shed light on the superiority of the proposed approach. in a w/t/l analysis on top-3 ranking performance, we find that across all test sessions meshbart has seen 30% wins, 52% ties, and 18% losses to mps on mrr@3. another analysis conditioned on sessions with only one preceding query shows that meshbart can produce at least one novel suggestion (i.e. queries not seen in the candidate pool) for 39.2% of the test sessions. the effect is more pronounced when the preceding query is rare or has a relatively smaller candidate pool. examples given in analysis of behavioral hypotheses. our design of behavioral hypotheses in section 3 is inspired by the users' interaction patterns. figure 3a illustrates the intensity of user clicking with respect preceding query (qn) candidates generative suggestions (meshbart) environmental technology environmental, environment environmental, environmental science, sustainability, environmental scientist part time adobe part time, part time marketing part time marketing, marketing, digital marketing, graphic design aviation security adelaide adelaide airport security aviation security, security, security officer, airport security to understand the inner workings of the meshed attentions, figure 3b visualizes the actual attention values assigned to all four of the presented hypotheses across different length buckets. on the one hand, the attention weight associated with k 1 (i.e., all preceding queries) are found to positively correlate with the growth of session length. longer search sessions might be due to the user actively exploring the search space, and this increased attribution signals the importance of explicit search intents from the modeling perspective. on the other hand, k 4 tends to receive less attention as the search session grows, indicating that the importance of the most recent interactions become diluted in long, exploratory search journeys. the value of recent interactions in generative modeling is best illustrated by figure 3c , where the attention weight of k 4 appears to positively correlate with the intensity of last-round clicking in a search session. these results suggest that our approach has the flexibility to draw information from different hypotheses in a unified query generation process. ## conclusions this paper presents an effective approach for incorporating user-induced interaction patterns as behavioral hypotheses into the query generation process. under an encoder-decoder transformer framework, the proposed tokenwise attentions demonstrate the desirable modeling working by placing emphasis on different behavioral hypotheses at different occasions. on a domain-specific search benchmark, our model outperforms all reference methods in aggregate and across varying session properties, demonstrating its effectiveness in a robust way. in future work, we will focus on producing novel continuations of the user's search intent, extending the approach to other domains, and automating the design of behavioral hypotheses. qualitative evaluation for open-ended generation is also an interesting topic on the roadmap.