Leveraging Order-Free Tag Relations for Context-Aware Recommendation (2021)

Abstract: Tag recommendation relies on either a ranking function for top-k tags or an autoregressive generation method. However, the previous methods neglect one of two seemingly conflicting yet desirable characteristics of a tag set: orderlessness and inter-dependency. While the ranking approach fails to address the inter-dependency among tags when they are ranked, the autoregressive approach fails to take orderlessness into account because it is designed to utilize sequential relations among tokens. We propose a sequence-oblivious generation method for tag recommendation, in which the next tag to be generated is independent of the order of the generated tags and the order of the ground truth tags occurring in training data. Empirical results on two different domains, Instagram and Stack Overflow, show that our method is significantly superior to the previous approaches.

## introduction recommendation techniques have been widely used for diverse applications both in symbolic and deep neural network frameworks. a conventional approach to recommendation is a ranking scheme that returns top-k relevant target items for a given query or a user profile. as such, recommending tags (e.g., hashtags and labeled tags), which is the main focus of this paper, has been treated as a ranking problem @xcite @xcite @xcite @xcite . these approaches, however, neglect inter-dependency among the tags (see figure 1 )in a way conventional information retrieval techniques do for ranking. when tags have dependency among themselves, especially with a query, it would be desirable to consider such dependency when the next tag is selected for recommendation. on the opposite side of the spectrum are the recent studies employing an autoregressive (ar) generation model for tag recommendation @xcite where a gru decoder @xcite enables the modeling of the dependency among the generated tags. while this approach considers dependency present in a sequence of tags, it overlooks the nature of tag recommendation where the output is a set of tags (orderlessness) rather than a sequence. for tag recommendation, it is important that all the tags generated to a certain point in time have identical influence on the decision for the next generation (inter-dependency) as in figure 1 . contrasting a list of tags against a sequence of words in a natural language sentence sheds light on the characteristics of the tag recommendation problem. unlike an ordinary sentence comprised of a sequence of tokens, a set of recommended tags is not predicated upon any syntactic rule. it is unordered and syntax-free, and yet exhibits dependency among the tags themselves based on their semantic relatedness. these seemingly conflicting characteristics suggest that tag recommendation be seen as an application with a new class of language, calling for a new method beyond the text generation framework widely adopted for nlp @xcite @xcite . more specifically, we note that the decoderoriented ar approach, even with a transformer @xcite decoder, is essentially limited in that the next token to be generated is highly dependent on the last generated token (see figure 1 ). while remembering what has been generated is helpful, furthermore, maximizing a sequence likelihood adopted in a typical text generation model is unnecessarily constraining for a tag recommendation; the order of generated tags is immaterial as they could be shuffled without changing their overall relevance to the query. in this paper, we propose the sequence-oblivious generation (sog) method for tag recommendation. its main feature is that a tag is generated independently from the order of the previously generated tags. instead, it iteratively expands the input query with the generated tag so that the expanded query affects the next tag prediction. our approach can be seen as analogous to pseudo-relevance feedback (prf) @xcite where previously retrieved items are deemed relevant and used to extract additional query terms for relevance feedback. in our approach, previously generated tags are added to the query and fed back to the process of predicting the next tag to be generated. note that sog is devised for the unique nature of generating syntax-free yet inter-dependent tags. instead of the usual rnn or transformer decoder where the sequence of tokens plays a key role, sog leverages the transformer encoder and its self-attention to draw on the identical flow of information from every input feature position, removing the sequence-dependent nature of decoder-based generation. moreover, our model is trained to ignore the order of the ground truth tags by maximizing on the entire set of tags (1-to-m) at every step of generation, which is distinguished from maximizing a sequence likelihood (1-to-1) in a typical autoregressive text generation task. also proposed in this paper is a scheme for late fusion of multi-modal context input, namely, text, image, time, location, and tags, as opposed to conventional late fusion approaches. we employ bert @xcite for tag generation to exploit its self-attention mechanism for multimodal fusion and its pre-trained language understanding capability. our intuition is that input features should be encoded simultaneously for their mutual influences, not as separately encoded parts followed by their fusion at a later stage. we conduct extensive experiments on two different domains (instagram and stack overflow) for recommendation and show that sog outperforms the ranking, ar generation and late fusion approaches by significant margins. we provide detailed analyses of the comparisons between sog and established baselines to shed light on the different perspectives of the proposed approaches for recommendation. ## related work the tag recommendation problem has mostly been studied as a ranking process, with models extracting top-k tags given an input query @xcite @xcite @xcite @xcite . they map the input features and tag embeddings to a common embedding space and learn with a pairwise ranking loss @xcite @xcite , or address tags as latent topics by applying topic modeling like latent dirichlet allocation @xcite @xcite . a recent line of work on tag recommendation takes a generative approach @xcite . however, their application of gru shows that tags are treated as an ordered sequence, neglecting the orderless yet interrelated characteristics of tags. this characteristics of a tag set pose a significant challenge to the encoderdecoder generation scheme common to text generation @xcite @xcite @xcite , where the autoregressive (ar) decoding mechanism heavily relies on the immediately preceding token. this ar approach bounds the model to sequential ordering of the target sequence. there have been attempts to use bert in a generative fashion for text generation @xcite , exploiting its language understanding capabilities. while they use bert to model sequential dependency, we explore its potential to ignore the sequential aspect. recent work @xcite takes a reinforcement learning (rl) approach to modeling the unordered yet dependent characteristics of a target domain. despite their learning objective to maximize reward on the orderless prediction, however, the model architecture still bound to the ar scheme. ## context-aware sog method given the multi-modal context feature types such as image, location, time, and text as well as tags generated to a particular point in time, our recommendation model predicts the next tag using all of them as input query. more formally: the tag generation process is sequenceoblivious, not following the way natural language sentence is generated. it does not abide by predefined syntactic or semantic rules and is oblivious to the sequential order. we define three indispensable components for sog: (i) encoder-based generation architecture with an unbiased exchange of information among tags (section 3.1), (ii) 1-to-m training scheme that mitigates the order constraint (section 3.2), and (iii) greedy decoding that neglects the complex syntactic and semantic constraints of a natural language (section 3.3). unlike the autoregressive generation or ranking schemes, our generation model produces an unordered yet interrelated target tags given an assortment of context features and previously generated tags. in section 3.4, we elaborate on how the four context feature types are fused for our sog model. ## conclusion this paper characterizes the tag recommendation tasks with and without multi-modal context information in the posts of instagram and stack overflow, respectively, and proposes a novel framework, sequence-oblivious generation (sog), that explicitly considers the inter-tag dependency and the orderless nature of tags. we address the drawbacks of the conventional ranking and ar approaches to tag recommendation and define it in a new way so that it attends to the characteristics of "tag language." for the new framework, we design the sequence-oblivious model and training and decoding strategies, together with the bert-based early fusion method for multi-modal features. in the extensive experiments on two different domains, we show that sog outperforms the the baselines by significant margins. also shown are the roles of the iterative query expansion with generated tags, the 1-to-m training scheme under the orderlessness assumption, the early fusion method over late fusion, and the adoption of greedy search for decoding. for future work, we plan to investigate if our generative framework can generalize over other tasks possessing the "tag language" characteristics.