LOHR ec: Leveraging Order and Hierarchy in Generative Sequential Recommendation (2025)

Abstract: The sequential recommendation task involves predicting the items users will be interested in next based on their past interaction sequence. Recently, sequential recommender systems with generative retrieval have garnered significant attention. However, during training, these generative recommenders focus only on maximizing the prediction probability of the next target item in the temporal sequence, while neglecting awareness of diverse plausible potential items.Although introducing large language models (LLMs) with world knowledge and adding a set of auxiliary tasks that can link item identifiers to their real-world meanings can alleviate this issue, the high inference costs associated with these LLM-based recommenders make them challenging to deploy in practical scenarios. In this paper, we propose a novel learning framework, LOHRec, which leverages the order and hierarchy in generative recommendation using quantized identifiers to further explore the performance ceiling of lightweight generative recommenders. Under fair comparisons with approximate backbone parameter sizes, comprehensive experiments show that all variants of generative recommenders using our framework outperform strong prior baselines across multiple datasets. Furthermore, we empirically demonstrate that LOHRec can efficiently align lightweight generative recommenders with LLM recommendation preferences in low-resource scenarios, further demonstrating its practical utility. Our code repository is available at [https://github.com/xjw-nlp/LOHRec](https://github.com/xjw-nlp/LOHRec).

## introduction recommender systems are increasingly popular in addressing the information overload problem on web platforms such as shopping sites, video platforms, and social media. these systems can help users discover items of interest and enhance their experience and engagement @xcite 2020; @xcite . among these paradigms of recommenders, sequential recommendation @xcite has recently garnered considerable attention due to its superior performance in modeling the dynamic evolving pattern of the chronological item sequence. in recent years, neural sequential recommenders have achieved remarkable progress, such as using cnns @xcite , rnns @xcite , or attention-based models @xcite to capture users' evolving interests over time. with the transformer architecture having demonstrated astonishing performance in a spectrum of domains @xcite , the transformer-based sequential recommender is increasingly popular in research and practical application @xcite . however, prior work typically represents items with atomic, random item ids, which do not fully leverage the rich multimodal information that describes those items @xcite . in addition, industrial settings often involve an enormous number of items-billions or more-so traditional recommendation schemes must maintain a very large item-id vocabulary. this complicates optimization and deployment and figure 2 : the overall framework of our lohrec. we enhance generative sequential recommenders by leveraging the order information for contrastive learning and conducting constrained decoding via the hierarchy of quantized semantic item ids. to address the challenges in the sequential recommendation, recent research has moved towards a generative retrieval paradigm by adopting neural quantized representations. specifically, @xcite proposes a residual quantization method that converts item embeddings into semantic ids, rather than using low-information numerical ids. this approach allows the number of representable items to be the product of the cardinality of each level in semantic ids, thereby significantly alleviating the scalability issues in large item corpora. additionally, modeling sequences of semantic item ids facilitates the discovery of deeper correlations between items, enhancing the learning efficiency of sequence transitions and improving the performance of recommenders. further, rather than optimizing transformer-based models from scratch, @xcite proposes leveraging the robust language comprehension capabilities of large language models (llms) for the sequential recommendation. they adapt llms for sequential recommenders through a series of auxiliary tasks, further raising the performance ceiling of generative recommenders. while the generative recommendation paradigm has shown promising performance across various public real-world datasets, several weaknesses are blocking the availability and reliability of the scheme. firstly, since these generative recommenders are commonly trained with maximum likelihood estimation (mle), the probability score assigned to the next target item is maximized given a user interaction history. however, we argue that in real-world scenarios, a user is often interested in multiple items given their current interaction state, and optimizing solely on the target item from the interaction history often leads to suboptimal performance. we demonstrate this phenomenon using a representative baseline, tiger, trained with mle loss @xcite . in figure 1 (left), the intra-list diversity of the top-10 recommendations plateaus and even decreases as beam width increases, suggesting that the generative recommender increasingly homogenizes its outputs due to an excessive focus on a single item. while leveraging llms with rich world knowledge to align item identifiers with actual item semantics can alleviate recommendation homogenization, the substantial inference cost of llm-based recommenders hinders their applicability in large-scale online environments. this work proposes lohrec, leveraging the order and hierarchy in the generative sequential recommendation, to address the limitations mentioned above. specifically, because generative recommenders use beam search at inference time and rank candidate outputs based on cumulative probability, we propose introducing a learning-to-rank objective during training to align the ordered relationships among multiple candidate items so that the probability ranking matches the desired ordering. in this way, items assigned higher probabilities at inference better reflect practical requirements, improving overall recommendation performance. on the other hand, considering the hierarchical semantic nature of item identifiers, we propose a hierarchical training paradigm that enables the model to be aware of a broader set of potential items rather than focusing solely on the target item, thereby jointly accounting for recommendation accuracy and diversity. to sum up, the main contributions of this paper can be presented as follows: ## conclusion and future work in this paper, we study improving generative recommendation performance via leveraging the ubiquitous ordered correlation and hierarchical semantics in quantized item ids. specifically, we demonstrate the effectiveness of lohrec in aligning the recommendation preferences of lightweight generative recommenders with those of llm-based recommenders. additionally, since lohrec only uses additional tokens from the semantic item id set, it can be applied to models with incompatible original vocabularies, demonstrating the sound generalizability. for future work, our approach can be naturally extended to larger parameter scales and even various pre-trained language models. additionally, our method can be applied to learning more types of sequential information to adapt to specific downstream tasks.