Decoding in Latent Spaces for Efficient Inference in LLM -based Recommendation (2025)

Abstract: Fine-tuning large language models (LLMs) for recommendation in a generative manner has delivered promising results, but encounters significant inference overhead due to autoregressive decoding in the language space. This work explores bypassing language-space decoding by directly matching candidate items with the LLM’s internal thought representations in the latent space, eliminating the time-consuming autoregressive process to reduce computational costs. Towards this, we introduce Light Latent-space Decoding ( L2D ), an effective and efficient latent-space decoding method. L2D represents user-preferred items by using the hidden states of test sequences reflecting the LLM’s internal thought, and obtains candidate item representations from the hidden states of training sequences labeled with the corresponding candidate items. It then matches the two types of representations to decode items, achieving latent-space decoding. In this way, it enables efficient decoding without altering the LLM’s generative tuning paradigm, thereby preserving performance. Extensive empirical results demonstrate that L2D is more than 10x faster than language-space decoding while maintaining or enhancing performance.

## introduction large language models (llms) have been widely fine-tuned on recommendation data in textual format to directly generate the next item of user interest based on historical interactions @xcite @xcite . this training paradigm aligns well with the generative nature of llms, enabling them to develop sophisticated user understanding and interestmining capabilities. when deploying these finetuned llms for personalized recommendations (llm4rec), a key research challenge is how to ef-figure 1 : illustration of language-space vs. latent-space decoding: latent-space decoding bypasses the slow language-space decoding and instead achieve decoding via directly matching candidate item with the llm internal 'thought' items in the latent space. it preserves the generative tuning paradigm to keep performance while enabling efficient decoding. current llm4rec methods primarily rely on the llm's internal decoding ability (i.e., languagespace decoding), which typically operates the llm in an autoregressive token-by-token manner to output the item representations in the language space (e.g., title) @xcite . in the autoregressive decoding, one token generation would wait for all preceding tokens' generations, incurring substantial time costs. worse yet, each recommendation request from users typically requires generating a list of items @xcite @xcite , linearly scaling the cost regarding the list size. while grounding techniques @xcite can reduce these costs by mapping each generated item to multiple actual items, they may lead to performance degradation. for example, in our findings, mapping only one generated item for top-10 recommendations would result in a performance drop of fifty percent compared to generating 10 items (c.f., figure 3 ). this work considers bypassing this languagespace decoding to enhance decoding efficiency. training a recommendation head to decode the next item by directly predicting its id provides a straightforward way to avoid language-space decoding, improving decoding efficiency. however, this training objective deviates from the language model's original goal of next-token prediction, which may hinder effective utilization of its pretrained knowledge. this raises a new question: can we bypass language-space decoding while still preserving the powerful generative training characteristics of the llm's recommendation tuning? by re-examining llm decoding, we can interpret the tuned llm as having already thought a recommendation target within its internal latent space in response to a user query, with autoregressive decoding merely serving to activate it. if we can find the appropriate representation for the item in this latent space-and represent all candidate items similarly-then efficient decoding can be achieved through representation matching within the same space. meanwhile, since this approach only extracts the llm's internal "thoughts" learned during llm's generative training phase, it preserves the original generative training objective during recommendation tuning. to obtain representations of the items internally considered by the llm, we propose using the last hidden states from the final llm layers corresponding to a given test sample, as the state primarily drives item generation. for candidate items, since the training set already provides matching pairs of hidden states and the item serving as ground-truth items, we can aggregate all the hidden states associated with the item to form an effective representation in the latent space, without incurring additional training cost. this approach is based on the idea that each paired hidden state captures a distinct feature aspect of the corresponding ground-truth item, allowing us to combine them into a meaningful and comprehensive target representation. to this end, we propose light latent-space decoding (l2d), a simple yet efficient method for latent-space decoding. after finishing generative training, we store the training samples' hidden states and their labels (i.e., ground-truth items) in a memory module and create each item's representation by aggregating its associated hidden states in the memory. then we decode items to recommend by finding the item whose representation is most similar to the test sample's hidden state using l2 distance. regarding the aggregation to form item representation, l2d offers two strategies: 1) global aggregation, which averages all associated hidden states for an item, and 2) local aggregation, which uses only the top-m most similar samples from the memory based on the test sample's hidden state. the global strategy provides a comprehensive representation, while the local strategy focuses on aspects most relevant to the test sample. the main contributions of this work are summarized as follows: ## llm-based generative recommender let d represent the user-item interaction data. the j-th sample in d is denoted as (s j , v j ), where s j represents a user's interaction history, and v j is the interacted item for the sample. notably, both s j and v j are in textual form. to train an llm-based generative recommender, we convert each sample (s j , v j ) into instruction data, using a fixed prompt template such as "a user has interacted with the following items: <s j >; which item would the user like next?", with v j as the ground-truth model output. then, the instruction data {(prompt(s j ), v j )} d can be utilized to fine-tune the llm. during inference, given a user's interaction history s to generate the next item, the llm first encodes the prompt into hidden states, formally: ## latent-space decoding in this section, we introduce our light latent-space decoding (l2d) framework, starting with presenting the overview and followed by a detailed description of its key components. ## experiments in this section, we conduct experiments on two widely-used real-world datasets to demonstrate the effectiveness of our l2d framework in balancing performance and inference overhead. we will showcase it by following research questions: rq1: how does the performance and inference overhead of our l2d compare to llm-based baselines? rq2: in which scenarios are global and local aggregation most suitable, respectively? rq3: how does the hyperparameter m affect l2d-l? rq4: how does the performance of l2d compare to the id-based classifier? rq5: what is the spatial efficiency of l2d? ## related work notably, some existing (large) language model (lm)-based approaches @xcite , such as recformer @xcite , can be viewed as representing candidate items in latent spaces and then matching them with the user input sequence encoded by the lm. however, they indeed modify the output layer of the lms, with the effectiveness of their matching process tied to the training process. as a result, they fail to achieve plug-and-play integration into existing advanced llm-based recommenders. in contrast, our method is decoupled from the training process, making it plug-and-play. additionally, these methods' training objectives deviate from the large language model's original goal of next-token prediction, which may hinder effective utilization of its pretrained knowledge. a detailed discussion is provided in appendix a.1. ## conclusion in this study, we emphasized that fine-tuning llms for recommendations in a generative manner is highly promising but encounters significant inference overhead due to the original autoregressive decoding strategy. to address this challenge, we proposed the l2d, which bypasses time-consuming autoregressive decoding in the language space and directly decodes items in llm's latent space. the l2d preserves the generative tuning paradigm to keep performance while enabling efficient decoding. our results highlighted the potential of latent space decoding as a fundamental advancement in llm-based recommender systems, and extensive results demonstrated the superiority of l2d.