Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision (2025)

Abstract: Instruction-following LLMs have recently allowed systems to discover hidden concepts from a collection of unstructured documents based on a natural language description of the purpose of the discovery (i.e., goal). Still, the quality of the discovered concepts remains mixed, as it depends heavily on LLM’s reasoning ability and drops when the data is noisy or beyond LLM’s knowledge. We present Instruct-LF, a goal-oriented latent factor discovery system that integrates LLM’s instruction-following ability with statistical models to handle large, noisy datasets where LLM reasoning alone falls short. Instruct-LF uses LLMs to propose fine-grained, goal-related properties from documents, estimates their presence across the dataset, and applies gradient-based optimization to uncover hidden factors, where each factor is represented by a cluster of co-occurring properties. We evaluate latent factors produced by Instruct-LF on movie recommendation, text-world navigation, and legal document categorization tasks. These interpretable representations improve downstream task performance by 5-52% than the best baselines and were preferred 1.8 times as often as the best alternative, on average, in human evaluation.

## introduction algorithms for discovering interpretable latent structures from observed data is a long-standing challenge in ai @xcite , with applications to bioinformatics @xcite , social science @xcite , and beyond. these methods help users make sense of large amounts of unstructured data to draw insights for various needs. we seek to develop an instruction-following latent factor discovery system that can adapt its discovery process based on user instructions. for example, as shown in figure 1 , a movie streaming platform (the user) may wish to analyze a dialogue corpus where its customers chat about movies, with the goal of gaining insights into different types of movies their customers are interested in. in this case, the latent factor discovery system should ideally discard properties in the data that are irrelevant to the platform's goal, for example, article words in lda @xcite 's output. meanwhile, the system should also produce informative latent factors with fine-grained details, which existing llm-based frameworks (e.g., @xcite ) cannot consistently generate for noisy data. importantly, we focus on the case where we do not rely on the properties of specific datasets to guide gradient-based latent factor models. such a constraint makes setting up task-specific supervision signals, e.g., auxiliary loss functions that exploit the structure of the corpus (such as sentiment labels or co-clicks associated with each document @xcite ; @xcite ) infeasible. classic latent factor models such as latent dirichlet allocation (lda) @xcite and @xcite are popular choices in mining latent patterns from data, however, they cannot flexibly follow user instructions. further, interpreting the latent space of classical latent factor models frequently requires reading-the-tea-leaves interpretations, such as examining clusters of keywords @xcite or interpreting sampled documents @xcite . as we will later show in our experiments, these noisy explanations only provide good interpretability for users when the most salient signals (i.e., words) in the dataset align with the user goal well. another recent emerging paradigm is to prompt llms for pattern discovery from documents @xcite @xcite . while these llm-based frameworks can adapt their behavior based on user instructions, these methods do not scale well to large-scale, noisy datasets. in particular, llm-based methods generally first prompt llms to generate potentially interesting pattern descriptions, such as topic names, prompted with the input data, then use llms to link data points to these generated descriptions @xcite . however, since such a process is purely llm-driven, its success is conditioned on an llm's ability to reason over the dataset of interest, which fails when the observed data exceeds the content understanding ability of llms @xcite . as we will later show in our experiments, when dealing with noisy, out-of-distribution data, these pure llmbased methods fail to consistently produce coherent results that are helpful for users. to address these challenges, we propose instruct-lf foot_0 , a framework that combines the instruction-following task-adaptability of llms with classical gradient-based latent factor modeling algorithms. unlike existing works on using llms for topic proposal and assignment @xcite , we propose simplifying llm-based operations in our framework to a minimum: an property proposal step that simply prompts llms for goal-related properties based on an input document, where each property is a natural language statement that describes a goal-oriented attribute of the input data point. after this, we estimate the occurrence of each candidate property across the dataset by factorizing and filling a sparsely filled data-property matrix, where an observed data-to-property linkage with high estimated score means a property is more likely generated from a data point via the property proposal step. finally, our framework clusters these properties into groups by estimating their correlation through their estimated occurrence across the dataset. the grouped properties then become explanations of the discovered latent factors, similar to how topic models use grouped keywords as discovered topics. we evaluate our proposed method on three scenarios: (1) analyzing a dialogue corpus where users recommend movies to each other, with the goal of understanding different types of user interest, (2) analyzing user-environment interaction logs on a text-based world simulator, alfworld @xcite , with the goal of discovering different meaningful states that are relevant to task completion, and (3) analyzing a set of american bill summaries, with the goal of discovering categories of bills. the first two scenarios require a model that can adapt to users' goal of discovery, while the third provides a testbed where classic latent factor models are known to perform well. automatic and human evaluation show instruct-lf can discover informative and task-relevant patterns from data, and rated as the best-picked model across various baselines in human evaluation. our contributions are as follows: we develop a new method, instruct-lf, that uncovers latent patterns that are relevant to the users' goal of discovery expressed in natural language from unstructured data. second, instruct-lf is the first work, to our best knowledge, to combine llm reasoning with classical gradient-based methods for latent pattern mining. finally, we perform comprehensive evaluations and show that our system uncovers goalrelevant, coherent, informative, and interpretable latent factors from data and is chosen most frequently as the best model against state-of-the-art baselines in human evaluation. ## related work latent factor models latent factor models (lfms) assume observed data in a dataset are governed by a set of latent factors. traditional algorithms for estimating latent factors from data such as lda @xcite , pca @xcite , and autoencoders @xcite . oftentimes, practitioners gain insight into the dataset by interpreting the learned factors, such as reading the topics in an lda model or data point user goal llm nl property descriptions for each data point in dataset do: datum dense embedding model prop. emb. distance score (1a) (1b) data points properties predicted compatibility score via emb. dist. latent factors goal-oriented data transformation latent factor discovery figure 2 : the proposed framework. instruct-lf generates a set of natural language property descriptions from data, i.e., documents (1a); then estimates the compatibility between each data point and each property (1b), and perform correlation-based grouping of properties to discover latent factors (2). the compatibility between each property is efficiently computed using a distilled dense text representation model. we provide additional details and examples in figure 4 and appendix j. llms for insight mining recently, there have been some successes in using llm-based systems for analyzing textual documents, such as using llms to discover topics from documents @xcite or clustering documents @xcite . the most relevant framework to our work is topicgpt @xcite , which uses llm to propose potential topics by sequentially iterating over the dataset and generating new topics based on prior outputs. ## problem setup quantitatively evaluating task-oriented latent factor discovery is challenging in that there is not always an intuitive method to measure whether the discovered latent factors are truly informative and related to the users' discovery goal. to this end, we select three use cases where the task usefulness of the discovered latent factors can indeed be quantitatively evaluated with task-specific evaluations. specifically, we adopt popular choice of evaluation in representation learning @xcite , and use performance derived from latent representations on downstream tasks as a proxy for evaluating the latent space of latent factor discovery models. the trends from automated evaluation are then corroborated via user studies. in particular, we experiment with (1) discovering factors related to users' interest in movies from conversational recommendation dialogues, (2) discovering factors related to users' actions from embodied navigation action logs, and (3) discovering factors related to document topics from a set of documents. each of these scenarios embeds a goal whose success can be quantitatively evaluated via downstream predictive tasks, namely conversational recommendation (section 6), user action prediction (section 7), and document labeling (section 8), which helps to verify the effectiveness of latent factor discovery systems. we provide detailed discussion on task-specific evaluations in experiment sections. ## experiments baselines we compare our method with (1) classic gradient-based baselines (lda @xcite and bertopic (grootendorst, 2022)), and (2) topicgpt @xcite , a recent state-ofthe-art llm-reasoning-based framework for topicmodeling. we adapt topicgpt by additionally including users' goal for discovery into its prompts. by default, topicgpt use early-stopping that stops the topic proposal process when a pre-defined number of duplicate concepts (100 in practice) are generated. to this end, we also experiment with another variant with such constraint lifted, denoted by topicgpt-full. while not all these frameworks are designed to take user goal into account, they represent the most applicable current method for uncovering goal-oriented latent factors from unstructured data. other than the default setting for instruct-lf, we additionally evaluate a binarized version of our model, instruct-lf-bin, where the estimated compatibility score in c are binary. this model variant represents the extreme case where the user wants to have full interpretabilty on whether cer-tain property is related to a data point as a binary value. for all llm-based methods, we evaluate with two representative llms: gpt-3.5 and gpt-4o. to evaluate whether instruct-lf is indeed less reliant on a strong base llm, we also report performance using mistral-7b @xcite , an open-source language model which prior work find cannot produce coherent topics due to noisy generation results @xcite . we discuss hyper-parameters (appendix h), efficiency (appendix l), case studies (appendix d) and prompt stability (appendix g) in the appendix. ## movie recommendation on conversational recommendation (cr) task, we use the inspired (hayati et al., 2020) dataset, a widely used cr dataset with semantically rich multi-turn dialogues. we follow the same dataset split procedures as in prior works @xcite , randomly partitioning the dataset into 731 training and 211 test samples. we provide additional dataset details in appendix h. to compare the quality of different methods on the dataset, we adopt nbcrs @xcite , a retrieval-based conversational recommender system that makes recommendations by outputting popular movies in a semantic neighborhood with any document representation methods. concretely, given a test dialogue history (i.e., previous interactions between the user and the assistant) and a method that is being tested, we encode the history into embedding using the latent space discovered by the corresponding method, and retrieve its k nearest neighbor in the embedding space. following recent works on conversational recommendation @xcite , we evaluate the performance of systems by hit@k w.r.t. the ground-truth movie mentioned in the response to the dialogue history, where @xmath0, 5, 20. the performance of the models is as shown in ## human evaluation and ablations task-relevance and informativeness while we quantitatively demonstrate the task-effectiveness of our framework with automated evaluation, a core requirement of a successful latent-factor discovery system is that it should uncover informative, task-relevant features and present the findings via interpretable signals to the user. to this end, we run a human evaluation on amazon mechanical turk with the question of whether instruct-lf can uncover good-quality latent properties. to assess the overall performance of different frameworks, we also ask human evaluators examine a discovered factor from all methods, and pick out one or more frameworks they prefer to use most when tasked to understand patterns in a dataset (see ap ## conclusion and future works we develop instruct-lf, a latent factor discovery framework that uncovers task-relevant, informative, and interpretable latent concepts from unstructured data based on users' instruction in natural language. instruct-lf combines the instructionfollowing ability of llms and the scalability of gradient-based latent factor models, demonstrating the promise of improving statistical algorithms with llm reasoning ability.