Quantile Regression with Large Language Models for Price Prediction (2025)

Abstract: Large Language Models (LLMs) have shown promise in structured prediction tasks, including regression, but existing approaches primarily focus on point estimates and lack systematic comparison across different methods.We investigate probabilistic regression using LLMs for unstructured inputs, addressing challenging text-to-distribution prediction tasks such as price estimation where both nuanced text understanding and uncertainty quantification are critical.We propose a novel quantile regression approach that enables LLMs to produce full predictive distributions, improving upon traditional point estimates. Through extensive experiments across three diverse price prediction datasets, we demonstrate that a Mistral-7B model fine-tuned with quantile heads significantly outperforms traditional approaches for both point and distributional estimations, as measured by three established metrics each for prediction accuracy and distributional calibration.Our systematic comparison of LLM approaches, model architectures, training approaches, and data scaling reveals that Mistral-7B consistently outperforms encoder architectures, embedding-based methods, and few-shot learning methods.Our experiments also reveal the effectiveness of LLM-assisted label correction in achieving human-level accuracy without systematic bias. Our curated datasets are made available at https://github.com/vnik18/llm-price-quantile-reg/ to support future research.

## introduction large language models (llms) have demonstrated remarkable capabilities across a wide range of tasks, including unstructured document processing @xcite . going beyond their original purpose of text generation @xcite , they have recently been extended to structured numerical prediction tasks such as time series * equal contribution. the intersection of llms and regression is particularly important for the longstanding task of text regression, where unstructured language must be mapped reliably to numeric outputs @xcite . traditional regression models often struggle with applications where crucial information lies in unstructured text, such as product descriptions or financial reports @xcite , requiring rich text understanding. this is crucial in domains like product pricing where heterogeneous features across categories (e.g., screen technology for televisions versus mileage for cars) make traditional unified feature representations inadequate for capturing category-specific dynamics. existing work on regression with llms has explored three main approaches to address these challenges: fine-tuning llms for specific numeric prediction tasks @xcite , using llm embeddings as features for downstream regression models @xcite , and leveraging in-context learning for zero-shot or few-shot numeric estimation @xcite . however, these approaches, with the exception of a few @xcite , focus on point estimates, overlooking a key limitation: the inability to quantify uncertainty. many real-world applications, such as price prediction, demand forecasting, and financial risk assessment, inherently require probabilistic outputs rather than single-value predictions @xcite . probabilistic modeling is essential in these applications to capture uncertainty and normal variation, mitigate risks, and improve decision-making @xcite . current work has neither explored proba-bilistic regression using llms in-depth nor evaluated the trade-offs between different llm regression approaches in a single study. this paper presents the first study of probabilistic regression using llms to encode unstructured text inputs, and take a step towards a systematic investigation of llm-based regression methods. we center our study on price prediction, a task that requires both nuanced interpretation of free-form text inputs and accurate distribution estimation. understanding the complete price distribution is essential in financial contexts where accurately modeling tail behavior is critical for effective risk management. in sum, this paper makes three key contributions. first, we propose a novel llm-based quantile regression approach that produces full distributions with strong calibration while maintaining sharp prediction intervals, and improves point estimation accuracy compared to traditional approaches. qualitative analysis shows that our model produces well-calibrated distributions that adapt to different price ranges and uncertainties across datasets, with tighter distributions for standardized products and appropriately wider distributions for items with more price variability. second, we systematically compare different llm architectures (decoder-only vs. encoder-only vs. traditional ml on text embeddings vs. in-context learning), multiple loss functions (squared error vs. pinball), and various data scales, while investigating training data contamination. we show that fine-tuning decoder models (e.g., mistral-7b) outperforms other approaches. our results confirm that model size, data scaling, and clean training sets all play critical roles in robust, generalizable llm-based probabilistic regression. third, focusing on the task of price estimation, we release three curated datasets (amazon products, craigslist used cars, used boats) with standardized splits. ## price estimation datasets we experiment upon three price prediction datasets from different domains publicly available for research use: amazon products @xcite , craigslist used cars listings, 2 and european boat 2 https://kaggle.com/datasets/austinreese/craigslist-carstrucks-data sales. 3 initial manual inspection revealed numerous instances with erroneous prices, i.e., unreasonably high or low relative to the items being sold (examples are provided in ## experimental setup quantile levels and point prediction: for all models that predict distributions, we take @xmath0, 1) into k equal-length sub-intervals to obtain τ . we discuss the impact of varying k and the smoothing parameter α in appendix c. we use models that produce a distribution both for generating probabilistic outputs and for evaluating point predictions. in the latter case, we take the predicted quantile at τ = 0.5 as the point estimate. additionally, we include baseline models trained solely with traditional squared error loss, using their direct predictions for comparison. baselines using llm embeddings: text features (title, description, attributes) are concatenated with appropriate field markers and converted to embeddings using the qwen2-7b-instruct embedding model @xcite . all baseline models using these embeddings are denoted by the "qwen-7b-emb" prefix. these embeddings serve as input features for five models: ridge regression and xgboost for point estimation, quantile regression (with two hidden layers) for distribution prediction, trained on log-transformed target, foot_1 and two nearest neighbor-based distribution prediction approaches. the first nearest neighbor model (knn) predicts distributions by using the empirical distribution of target values from selected neighbors in the training set, while the second variant employs a radius-based selection criterion (rknn) with a minimum neighbor requirement. all hyperparameters are selected using 5-fold cross-validation. fine-tuned lms with quantile head: we finetune mistral-7b @xcite , phi-3b (abdin et al., 2024), qwen-500m @xcite and xlm-roberta @xcite models with a quantile regression head. in-context learning: we evaluate two state-ofthe-art llms, claude-3.5-sonnet and nova pro @xcite , both zero-shot and few-shot. for few-shot learning, we implement three example selection strategies: (i) random sampling; (ii) category-based stratified sampling and (iii) similar item sampling based on cosine similarity of qwen2-7b embeddings. the latter two leverage domain similarity for potentially better price estimation (prompts in figure 7 ). the second set of metrics measures the distributional quality of the predicted quantiles qτ ( for each metric, we report 95% confidence intervals with bootstrap resampling (1000 iterations): ci 95% (m ) = [ m(0.025) , m(0.975) ] where m(q) denotes the q-th quantile of the bootstrap distribution of metric m . further details about all metrics and the training process are presented in appendix c. ## conclusion we demonstrated the effectiveness of llms with quantile regression heads for probabilistic price prediction from unstructured inputs which not only produce well-calibrated price distributions but also achieve superior point estimates compared to traditional approaches. our mistral-7b-quantile model outperforms traditional approaches and few-shot, in-context learning across multiple datasets, with performance notably improving with larger model sizes and training volumes. our findings establish a foundation for probabilistic regression with llms and showcase their capability in complex numeric prediction tasks using unstructured input. there are several promising avenues for future research, such as hybrid architectures combining decoder-only models with traditional pricing approaches, explicitly incorporating domain knowledge about pricing and market dynamics, exploring advanced llm reasoning techniques, and building more interpretable and reliable models that provide insights into they make pricing decisions. the llm-based regression approach could also be applied to a number of other existing text-based tasks, such as financial forecasting of return and volatility from news articles and social media, sentiment analysis, and text readability scoring.