Predicting Text Preference Via Structured Comparative Reasoning (2024)

Abstract: Comparative reasoning plays a crucial role in predicting text preferences; however, large language models (LLMs) often demonstrate inconsistencies in their reasoning, leading to incorrect preference predictions. While approaches like Chain-of-Thought improve accuracy in many settings, they struggle to consistently distinguish the similarities and differences of complex texts. We introduce SC^2 , a model that prompts LLMs to predict text preferences by generating structured intermediate comparisons. SC^2 begins by proposing aspects for comparison, followed by generating textual comparisons under each aspect. We select consistent comparisons with a pairwise comparator that ensures each comparison of a given aspect clearly distinguishes differences between texts, significantly reducing hallucination and improving consistency. Our empirical studies across various NLP tasks, including summarization, retrieval, and automatic rating, demonstrate that SC^2 ‘s enhanced performance in text preference prediction is significant.

## introduction comparative reasoning is crucial for predicting text preferences, as deciding the best out of a set of texts requires careful examination of the similarities and differences across the documents. hence, comparative reasoning has been especially useful in nlp tasks such as text summarization @xcite , search ranking @xcite , and automatic evaluation @xcite , where text preference prediction is a key step. however, as corpora grow more dense and complex across domains, accurate comparative reasoning becomes increasingly challenging. existing approaches rely on pretraining or fine-tuning models @xcite at the cost of massive human annotation and computation. with the emergence of large language models (llms) @xcite @xcite , prompting approaches like chain-of-thought (cot) @xcite offer a promising solution for enhancing comparative reasoning. these approaches leverage llms' exceptional language generation capabilities without incurring significant overhead. nonetheless, llms exhibit arbitrary and erroneous outputs when prompted for comparative reasoning @xcite . specifically, llms demonstrate inconsistency in their comparative analyses of texts. figure 1 (bottom left) provides an example of logically inconsistent llm reasoning using zero-shot cot prompting. the llm's generated explanation initially describes a property as common to the text pair (highlighted in green), but later implies that the same property is a strength of just one of the documents (highlighted in yellow). this inconsistency in the llm's comparative analysis leads to an incorrect prediction. to address these challenges, we present sc 2 , a structured comparative reasoning model that constructs an intermediate structured representation contrasting two text corpora for more accurate text preference prediction, as illustrated in figure 1 . first, sc 2 proposes a set of aspects from text pairs to guide the comparison step. second, sc 2 generates textual comparisons for every aspect. we refer to aspects and comparisons together as intermediate structured representations. to improve the consistency of reasoning (e.g., a contrastive comparison of a aspect should not overlap with a common comparison), sc 2 adopts approximate inference: sc 2 samples multiple responses in generative process and uses a pairwise comparator to select the most consistent intermediate structured representation for final preference prediction. we demonstrate the effectiveness of sc 2 in improving text preference prediction across various aspects only in ß shared in ß and à ## related work prompting large language models llms have recently advanced the state-of-the-art performance across many nlp tasks @xcite @xcite @xcite . these llms have demonstrated the capability to provide chain-of-thought explanations that elucidate their reasoning processes @xcite . however, the chain-of-thoughts generated by llms are often arbitrary or contradictory @xcite @xcite , unfaithful to the facts @xcite or lacking robustness to rephrased questions. to mitigate these issues, several works aim to leverage consistencybased @xcite , or verification-based approaches @xcite to improve the reasoning capacity of llms, yet the benefit of such additional techniques are still ambivalent @xcite . furthermore, all these advanced techniques still concentrate on processing raw-text inputs, thereby overlooking the integration of structural information. moreover, they lack implementations of explicit consistency constraints, which is crucial for maintaining logical coherence in generated outputs. ## methods our model, sc 2 , produces comparative reasoning for text preference prediction that applies to densely written texts, generalizes to multiple domains, and ensures consistency. in this section, we give the generative process and inference procedure for sc 2 . our primary focus is ensuring the comparisons consistently distinguish similarities and differences between texts. ## experimental setup aspect model we experiment with two models for generating aspects: the online aspect model and the offline aspect model. both models use palm-2-l to obtain aspects. the online aspect model dynamically generates aspects using the cot paradigm @xcite to deduce aspects based on text inputs and applies self-consistency @xcite to select the most agreeable aspect for each text pair. however, this model is costly due to the extensive use of llm api calls for every pair of tasks. the offline aspect model extracts aspects from a collection of text corpora, adapting the concept from pang et al. (2021) but employing llms. specifically, this model prompts an llm to extract aspects from each text within the collected corpora (50 pairs of texts for each task in this paper). it then prompts an llm to refine and consolidate all generated aspects. finally, we identify five fixed aspects as to use directly for any text pair of one task. this aspect model significantly reduces costs by allowing offline refinement of aspects. refined aspects are fixed, thus they can be directly used without any additional expense. in our experimental studies, we report only the best results for both baselines and in this section. to understand the impact of the aspect model, we detail its effects in our analysis section. comparison model we use palm-2-l as the major llm backbone of comparison model of sc 2 to produce intermediate structured representations. preference model for the final text preference prediction model, we experiment with two other llm backbones differing in their model capacity. we aim to prove that the intermediate structured representations produced by sc foot_1 with palm-2-l can help any backbone llms to predict text preference more accurately, regardless of their capacity. specifically, we have used openai's gpt-3.5, and gpt-4 (openai, 2023) in our experiments. prompting templates prompts used in different models can be found in our appendix. note that we do not tailor the preference model's prompts; instead, we adapted the templates from @xcite for a fair comparison across baselines 2 . hyperparameters as sc 2 searches for the best comparisons during the inference stage, as a result, we have a hyperparameter |c|, referring to the number of samples generated by the comparison model. baselines for evaluation, we consider several baselines, primarily focused on the llm-based prompting approaches. below is a detailed overview of these baselines: specifically, cot-selfcon first samples multiple responses from an llm using the same prompt and text pair input. subsequently, cot-selfcon aggregates all responses to identify the most frequent answer. in our experimental studies, we set the number of sample responses to 8 and use a majority vote to determine the desired response, randomly selecting a response in the event of a tie. datasets (1) tl;dr @xcite : we use openai's filtered reddit and cnn/daily mail tl;dr dataset. openai also created a preference dataset # samples avg. length tl;dr-cnn/dm 256 572 tl;dr-reddit 259 362 antropic-helpful 250 102 antropic-harmless 249 93 trec news 291 947 avg 278 433 dataset sampling as datasets that have been used in the past are in large volumes, we only sampled a small ratio of them due to the cost of running all experiments. we sample roughly 250-300 data points from each dataset uniformly. more details of the sampled dataset can be found in ## analysis to further understand the benefit of using sc 2 to produce an intermediate structured representation, in this section, we conduct ablation studies and indepth analysis. we also implement a user study to explore the potential of using sc 2 to inform human beings' decisions. ## conclusion this paper presents sc 2 , a structured comparative reasoning model for improving text preference prediction. sc 2 constructs intermediate structured representations to explicitly contrast text pairs, incorporating a consistency comparator to enhance accuracy and coherence. comprehensive experiments across text summarization, retrieval, and response rating tasks demonstrated that sc 2 significantly improves consistency and achieves state-of-the-art performance. analyses confirm the effectiveness of sc 2 's structured reasoning approach and consistency enforcement. our human evaluations show that sc 2 interpretations can assist users in making informed decisions.