Hierarchical Indexing for Retrieval-Augmented Opinion Summarization (2024)

Abstract: We propose a method for unsupervised abstractive opinion summarization, that combines the attributability and scalability of extractive approaches with the coherence and fluency of Large Language Models (LLMs). Our method, HIRO, learns an index structure that maps sentences to a path through a semantically organized discrete hierarchy. At inference time, we populate the index and use it to identify and retrieve clusters of sentences containing popular opinions from input reviews. Then, we use a pretrained LLM to generate a readable summary that is grounded in these extracted evidential clusters. The modularity of our approach allows us to evaluate its efficacy at each stage. We show that HIRO learns an encoding space that is more semantically structured than prior work, and generates summaries that are more representative of the opinions in the input reviews. Human evaluation confirms that HIRO generates significantly more coherent, detailed, and accurate summaries.

## introduction online review websites are a useful resource when choosing which hotel to visit or which product to buy, but it is impractical for a user to read hundreds of reviews. automatic opinion summarization aims to aggregate a large and diverse set of customer reviews about a particular entity into a single, easy to read summary. a good summary should accurately reflect the balance of opinions in the input reviews, highlighting the most common or popular opinions, while omitting unnecessary details. a useful summary should also help compare between competing options, and include points that differentiate the current entity from others. early work on opinion summarization extracted reviewers' sentiment about specific features @xcite or identified salient sentences based on centrality @xcite , while more recent methods have proposed extractive models that use learned feature spaces @xcite . prior work on abstractive opinion summarization has almost exclusively either required costly supervision @xcite or has assumed that the number of input reviews is limited @xcite @xcite . this defeats the point of a summary: a user could feasibly read 8 reviews in a reasonable period of time. a good summarization system should be scalable, since popular products online may receive thousands of reviews. it should also be attributable, offering some evidence to justify its output. paraphrasing @xcite , we say that a statement s is attributable to some evidence e, if a generic reader would agree that 'according to e, s is true'. finally, it should generate summaries that are coherent and faithful to the input reviews. large language models (llms) have been shown to generate highly fluent summaries in the news domain @xcite but are a flawed solution because current instruction-tuned models are not attributable, and because their context windows limit the number of reviews they are able to base their summaries on. models with long context windows have been proposed @xcite but these are not currently instruction-tuned, and it has been shown that llms are biased toward information at the start and end of the input @xcite . our approach, hierarchical indexing for retrieval-augmented opinion summarization (hiro), identifies informative sentences using hierarchical indexing and then passes the selected sentences as input to a llm, similar to retrieval-augmented generation @xcite . by separating the steps of content selection and generation, we can combine the attributability and scalability of the discrete representation with the strong generative abilities of llms, leading both to a higher quality index and to more informative and coherent output summaries. hiro consists of three modules, allowing for increased control, flexibility and interpretability. the hierarchical indexer is an encoder that maps sentences from reviews to paths through a hierarchical discrete latent space. the retriever uses the index to identify clusters of sentences for each entity that contain popular and informative opinions. these sentence clusters are passed to a generator, a pretrained llm, that generates coherent summaries that are grounded in the retrieved sentences. our contributions are as follows: our code and dataset splits are available at https://github.com/tomhosking/hiro . louis and maynez (2023) use a natural language inference (nli) model to construct 'silver standard' summaries to use as training data for fine-tuning a pretrained language model (t5, @xcite . however, their approach is computationally very expensive, calculating over 1b pairwise entailment relations between sentences in the training data, before fine-tuning a llm. by contrast, hiro uses a lightweight indexing encoder, combined with an off-the-shelf llm that is prompted in a zero-shot manner. other work has used specialized training data to train supervised models (e.g., @xcite ; however, such data is expensive to collect and is not generally available in every language, domain, or setting. ## overview let r e be a set of reviews about an entity @xmath0, where each review @xmath1. the goal is to generate a textual summary that includes the most informative opinions from r e , while abstracting away the details specific to any one review. hiro generates summaries following a modular approach, depicted in figure 1 . we learn an index structure that maps each sentence x from the input reviews to a path q 1:d through a discrete hierarchy. we choose a hierarchical representation so that sentences are grouped at a useful level of abstraction; the upper levels of the hierarchy should partition sentences by topic, while the lowest levels should group together sentences with equivalent meaning. at inference time, we encode all sentences from the reviews, then identify the paths or subpaths q 1:d within the index that are particularly popular, and retrieve the corresponding sentences. this selection process is query-free, instead relying on properties of the hierarchical index to determine the frequency of different opinions. by indexing sentences hierarchically according to their semantics, we can easily identify opinions or topics that occur frequently by simply counting their occurrence in the index. finally, we generate a summary by passing the retrieved clusters of sentences as input to a llm. this retrieval-augmented usage of llms allows us to benefit from the fluency and coherence of llms, while retaining the attributability and scalability of extractive opinion summarization methods. we now detail the three modules that constitute hiro, evaluating each of them in turn to confirm their efficacy compared to previous methods. ## learning a hierarchical indexing our goal is to learn an encoding space where sentences with similar meanings are grouped together. the space should be discretized so that frequent opinions can be easily identified by counting the membership of each part of the index, and it should be hierarchical so that opinions may be aggregated at an appropriate level of granularity, rather than by details or phrasings specific to a particular review. finally, the encoding space should be structured semantically, to enable accurate aggregation of opinions; sentences with equivalent meaning should clearly be indexed to the same point in the hierarchy, while sentences that are topically related but not equivalent should be grouped together at a higher level. we base our encoder on hercules @xcite . hercules uses an encoder-decoder architecture with a discrete hierarchical bottleneck to generate summaries. it is trained as a denoising autoencoder, and therefore needs to learn a representation that is both compressed enough to enable aggregation, but also expressive enough for the decoder to be able to generate meaningful output. these factors are in direct competition, with the compressed bottleneck leading to output that is generic and contains hallucinations. by contrast, hiro uses an external llm as the 'decoder', allowing us to focus on learning a representation that is useful for identifying informative opinions. ## retrieving popular opinions recall that a good summary should include opinions that occur repeatedly in the input reviews, as well as opinions that differentiate the current entity from others. in section 4 we showed how the hiro encoder maps a single sentence x to a path q 1:d . we now exploit the discrete, hierarchical nature of the representation space to index a large number of review sentences, then identify informative sentences to use to generate a summary. we hypothesize that our content selection method leads to clusters that better represent the balance of opinions in the input reviews. ## generating coherent summaries given the retrieved clusters of sentences for each entity, we want to generate a coherent and fluent textual summary. llms are well suited to this constrained rewriting task, and we leverage the zero-shot abilities of instruction-tuned models to map clusters of sentences to a readable summary. ## conclusion we propose hiro, a modular method for unsupervised opinion summarization that uses a hierarchical index over sentences to select clusters of prevalent opinions. our approach leverages pretrained large language models to generate coherent and fluent summaries that are attributable and accurately reflect the popularity of opinions in the input reviews. extensive experiments show that, as well as generating higher quality summaries, hiro learns a more semantically distributed representation than competitive baselines. while we limit our experiments to opinion summarization, we believe that hiro could be usefully applied to a wide range of other retrieval-augmented generation tasks.