Generating Hotel Highlights from Unstructured Text using LLM s (2024)

Abstract: We describe our implementation and evaluation of the Hotel Highlights system which has been deployed live by trivago. This system leverages a large language model (LLM) to generate a set of highlights from accommodation descriptions and reviews, enabling travellers to quickly understand its unique aspects. In this paper, we discuss our motivation for building this system and the human evaluation we conducted, comparing the generated highlights against the source input to assess the degree of hallucinations and/or contradictions present. Finally, we outline the lessons learned and the improvements needed.

## introduction it is crucial to provide updated and accurate content so that travellers can make informed choices about which accommodation to book. content such as images, descriptions, reviews, facility and amenity information, and maps helps travellers compare different accommodations to determine their suitability. given the diversity of content, it is not immediately apparent why a traveller should choose one accommodation over another. while images, descriptions, and reviews can help, they require travellers to extensively analyse and then come up with an assessment before making decisions. this can be challenging for travellers as content styles between accommodations are not uniform. reviews, for example, can often be terse and written in various styles, with travellers only selectively mentioning aspects from their own perspective. descriptions, on the other hand, while more objective, can be quite verbose and may also selectively mention aspects from the perspective of the hotelier. past systems such as the sure @xcite and hotel scribe @xcite have focused more on either summarising opinions or describing an accommodation instead of surfacing unique aspects. to streamline information access for travellers, we developed the hotel highlights project. these highlights are concise, one to two sentences summarising an accommodation's unique selling points, derived from traveller reviews and descriptions, allowing travellers to quickly grasp a property's distinctiveness. to accomplish this, we will discuss the challenges of using llms for summarisation ( §2). afterwards, we will explain our system implementation for generating hotel highlights ( §3). we then describe our human evaluation ( §4) and the results obtained ( §5). finally, we will discuss our conclusions from the findings obtained and potential future work ( §6). ## llms and summarisation until very recently, fine-tuning pre-trained models, such as bart @xcite , on domainspecific datasets has been seen as the leading paradigm for text summarisation @xcite . however, the rise of very large language models (llms) and the success of prompting these models have shown an alternative approach with these models being able with only a few demonstrative examples to generate convincing summaries without the need for updating model parameters @xcite . when evaluated with human evaluators, there seems to be a strong preference for summaries generated by llms like gpt-3 @xcite . this has led some to declare that the task of summarisation is "almost dead" due to the ability of llms to consistently outperform summaries generated by fine-tuned models @xcite or, in other cases, be on-par with human summarisation @xcite . however, the reasons for their success is not well understood. another area of focus has been trying to understand how faithful a model is to the input it has summarised. a model that hallucinates cannot be considered faithful. @xcite define two types of hallucinations: intrinsic hallucinations, where the model misrepresents facts from the input, and extrinsic hallucinations, where the input is ignored and the extraneous text has no relation to the input. for the remainder of the paper, to prevent any confusion with extrinsic and intrinsic evaluation methods, we will use the term "contradiction" to refer to intrinsic hallucination and "hallucination" to refer to extrinsic hallucination. while automatic metrics, such as rouge, are commonly used to evaluate textual similarity, they are inadequate for assessing faithfulness. this inadequacy arises because a high degree of similarity does not necessarily imply faithfulness @xcite . therefore to evaluate the factual accuracy of generated texts, it is necessary to have a robust human evaluation methodology in place @xcite . ## system implementation we created a minimum viable system with data selection, generation with llms and post processing, illustrated in figure 1 . ## human evaluation we conducted a human evaluation experiment to better understand the quality of the generated highlights. we sampled 40 accommodations by limiting descriptions between 400 and 1000 characters in length. description length was restricted to minimise annotators' cognitive load while still containing a decent amount of information about the accommodation. from each accommodation, we selected three highlights, resulting in a total of 120 highlights evaluated in this experiment (40 accommodations * 3 @xmath0. figure 2 shows an example of a hotel description, along with highlights with no divergence, hallucination, or contradiction. ## results out of 119 participants (whole group), 84 answered the validation question with hallucination (henceforth, the success group), 13 with contradiction, 19 with both, and 3 with no divergence. in the remainder of this section, results will be reported for the whole group, with references to the success group when there are noticeable differences. in more than half of the cases (53.22%), participants did not detect any divergence in the highlights. among the cases marked as divergent, hallucinations were the most common (23.39%), followed by contradictions (13.67%), and lastly both hallucination and contradiction (9.72%). furthermore, we evaluated the average rating scores for each intrinsic feature across the four batches. the results showed that grammaticality consistently received the highest ratings. notably, batch 3 received the lowest ratings on all questions, which may suggest differences in the participants or the difficulty of the questions. detailed per-batch results can be found in appendix b. ## conclusion & future work as perceived by annotators, while 53.22% of cases show no divergence, there is still a significant number of hallucinations and contradictions, with the majority coming from the location theme as compared to other objective themes. given the low inter-annotator agreement, this suggests that even with training, the task of evaluating divergences is difficult. an observation also seen by @xcite in trying to obtain high agreement with not just crowd workers, but also with experts. we expected that highlights with divergences would receive lower intrinsic ratings, and this expectation was confirmed in the evaluation. additionally, the average rating of grammaticallity is relatively higher compared to the other intrinsic qualities, which aligns with the assumption that llms have high grammatical correctness.