A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generation (2024)

Abstract: Generating natural language text from graph-structured data is essential for conversational information seeking. Semantic triples derived from knowledge graphs can serve as a valuable source for grounding responses from conversational agents by providing a factual basis for the information they communicate. This is especially relevant in the context of large language models, which offer great potential for conversational interaction but are prone to hallucinating, omitting, or producing conflicting information. In this study, we conduct an empirical analysis of conversational large language models in generating natural language text from semantic triples. We compare four large language models of varying sizes with different prompting techniques. Through a series of benchmark experiments on the WebNLG dataset, we analyze the models’ performance and identify the most common issues in the generated predictions. Our findings show that the capabilities of large language models in triple verbalization can be significantly improved through few-shot prompting, post-processing, and efficient fine-tuning techniques, particularly for smaller models that exhibit lower zero-shot performance.

## introduction accessing structured information through natural language interfaces has garnered significant research interest in natural language processing (nlp) @xcite . for instance, the emerging information retrieval paradigm of conversational search frames information-seeking processes within multiturn dialogue interactions. conversational search facilitates exploring and progressively narrowing the search scope to relevant knowledge items within an information space. these search-oriented conversational interfaces are often connected to structured data sources like knowledge graphs. however, a key challenge lies in mediating be-tween natural language, in which users express their queries, and machine-readable knowledge representations. the task of data-to-text generation focuses on this issue, taking structured data as input to produce coherent, human-readable text, which has been extensively studied with approaches ranging from rule-based to supervised neural networkbased techniques. over the last years, the field of nlp has witnessed a shift in methodologies with the advent of pre-trained large language models (llms). unlike traditional supervised learning approaches that rely on annotated datasets, llms are trained in a self-supervised manner, predicting tokens within vast amounts of unlabeled data. combined with scaling up the model size and training corpora, this approach has demonstrated remarkable emergent capabilities of llms and their prowess in multitask learning @xcite ). an advantage of llms lies in prompt-based (in-context) learning. through carefully defined prompts, these foundation models can perform multiple tasks like question-answering, semantic parsing, or text summarization @xcite . more recently, there has been a growing interest in optimizing llms for conversational interactions by pre-training on dialogue corpora, instruction finetuning, and reinforcement learning from human feedback @xcite . although llms offer tremendous potential for conversational interaction, owing to their ability to produce responses for arbitrary input, they have known limitations, such as the risk of hallucinating or omitting important information and a lack of transparency regarding the origins of information sources from which they derive their outputs @xcite . in order to mitigate these limitations, it becomes imperative to ground their generated outputs in verifiable factual data from knowledge graphs. however, there has been insufficient systematic investigation into their pro-ficiency in verbalizing graph-structured data input. to assess llms in knowledge-based text generation, we compare four models of different sizes and training objectives, with a primary focus on models optimized for conversational interaction. based on the popular webnlg benchmark dataset, we evaluate the models' performance in generating natural language text from semantic triples. through multiple experiments, we analyze different configurations of models and prompting techniques, discussing insights about their individual capabilities and limitations. our contributions include: ## related work existing works from the nlp literature have explored knowledge-based text generation, with significant advancements driven by new deep learning architectures and fine-tuning language models on downstream tasks @xcite . for triple-to-text generation, many evaluations use the established webnlg benchmark @xcite . several studies have focused on comparing neural pipeline versus endto-end approaches, assessing supervised versus unsupervised training regimes, and developing frameworks for making text generation more controllable through neuro-symbolic methods @xcite @xcite . concerning pre-trained language models, chen et al. (2020) were among the first to propose the task of few-shot natural language generation. with just 200 table-to-text training examples, their approach achieves strong performance and good generalization. by collecting a novel dataset and experimenting with few-shot fine-tuning, @xcite demonstrate that pre-trained language models trained with a diverse set of labels exhibit robustness in verbalizing knowledge graph relations, being capable of generalizing to novel domains. another study from @xcite al. (2021) highlights 1 github: https://github.com/sebischair/llm-kg-d2t the ability of pre-trained language models (plms) to uncover hidden mappings between linguistic tokens and real-world concepts. conducting experiments on four datasets, the authors show the effectiveness of their awakening latent grounding approach for generating structured queries from text. similar to our work, @xcite assess capabilities of llms but for text-to-graph generation with the gpt-3.5-turbo model. they develop a prompting framework with iterative verification, improving the generation quality. in contrast, our objective is to achieve a comprehensive understanding of conversational llms for triple verbalization rather than solely concentrating on individual use cases or models. to the best of our knowledge, we are the first to conduct a comparative analysis of conversational llms and prompt configurations on the task of triple-to-text generation. the empirical approach employed in this study is related to our previous work on evaluating llms for semantic parsing for conversational question answering over knowledge graphs @xcite . ## experiments experimental setup we conduct our experiments on the webnlg+ 2020 dataset, a dbpediabased triple-to-text benchmark with a total of 1,779 test examples @xcite . as evaluation metrics, we calculate the lexical similarity between model outputs and human annotations using bleu @xcite , meteor @xcite , and ter @xcite . since these metrics mainly focus on lexical overlaps, we also use the bertscore metric, which captures semantic similarity @xcite . as a commercial state-of-the-art llm, we include gpt-3.5-turbo (chatgpt) (openai, 2022) in our comparison. it is optimized for conversations and has demonstrated remarkable zero-shot performance on various nlp tasks. consequently, it is often used as a benchmark for comparing llms. we ran our experiments with the model released in june 2023 (gpt-3.5-turbo-0613). further, we opted to test llama, a collection of open-source llms from meta @xcite , achieving competitive performance on various benchmarks. we include three model variations with 7b parameters of the first llama version. in addition to the non-conversational base model (llama-7b), we included a fine-tuned model (llama-ft-7b) which we trained on webnlg examples in a conversational format. to have a sufficiently large fine-tuning corpus, we created a new dataset encompassing 26,422 conversations from all 13,211 webnlg training examples. we ensured that each triple-to-text example appeared, on average, five times in different contexts. the conversations have different lengths and contain verbalizations from various triple categories. the training was done through low-rank adaptation (lora), a method that fine-tunes only a subset of the model's parameters, referred to as low-rank matrices, rather than updating the entire parameter space, improving the fine-tuning efficiency @xcite . during training time, the model takes in a full conversation in chat completion format, characterized by a series of turns attributed to the user or assistant role (i.e., the model learns from a sequence of sequence-to-sequence examples). we employed five training epochs, a per-device training batch size of eight, and used a half-precision floatingpoint format (fp16). another fine-tuned llama model we compared is vicuna. it was trained on a corpus of around 70k user-shared chatgpt conversations crawled from the sharegpt website. preliminary evaluations from @xcite demonstrate that vicuna exhibits a higher level of detail and structure in its responses than llama, highlighting the advantage of fine-tuning on dialogue data. the llama and vicuna models are prompted in the chat completion structure of the fastchat 2 platform, replicating openai's chat completion api endpoint with a structured list of system, user, and assistant messages. we set the token limit to 128 and the temperature parameter to 0, maximizing deterministic generation by favoring high-probability words. the zero-shot prompt contains only the following system mes-2 fastchat: https://github.com/lm-sys/fastchat sage with a triple verbalization instruction: "sys-tem: generate a concise text for the given set of triples. ensure that the generated output only includes the provided information from the triples.". the few-shot prompt expands the instruction with three in-context examples provided as user and assistant messages in the format: "user: input triples: [{'object': 'mike_mularkey','property': 'coach','subject': 'tennessee_titans'}] "assistant: output text: mike mularkey is the coach of the tennessee titans." results of performance metrics examining the scores, llama-ft-7b demonstrates superior performance compared to the other models. even without few-shot examples, it effectively learned from fine-tuning to handle the triple verbalization task, gaining only a minor performance increase through few-shot prompting. the second-ranking model, gpt-3.5-turbo, shows similar scores, which is remarkable because it was not explicitly trained for triple-to-text generation. notably, vicuna achieves a performance level almost on par with the much bigger gpt-3.5-turbo model when it was provided with in-context examples and the output was post-processed. in the zero-shot setting, vicuna could not match the scores of gpt-3.5-turbo but outperformed llama-7b. although llama is the worst-performing model, it claims the most significant improvements through fewshot prompting and post-processing, with scores not too far from vicuna. the metrics collectively suggest that all tested llms can generate reasonable output text from knowledge graph triples. besides, we observe that while all models show improvements with few-shot prompting or postprocessing, models trained on conversations like vicuna require less post-processing and exhibit better zero-shot proficiency, resulting in comparatively smaller performance gains from post-processed outputs or in-context examples. analysis and discussion our experiments reveal that llms, especially those fine-tuned on conversations, are capable of triple-to-text generation without explicit training. however, as expected, the fine-tuned llama-ft-7b model achieved the best overall performance. the webnlg triple verbalization task involves different subtasks, such as segmentation of the input data, lexicalization of the dbpedia properties, information aggregation, and surface realization of grammatically correct text @xcite . all of these subtasks are handled by llms in an end-to-end manner. in direct comparison to state-of-the-art models evaluated on webnlg like control prefixes (bleu: 0.62, meteor: 0.45, ter: 0.35) from @xcite or t5-large+wiki+position (bleu: 0.61, meteor: 0.44, ter: 0.36, bertscore: 0.96) from @xcite , the llms' lexical similarity metrics are worse. yet, when looking at semantic similarity, the bertscore metric of the llama-ft-7b model is identical at 0.96. we hypothesize that the lower lexical similarity is partly caused by the concise writing style of the webnlg human ground truth verbalizations, aggregating as much information as possible in succinct sentences. while many webnlg annotations are as short as possible (e.g., "the 98.0 minute film super capers starring danielle harris was written by the director ray griggs."), the more verbose output of llms like gpt-3.5-turbo consists of multiple sentences (e.g., "danielle harris stars in the movie super capers. the writer of the movie is ray griggs. the movie has a runtime of 98.0 minutes."). this concise writing style can be better learned and replicated by llama-ft and other fine-tuned models. we also observed that the llms had a tendency to occasionally use passive voice, initiating sentences 1xpehuri7ulsohv %/(86fruh ## conclusion we compared the abilities of llms in knowledgebased text generation. our results indicate that even smaller 7b-llms exhibit reasonable performance in verbalizing triples, conveying intended meanings and facts in a coherent manner, although they might not always be factually accurate or perfectly replicate the writing style of human annotations. we also discussed model-specific differences and common generation issues that can be mitigated through few-shot prompting or fine-tuning. in future work, we plan to investigate how our findings generalize to more complex graph data structures.