C onv KGY arn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models (2024)

Abstract: The rapid evolution of Large Language Models (LLMs) and conversational assistants necessitates dynamic, scalable, and configurable conversational datasets for training and evaluation.These datasets must accommodate diverse user interaction modes, including text and voice, each presenting unique modeling challenges. Knowledge Graphs (KGs), with their structured and evolving nature, offer an ideal foundation for current and precise knowledge.Although human-curated KG-based conversational datasets exist, they struggle to keep pace with the rapidly changing user information needs.We present ConvKGYarn, a scalable method for generating up-to-date and configurable conversational KGQA datasets. Qualitative psychometric analyses demonstrate ConvKGYarn’s effectiveness in producing high-quality data comparable to popular conversational KGQA datasets across various metrics.ConvKGYarn excels in adhering to human interaction configurations and operating at a significantly larger scale.We showcase ConvKGYarn’s utility by testing LLMs on diverse conversations — exploring model behavior on conversational KGQA sets with different configurations grounded in the same KG fact set.Our results highlight the ability of ConvKGYarn to improve KGQA foundations and evaluate parametric knowledge of LLMs, thus offering a robust solution to the constantly evolving landscape of conversational assistants.

## introduction the proliferation of llms and conversational assistants in daily user interactions comes with the need for dynamic datasets to stress-test their ability to handle evolving knowledge-seeking questions. kgs have long been recognized for capturing structured representations of the world @xcite . they represent concepts and entities as nodes, while edges form semantic relationships to * work done while at apple define facts. kgs have strong roots in various fields, including natural language processing @xcite , recommender systems @xcite , and information retrieval @xcite . integrating llms with kgs has advanced several nlp tasks @xcite @xcite @xcite . this synergy unlocks new avenues for conversational kgqa scenarios like those targeted by convquestions @xcite . convquestions highlights the potential of combining llms with kgs for accurate and attributed responses in conversations @xcite . recent advancements in text retrieval have demonstrated the efficacy of llm-generated synthetic data in enhancing downstream systems, from query synthesis @xcite @xcite and llm-based ranked list reorderings @xcite to training highly effective smallscale models through automated prompt optimization @xcite . these developments underscore the opportunity to leverage synthetic data strategies from llms. however, existing qa datasets lag behind evolving user needs. we introduce convkgyarn, a method for generating large-scale configurable conversational kgqa datasets. psychometric evaluation show convkgyarn produces high-quality conversational data, scaling entity and fact coverage while incorporating diverse user interaction styles. evaluating convkgyarn-generated datasets with various llms reveals their struggle with fact recall, emphasizing the need for retrievalaugmented systems. model effectiveness varies across different user interaction styles, highlighting the importance of building adaptable llms. through this work, we aim to shed light on building evolving datasets that can train and test conversational assistants of the future. 2 convkgyarn figure 1 illustrates the entire pipeline of the conv-kgyarn system. we first introduce the key notations and definitions to set our terminological and conceptual framework. next, we dive into each module that comprises the convkgyarn system. ## experimental setup we use a wikidata dump with a june 2023 knowledge cut-off for all our experiments. the dump, with roughly 100m entities, was filtered to include only english entity names and interesting types, resulting in 29m entities with 196m facts. for the llm predicate selector, we used the gpt-4-0613 endpoint. we query at most 50 predicates to avoid overwhelming the model, processing each type-predicate pair segment by segment. for predicates with linked qualifiers, we include the relationship predicate in the input, selecting those relevant for conversational factoid qa. for synthetic question template generation, we use the gpt-3.5-turbo endpoint, providing two in-context examples per prompt to align generations with the expected template format. to handle textual interactions, we utilize the "logit_bias" field to penalize the model when it generates question words (wh-words or how), ensuring adherence to instructions and in-context examples. for typo augmentation, we apply one of the following textattack attacks @xcite at random to each question turn: wordswaprandomcharacterdeletion(), wordswapneighboringcharacterswap(), or wordswapqwerty(). each question turn receives a single "meaningful" typo. we introduce a single "meaningful" typo to each question turn. ## dataset statistics the general set from convkgyarn comprises 29m entities and 196m facts from filtered wikidata, excluding related entities. each fact can generate 24 possible questions: 12 from voice interactions (three each from original, deixis, disfluencies, and deixis_disfluencies sets) and 12 from textual interactions (three each from original, deixis, typos, and deixis_typos sets). this enables diverse conversation generation, providing a large-scale resource for training conversational agents and exposing language models to high-quality synthetic data. the dataset includes 274 unique types and 1252 unique predicates, enhancing the complexity and realism of factoid conversations. this scale and coverage surpass human-curated datasets like convquestions @xcite , which contain 11k real-user conversations averaging five questions each, limited to five primary entity types. in contrast, the related set focuses on popular human-type entities, containing 210k entities and 6.1m facts. despite its smaller scale, it offers a high density of interconnected information with an average of 54 questions per fact (an additional 30 from related entity-specific follow-up questions). this set includes 95 unique types and 265 unique predicates, providing a targeted dataset for detailed exploration and evaluation of conversational systems focused on human-centric entities. ## results to evaluate convkgyarn's efficacy, we employ three complementary methods: (1) single-model rating, (2) pairwise comparison, and (3) parametric knowledge evaluation of llms. single-model rating, using likert scores, offers scalability but has limitations. it relies on absolute judgments, which can be less reliable than relative comparisons @xcite and lead to biases among annotators @xcite . pairwise comparison mitigates these issues by facilitating relative judgments. however, it becomes less efficient when comparing multiple models, often requiring re-evaluation of baselines upon introducing new models @xcite . lastly, we assess the effectiveness of llms on convkgyarn-generated conversational factoid qa datasets, examining their fact recall abilities through llm-as-a-judge evaluation. while this scales well and often correlates strongly with human annotations, it may suffer from selfenhancement bias, where llms favor their own generated answers @xcite . this multifaceted approach ensures a comprehensive evaluation of convkgyarn from both human and automated perspectives, leveraging each method's strengths to offset others' weaknesses. ## conclusions in this paper, we introduced convkgyarn, a novel framework to generate dynamic and scalable conversational datasets for kgqa. our system leverages the structured representation of kgs to produce conversational datasets that can adapt to the evolving information needs of the user and knowl-edge captured by kgs. our extensive evaluations demonstrate that convkgyarn effectively generates well-configured high-quality kgqa datasets. by conducting rigorous qualitative and quantitative tests, we showcased that the datasets generated are versatile across various conversational scenarios, allowing us to test models on their effectiveness with different facets of user interactions and linguistic phenomena. furthermore, psychometric analyzes highlighted that the conversations generated from convkg-yarn were comparable to those from traditional human-curated datasets, scoring highly on the metrics of relevance, fluency, cohesiveness, and grammar (when targeting these attributes) while being a few orders of magnitude larger in scale. an important finding from our work is the adaptability of convkgyarn to handle different types of interactions, such as text and voice, by appropriately configuring conversations to fit criteria and attributes such as deixis, disfluencies, and typos. in addition, our system's ability to dynamically integrate updates from kgs ensures that the conversations remain current and factually accurate, addressing one of the significant challenges in existing conversational kgqa datasets. convkgyarn enhances the testing capabilities of llms and qa systems in adapting to the evergrowing knowledge landscape and also facilitates high-quality evaluation across different forms of user interactions, each with their nuances.