Optimizing Cost-Efficiency with LLM -Generated Training Data for Conversational Semantic Frame Analysis (2025)

Abstract: Recent studies have shown that few-shot learning enables large language models (LLMs) to generate training data for supervised models at a low cost. However, for complex tasks, the quality of LLM-generated data often falls short compared to human-labeled data. This presents a critical challenge: how should one balance the trade-off between the higher quality but more expensive human-annotated data and the lower quality yet significantly cheaper LLM-generated data? In this paper, we tackle this question for a demanding task: conversational semantic frame analysis (SFA). To address this, we propose a novel method for synthesizing training data tailored to this complex task. Through experiments conducted across a wide range of budget levels, we find that smaller budgets favor a higher reliance on LLM-generated data to achieve optimal cost-efficiency.

## introduction it is costly to construct training data with human annotation for supervised learning models (slms). in recent years, large language models (llms) like gpt-4 have demonstrated remarkable abilities in generating coherent text, understanding context, and following complex specifications to accomplish tasks @xcite . therefore, there have been many attempts to leverage existing llms as data synthesizers to generate training data for slms, aiming to reduce data costs. studies have indicated that using llm-generated data can cut costs significantly while achieving a comparable performance against human-annotated data for certain tasks @xcite . in this paper, we explore the feasibility of synthesizing training data for conversational semantic frame analysis (sfa). sfa captures knowledge exchanged between speakers by extracting semantic frames, which consist of a trigger (the main action) is it okay to use high heat? yes, that will be fine. and its arguments (details of the event). for example, in figure 1 , the triggers "line up" (place) and "fry" (bake_fry) are annotated, with corresponding arguments like object, time, and temperature linked to them. an important characteristic of these dialogues is the frequent repetition and confirmation of technical details. for example, in figure 1 , the interviewer's question introduces a new argument to an existing frame. refer to figure 12 in appendix for a longer and more complex annotation example. we expect llm-generated data for sfa to be of lower quality than human-annotated data, as sfa is significantly more complex than the tasks typically addressed in previous llm-based data synthesis studies @xcite @xcite . these studies have primarily focused on simpler tasks such as sentence-level labeling, extracting relation triplets, or tasks with fewer recurring entities and relations. furthermore, @xcite demonstrated that few-shot llms generally underperform in many information extraction tasks, such as named entity recognition, compared to supervised baselines. given these findings, it is reasonable to expect that llm-generated data for sfa will also be of lower quality than human-annotated data. given that llm-generated data for sfa may we mixed human data and llm-generated data to create data mixtures up to a specific budget. the ratio of human data to llm-generated data was adjusted in increments of 0.1. these data mixtures were then used to train our sfa model to identify the ratio that achieves optimal cost-efficiency. we propose a novel method for synthesizing training data using an llm for the challenging task of sfa, generating two types of data: human-pseudo (hp) and pseudo-pseudo (pp). pp data comprises pseudo-dialogues and pseudo-labels that are both synthesized by an llm, whereas hp data combines human dialogues sampled from a humanannotated dataset with pseudo-labels generated by the llm. by comparing the performance of models trained on hp and pp data, we aim to determine whether the text component (dialogues) or the label component plays a more critical role in improving sfa performance in this situation. our empirical results reveal a clear trend across various budget levels: as the budget decreases, the optimal ratio shifts toward relying more on llmgenerated data. conversely, when the budget is sufficiently large, incorporating llm-generated data can actually harm performance. another key contribution of our work is the direct comparison between hp and pp data. our findings demonstrate that pp data is highly competitive with hp data, indicating that, in this context, replacing humangenerated text with llm-generated text is a viable and cost-effective option. we believe our findings can be applied to sfa in other technical domains or similar tasks (e.g., frame semantic parsing). frame semantics can be used to capture critical information in dialogue situations. @xcite proposed using frame semantics in the domain of disaster response. the extracted information is used to capture and interpret verbal team communication for mission process assistance. @xcite tackled argument detection in a multi-sentence setting to better capture events that span across sentences, which is similar to our setting that is done on the dialogue level. in this study, we focus on conversational sfa in japanese interview dialogues, using the cooking section of the eidc dataset @xcite for the experiments and analyses. supervised learning models (slms) for sfa previous studies have employed probabilistic model @xcite and rnn-based model @xcite as slms for fsp. @xcite introduced transformerbased models @xcite to fsp, utilizing a sequence-to-sequence transformer model and framing fsp as a text generation task by tagging entities with index numbers for tokens. in @xcite , an encoder transformer model was used to address sfa in a cascaded manner: first, a trigger detection model identifies triggers within the context, and then a separate argument detection model determines the arguments for each trigger. however, we are concerned that this cascaded approach might introduce error propagation. therefore, in this paper, we adopt jamie @xcite , an encoder-centric model that simultaneously detects entities and their relations, offering an end-to-end solution for sfa. llms for sfa-like tasks while no existing work directly targets sfa using llms, recent studies have explored related tasks, such as named entity recognition (ner) and relation extraction (re). @xcite reformulated ner as a textgeneration task by wrapping entities in tag pairs, allowing llms to process them efficiently. @xcite and @xcite enhanced llm performance on re tasks by improving prompt design. @xcite tackled various nlp tasks, including ner and re, by utilizing improved prompting and few-shot retrieval methods, similar to the approaches in @xcite and @xcite . these studies, along with the method proposed by @xcite , have inspired our prompt design for sfa using an llm (figure 4 ). llms as data synthesizers there have been numerous efforts to utilize llms for generating synthetic data to train slms. @xcite utilized few-shot gpt-3 to generate labels for natural language understanding and generation tasks, achieving performance comparable to human labeling while significantly reducing costs. @xcite explored various methodologies for generating labeled data using gpt-3 and demonstrated results on par with human-labeled data in tasks such as sentiment triplet extraction. @xcite employed gpt-3.5 with chain-of-thought reasoning @xcite as an alternative to crowdsourced annotators, demonstrating performance that was either superior to or on par with human annotators. however, these studies focus on tasks that are less complex than sfa. they either involve a single label per sequence, extract fewer entities, or do not include relations. additionally, they do not provide an analysis of the trade-off between human and llm-generated data. ## preliminaries we define semantic frame analysis (sfa) and introduce the eidc dataset @xcite , which contains sfa annotations and is used in this study. ## data synthesis with an llm this section presents our methodology for constructing training data for conversational semantic frame analysis using an llm. ## experimental settings to study how to achieve optimal cost-efficiency by collecting both human and llm-generated data with a fixed budget, we conducted the following steps for the experiments. in the following sections, we provide detailed descriptions of these steps. an overview of the costefficiency analysis is demonstrated in figure 2 . ## results and analyses the objectives of the cost-efficiency analysis are as follows: we analyze the experimental results to answer these objectives in the following sections. ## conclusion in this paper, we conducted a comprehensive analysis to evaluate the cost-efficiency to combine llm-generated data with human-annotated data for japanese conversational semantic frame analysis under various budget constraints. we proposed a novel method to synthesize two types of training data: human-pseudo (hp) data and pseudo-pseudo (pp) data, for the experiments and analyses. our findings indicate that the ideal ratio to combine human and llm-generated data is budget-dependent, with a tendency to favor a higher proportion of llm-generated data as the budget decreases. furthermore, our results suggest that fully synthesized data (pp data) is a viable option, as it is significantly cheaper while maintaining comparable performance levels to the half-synthesized counterpart (hp data). in future work, we aim to extend our analysis to other domains and tasks to validate the generalizability of our findings.