R oute N ator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLM s (2025)

Abstract: This paper addresses fine-tuning Large Language Models (LLMs) for function calling tasks when real user interaction data is unavailable. In digital content creation tools, where users express their needs through natural language queries that must be mapped to API calls, the lack of real-world task-specific data and privacy constraints for training on it necessitate synthetic data generation. Existing approaches to synthetic data generation fall short in diversity and complexity, failing to replicate real-world data distributions and leading to suboptimal performance after LLM fine-tuning. We present a novel router-based architecture that leverages domain resources like content metadata and structured knowledge graphs, along with text-to-text and vision-to-text language models to generate high-quality synthetic training data. Our architecture’s flexible routing mechanism enables synthetic data generation that matches observed real-world distributions, addressing a fundamental limitation of traditional approaches. Evaluation on a comprehensive set of real user queries demonstrates significant improvements in both function classification accuracy and API parameter selection. Models fine-tuned with our synthetic data consistently outperform traditional approaches, establishing new benchmarks for function calling tasks.

## introduction digital content creation platforms increasingly rely on natural language interfaces to make complex design tools accessible to non-technical users. a critical challenge lies in accurately translating user queries into appropriate function calls @xcite for instance, when a user requests "find me an image of an elephant with the background being taj mahal", the system must orchestrate multiple api calls for searching, background removal, and compositing. in this paper, we specifically address the challenge of training models to classify user intent into two distinct categories: queries that can be fulfilled through search api operations versus those requiring generation through generative-aipowered apis. given a user query, our model determines: (1) whether to route the request to "search" or "generate" operations based on user intent, (2) the appropriate content type parameter selection (e.g. photo, template, background, video), and (3) prompt optimization specific to each api type-simplifying "search" queries while preserving detailed specifications for "generate" api requests. the examples below in listing 1 demonstrate how our model processes and classifies different types of user queries: listing 1: example function calls for user queries @xmath0: ' find me an image of an elephant ', ' output ': { ' function ': ' search ', ' content_type ': ' photo ' , ' extracted_prompt ': ' elephant ' } } , @xmath1: ' create a birthday invitation for my nephew whose birthday is on january 21 ' , ' output ': { ' function ': ' generate ' , ' content_type ': ' template ' , ' extracted_prompt ': ' invitation for nephew ' s birthday on january 21 ' } } while existing function calling models @xcite show promise, their performance on specialized domains remains suboptimal and privacy restrictions on production data create training challenges. to address these limitations, we present two key contributions: 1. a methodology for incorporating structured domain knowledge into synthetic data generation that leverages: (a) techniques for extracting generalizable patterns from content metadata; (b) methods for utilizing domain-specific knowledge graphs to generate contextually relevant queries. 2. a novel router-based architecture for synthetic data generation featuring: (a) multiple specialized llm prompt templates as distinct routes; (b) a weighted routing mechanism using population-level statistics; our approach (figure 1 ) improves downstream model performance while producing balanced training data across content types, with well-distributed keywords and diverse sentence structures that better align with real-world user interactions. ## related work the challenge of generating high-quality synthetic training data for language models has been explored through various approaches. prior work in generating high-quality synthetic training data for language models spans three key categories: instruction-tuning approaches have shown significant promise, starting with self-instruct's @xcite 175-seed task framework. the field expanded through wizardlm's evolinstruct @xcite , unnatural instructions @xcite , flan @xcite , @xcite , alpaca @xcite , prompt-breeder @xcite , and template-based generation @xcite . multimodal synthetic data generation advanced through visual instruction tuning @xcite , minigpt-4 @xcite , and internvl @xcite , incorporating visual and textual information for enhanced data generation. function calling approaches, exemplified by gorilla @xcite building on self-instruct @xcite , addressed api parameter matching challenges, though lacking domainspecific knowledge integration. our work differs from previous approaches in several key aspects. (1) we focus specifically on generating synthetic data for function calling while maintaining real-world query distributions. unlike general instruction tuning approaches, we target the unique challenges of function calling data generation which include precise parameter matching requirements, maintaining real-world api usage distributions and handling complex nested function calls. (2) we introduce a novel router-based architecture that combines multiple generation strategies. we extend existing router-based approaches by adding weighted probabilistic sampling and by using population-level statistics to guide routing decisions. we also combine text-to-text and vision-totext generation paths. (3) we incorporate domainspecific knowledge while respecting privacy constraints by not directly referring to the real-world datasets. (4) we leverage multiple modalities (text and images) to increase the diversity and quality of generated data, particularly for visual contentrelated apis. we introduce novel evaluation metrics measuring the qualitative alignment of the synthetic data with real-world data covering contenttype alignment, diversity in data types generated, word length variability, and positional variance of key terms within sentences. ## methodology and experiments this section details how our data generation approach evolved. our initial approach employed rule-based templates that combined content metadata with domain-specific knowledge graph (kg) relationships @xcite between different aspects of digital content creation. consider a snapshot of the knowledge graph sub-graph (figure 2 ). this sub-graph consists of interconnected nodes representing user intents ("birthday", "diwali", "baby shower"), design types ("card", "invite"), scene objects ("cake"), and associated actions ("cut a cake"). each edge between the nodes represents a relationship between them. these connections were created using the historical and semantic relationships seen between different entities like user intents, design types, scene objects and actions. they enable the generation of semantically coherent queries by following established relationships between concepts. we create synthetic "search" api data by generating random prompts combining the related entities (e.g. intent and design type) with search synonyms e.g. "find me", "search for", "look for", "search", "show me". ## results and analysis we analyze the results by first looking at the data diversity of the router based synthetic data (word count distribution, content type diversity, positional diversity of keywords and query length distribution) and comparing it with other synthetically generated datasets. we then focus on the performance metrics of different variants of gorilla model across different synthetic datasets. we also show the performance improvement of small language models (slms) fine-tuned on our router based synthetic dataset compared to their base model. ## conclusion llms are fueling efforts to develop systems that accurately interpret user queries and map them to function calls. however, the scarcity of real-world user data and privacy constraints on training with it necessitate synthetic data generation. existing synthetic data generation approaches lack the diversity and complexity needed to mirror real-world interactions, limiting model performance. we introduced a novel architecture for generating highquality synthetic training data. our approach integrates content metadata and domain-specific kgs with text-to-text and vision-to-text models, producing more varied and representative data. through iterative development, we arrived at a router-based multi-modal architecture that enhances data diversity and improves model training outcomes. our model demonstrates gains in function mapping accuracy, although further improvement is possible in content-type classification.