Visual Editing with LLM -based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications (2024)

Abstract: We present a practical distillation approach to fine-tune LLMs for invoking tools in real-time applications. We focus on visual editing tasks; specifically, we modify images and videos by interpreting user stylistic requests, specified in natural language (“golden hour”), using an LLM to select the appropriate tools and their parameters to achieve the desired visual effect.We found that proprietary LLMs such as GPT-3.5-Turbo show potential in this task, but their high cost and latency make them unsuitable for real-time applications.In our approach, we fine-tune a (smaller) student LLM with guidance from a (larger) teacher LLM and behavioral signals.We introduce offline metrics to evaluate student LLMs. Both online and offline experiments show that our student models manage to match the performance of our teacher model (GPT-3.5-Turbo), significantly reducing costs and latency.Lastly, we show that fine-tuning was improved by 25% in low-data regimes using augmentation.

## introduction videos are a powerful communication and storytelling medium, gaining popularity through social media and video-sharing platforms. this surge has inspired many to create content. however, the complexity of video editing, with its numerous parameters and their interactions, poses significant barriers for beginners @xcite . using natural language as an interaction medium for video editing can mitigate this challenge. text-to-video, diffusion-based models that support instruction-guided video editing have demonstrated impressive results. however, they are computationally expensive, slow, and still lack in visual quality and user control over the generated video @xcite @xcite . this makes them unsuitable for real-time mobile * work done during an internship at lightricks. applications, which need to combine high editing quality, low execution cost and fast response. we believe that instead of relying on an end-toend approach that treats deep learning models as black boxes, it is more beneficial to teach llms to use existing, specialized tools. this approach is also more interpretable. we are encouraged by recent advances in llms that demonstrated the effectiveness of building ai agents that leverage multiple external tools with llms @xcite @xcite , in particular for vision or vision-language tasks @xcite @xcite . in our work, we leverage llms to invoke existing, traditional video editing tools that are specialized for our task. our aim is to implement an ai assistant in our video editing mobile app, democratizing advanced capabilities. as a proof-of-concept, we focused on tonal color adjustments, allowing users to change a video's appearance via textual instructions (e.g., "golden hour"; see figure 1 ). learning tool chaining through prompt engineering and in-context learning often relies on proprietary llms like gpt-3.5 @xcite . these models are expensive, not publicly available, and slow, posing significant challenges for online production systems. we propose a distillation approach based on fine-tuning an open source (smaller) student llm for tools usage using the output from a (larger) teacher llm, enhanced by user behavioral signals. we create offline metrics to evaluate model performance, involving the choice of tools to apply and their parameters. this evaluation is challenging due to continuous parameter values and to our creativity-focused use case, with no single correct answer. finally, we develop a data augmentation scheme and demonstrate a 25% improvement in the common real-life scenario of low-data regimes. (smaller) student llm for invoking tools, using a (larger) teacher llm and behavioral feedback. we demonstrate the effectiveness of our approach in real-time production settings for visual editing. our solution achieves low cost and latency, making it suitable for industry applications. (2) we develop offline evaluation metrics for complex llm tool chaining. (3) our experiments, both online and offline, show that our smaller student models succeeded in matching the performance of our teacher, gpt-3.5-turbo. additionally, we show a 25% improvement in fine-tuning in low-data regimes using data augmentation. (4) our code and dataset are publicly available at our project website: https://www.orensultan.com/ ai_recolor.github.io/ . ## problem statement our visual editing task deals with color gradinga post-processing procedure that alters the appearance of an image or a video by adjusting its tonal colors. our application features three tonal adjustment tools: global adjust (global color range), selective adjust (selective color ranges), and filters. each tool has up to a dozen parameters, which can be difficult for beginners to set correctly. in our task, the user provides an asset (image/video) and a free-text description of the requested appearance. this raises the following challenges: (1) how to interpret the user's intent, which can be vague or require specific knowledge (e.g., given "the matrix" request, it should recognize the distinctive imagery associated with the movie, characterized by a green tint, high contrast, and cyberpunk aesthetic). ( 2 the output is of the following form: figure 1 shows examples of various input images (top), with intents (middle), and outputs (bottom). see appendix a.1 for details on tool parameters. ## our distillation framework approach our goal is to automate our visual editing task using llms. in our proof-of-concept, we found that proprietary llms, like gpt-3.5-turbo, can solve this task using preliminary prompts (based on an evaluation conducted by five experts from our team, who assessed the results across 20 different inputs). however, their high cost and latency make them unsuitable for real-time industry applications. we employ a distillation framework approach (see figure 2 ). we generate data by collecting outputs from a teacher llm based on user intents. the teacher llm selects relevant tools and sets their parameters. if multiple users express the same intent, this could result in multiple outputs per intent. we ensure high-quality data by retaining the best random split training data test data data processing for fine-tuning (3) offline evaluation (2) supervised fine-tuning (sft) a/b test (5) online evaluation (1) data collection -user's intent -teacher llm tools' parameters -user's feedback information augmentation (4) data augmentation fine-tuned student llm tool's selection & parameters prediction evaluation fine-tuned student llm (a) fine-tuned student llm (b) mistakes (1) we create a dataset by collecting user intents and the output (or potentially multiple outputs, if several users expressed the same intent) of our teacher llm. we ensure high quality by keeping outputs users chose to export frequently (one output with the highest export rate per intent). after data processing, we randomly split the data into training and test sets. (2) we fine-tune a smaller student llm on our dataset. (3) offline, we evaluate the student llm's selection of tools and predicted parameters. (4) to improve fine-tuning in low-data regimes, we use an llm to augment the training data by generating similar samples (e.g., "cool tone" from "cool morning") to mistakes of the student llm. (5) if a better student model is found offline, we conduct an online a/b test. results based on user feedback, filtering out those with no engagement. the retained data samples are injected into prompts for fine-tuning and are randomly split into training and test sets ( §3.1). then, we fine-tune a much smaller student llm on this dataset ( §3.2). since comparing between fine-tuned models in an online a/b test is costly and takes time, we design offline evaluation ( §3.3) metrics to predict the model's performance online. to improve fine-tuning in low-data settings, we use augmentation by having another llm generate similar samples for those the student llm got wrong during training, then add these to the training set ( §3.4). finally, to compare the actual performance of two fine-tuned student llms, we conduct an online a/b test ( §3.5). ## experiments we focus on the following research questions: rq1. how well do student llms perform, and do they effectively mimic the teacher llm? rq2. is augmentation effective in low-data regimes? ## conclusions and future work we introduced a novel nlp application for automatic video editing using llms, focusing on tonal color adjustment. we fine-tuned a (smaller) student llm with guidance from a (larger) teacher llm, while leveraging user behavioral signals. we proposed offline evaluation metrics and showed that our student models succeeded in matching the performance of our teacher model (gpt-3.5-turbo) in both offline and online experiments. our solution significantly reduces costs and latency, crucial for real-time industry applications. in the future, we plan to test potential fine-tuning improvements by adding rationale as an additional label for supplementary supervision in a multi-task framework, as in @xcite . we also aim to quantify the benefits of integrating user signals versus relying solely on unfiltered teacher llm outputs, and to explore other methods for combining user feedback, including personalization. we also plan to extend our one-hop responses to conversational agent / dialogue system. more broadly, we aim to apply our research to additional tools, features, and applications (e.g., light effects, transition between clips, etc.). our code and data can be found at our project website: https://www. orensultan.com/ai_recolor.github.io/ . we hope to inspire researchers to adopt our best practices in developing novel multi-modal real-time applications using tool chaining.