MOSEL : Inference Serving Using Dynamic Modality Selection (2024)

Abstract: Rapid advancements over the years have helped machine learning models reach previously hard-to-achieve goals, sometimes even exceeding human capabilities. However, achieving desired accuracy comes at the cost of larger model sizes and increased computational demands. Thus, serving predictions from these models to meet any latency and cost requirements of applications remains a key challenge, despite recent work in building inference serving systems as well as algorithmic approaches that dynamically adapt models based on inputs. Our paper introduces a new form of dynamism, modality selection, where we adaptively choose modalities from inference inputs while maintaining the model quality. We introduce MOSEL, an automated inference serving system for multi-modal ML models that carefully picks input modalities per request based on user-defined performance and accuracy requirements. MOSEL exploits modality configurations extensively, improving system throughput by 3.6 \times with an accuracy guarantee. It also reduces job completion times by 11 \times compared to modality-agnostic approaches.

## introduction recent advancements in deep learning has enabled deep neural networks (dnns), especially transformers, to far exceed human capabilities in various computer vision and natural language processing tasks @xcite . however, the computational requirement of the largest machine learning (ml) models has doubled every few months, resulting in a 1,000,000× increase from 2012 to 2020 @xcite . the increasing size of the models presents fundamental challenges in terms of latency and cost when they are commissioned for inference @xcite @xcite . these challenges has driven the development of inference serving systems. these systems, hosted by cloud providers, deploy ml models to deliver fast and accurate responses to queries. providers guarantee service level objectives (slos) for latency or accuracy while aiming to optimize hardware utilization and maximize throughput. one approach to mitigate inference overheads and improve throughput is through accuracy scaling, which adapts model accuracy to varying query demands. an inference task involves three components: system, model, and input. prior work focuses on optimizing the system and model aspects. system optimizations employ techniques such as batching @xcite @xcite , sharing @xcite , and scheduling @xcite @xcite . however, these techniques often require additional computational resources or powerful accelerators to handle higher query demands, which may not always be feasible due to the limited availability and flexibility of hardware resources. on the other hand, model optimizations often replace a large model with a more cost-effective variant, typically obtained using ml compression techniques, such as distillation @xcite , pruning @xcite and quantization @xcite . however, this approach necessitates multiple model replicas, wasting storage space and introducing overhead for switching between replicas and execution backends @xcite . in this paper, we propose an orthogonal and complementary perspective on accuracy scaling. in particular, we propose modulating the input, specifically via selectively using parts of it. we demonstrate its usefulness in the context of multi-modal learning @xcite , an emerging and important class of ml techniques that combine data from different modalities to provide prediction cooperatively, enhancing prediction accuracy. as we describe in section 3, we empirically find that some modalities (e.g., the audio modality in the textless vision-language transformer (tvlt) model @xcite ) contribute significantly to prediction accuracy without major resource use (e.g., memory) and processing time. in contrast, other modalities (e.g., the video modality in tvlt) consume significant resources and incur latency while only marginally improving accuracy. we leverage the above insight in inference settings and propose that modalities be selectively enabled or disabled based on application requirements and workload patterns, creating novel opportunities to exploit the trade-off between speed and accuracy that multi-modality presents. we refer to it as modality selection, which complements existing accuracy scaling techniques and can be directly applied to the original model. we assume that queries generally favor higher accuracy whenever resource permits, but can tolerate reduced accuracy for timely responses under resource constraints, provided that slos are not violated. this assumption is particularly relevant to applications like recommendation systems @xcite or real-time applications, where response time outweighs the need for accuracy @xcite . we build mosel, an automated inference serving system for multi-modal models that selects input modalities per request based on user-defined latency and accuracy slo and system load. our approach ensures scaling and performance during inference by dividing it into offline and online components. the offline component is designed to quickly generate a rich repository of modality selection strategies, enabling the online component to make informed decisions. for the online component, we ensure that, at inference time, late-enqueued jobs meet their latency requirements. we facilitate jobs ahead in the inference queue by dynamically reselecting modalities to ease the queueing load; this allows later-enqueued jobs to run at the required accuracy without missing their latency targets. we evaluate mosel on a set of representative multi-modal models that utilize commonly-seen architectures (transformer @xcite , bert @xcite , @xcite ). we show that mosel outperforms modalityagnostic approaches in resource utilization and query spike tolerance, reducing job completion times by up to 11× and handling up to 3.6× more requests with accuracy guarantees. moreover, mosel achieves up to 4.6× throughput when combined with quantization techniques. ## background multi-modal learning: multi-modal learning techniques are shown to surpass unimodal techniques by exploiting the complementary nature of different modalities, such as text, image, audio, and video @xcite . the existing techniques can be broadly classified into two categories: early fusion @xcite @xcite and late fusion @xcite @xcite . early fusion combines modalities at an early stage, blending features before further processing, as seen in tvlt @xcite . late fusion processes each modality separately and merges outcomes later, exemplified by the temporal binding network (tbn) @xcite . some methods attempt to combine properties from both early and late fusion @xcite @xcite @xcite . we demonstrate in figure 1 that multi-modalities present complexities due to varied resource requirements and performance traits. inference and its challenges: increased accuracy of dnns has led to their wide adoption in realworld applications resulting in increased production costs @xcite @xcite . inference serving systems use pre-trained ml models for predictions and manage resources to meet diverse user requests and application requirements @xcite @xcite . additionally, inference serving systems must manage dynamic workloads for cost and resource efficiency @xcite @xcite . the complexity increases when diverse services, each with unique service level objectives (slos), contend for shared model resources. comparing to serving systems that handle uni-modal models, serving multi-modal models with resource and latency-awareness has not been fully explored. ## opportunities & challenges accuracy across modalities: in multi-modal dnns, the importance of each modality can vary based on task, data, and model architecture @xcite @xcite . figure 1 (left) illustrates that some models, like tvlt, can achieve high accuracy without using all modalities. this shows data of different modalities contribute differently to the model accuracy. system implications: different modalities uniquely impact latency and memory consumption due to their distinct data representations and processing methods. for example, in tvlt, the audio modality is more efficient than video in memory usage and latency, with minimal accuracy tradeoffs, shown by figure 1 (left, middle). memory consumption scales with sequence length in attention mechanisms @xcite , selectively using subset of input modalities means shorter sequences and reduced memory usage. many recent works @xcite @xcite @xcite adopt similar attentionbased multi-modal models, which can also benefit from using fewer modalities to reduce latency and memory consumption. opportunities: applications provide inference systems with varying slos for accuracy and latency. these varying requirements offer opportunities for adaptive multi-modal selection, which previous systems haven't explored. modalities can be enabled or disabled based on application needs (e.g., serving latency) and resource availability. for instance, under high load, prioritizing ultra-low la-tency to prevent resource contention is crucial. in such scenarios, employing only the audio modality in tvlt helps reduce latency by 11× with minimal accuracy loss, as shown in figure 1 . conversely, under low load, using both video and audio ensured highest accuracy due to resource availability. we refer to this method as accuracy scaling @xcite , which adapts the inference accuracy to meet varying query demands. fully achieving accuracy scaling raises the following challenges. challenge 1: find optimal modalities to use. figure 2 illustrates the challenges of multi-modal inference. each job, consisted of multiple requests submitted by an application, has specific accuracy and latency slos. job 1 with an audio modality runs from time 0 to 20, job 2 arrives at 10 and starts at 20, and job 3 arrives shortly after 20. all jobs are executed in the order they arrive (a first-in-first-out, fifo, manner). each job requires a modality selection strategy to determine the modalities to use for each request. figure 2 shows six possible strategies for job 2 or job 3. for example, s 1 uses both modalities for both requests, while s 4 uses only the audio modality. figure 2: : job 1 runs from timestamp 0 to 20. job 2 arrives at timestamp 10 and starts at 20 after existing job 1 finishes. one of job 2's strategies, s 6 , has an accuracy of 0.67+0.7 2 = 0.685, failing to meet its accuracy slo (plan 1). similarly, s 4 also fails with an accuracy of 0.67. job 3 arrives shortly after 20 with a deadline of 150. if job 2 selects s 1 , it occupies the system until 140, leaving job 3 unable to meet its deadline (plan 2). by selecting a lower accuracy modality, job 2 can free up resources for job 3 (plan 3) allowing job 3 to use a higher accuracy video modality (plan 4). ure 2 illustrates that multiple strategies can yield valid accuracy. but we note that some strategies that create opportunities for a job potentially come at the cost of other jobs. in particular, greedily increasing accuracy for a job comes at the cost of increased resource consumption that may in turn hurt other jobs. this is illustrated by plan 2 in figure 2: it offers great accuracy for job 2 by selecting both modalities for both requests (effective accuracy of 0.8) and finishing exactly by 140 time units. but, it leaves no room for job 3 to finish by its deadline. on the other hand, by lowering accuracy for some jobs, we are left with extra resources that can be used to improve the outcomes for other jobs; e.g., in plan 3, we use just the audio modality for one of job 2's requests, yielding an effective accuracy of 0.735, which allows job 3 to start at time 100 and use the audio modality for both its requests in order to finish by time 140 with an accuracy of 0.67. in fact, we can improve job 3 -by picking a higher-accuracy modality (video) for one of its requests, job 3 achieves an effective accuracy of 0.685 (plan 4), while finishing at its deadline of 150. the online stage 1 2 ... 30 45 10 15 60 85 0.6 0.72 ... 1 :10 :60 2 :45 :70 ... a+v j1 j2 less-than-optimal strategies for some jobs in the queue to enable other later-coming jobs to meet their objectives. to tackle the challenges for models that dynamically adapt to input data, we need techniques that adapt to the changing slos and query load across jobs. existing inference serving systems leverage various techniques, including autoscaling (microsoft azure; amazon web services), model switching @xcite , batching @xcite , predictive serving @xcite and preemption @xcite . inference systems for multi-modality such as @xcite focus on speculatively executing modalities using augmented data. all of these techniques are agnostic to input data modalities and to the possibility of exploiting them for efficiency. ## formulation this section outlines how we identify optimal modality selection strategies to minimize latency while meeting accuracy slos for different job sizes. we also explain how we dynamically adjust these strategies in real-time to adapt to varying system resources. ## mosel implementation mosel is implemented in 3k lines of python code. the offline profiler uses pytorch @xcite to execute 1 dnns on the gpu and profile 2 system metrics through cuda api. we use @xcite to generate 3 the offline modality selection strategies. gekko is an optimizer that solves large-scale mixed-integer and differential algebraic equations with nonlinear programming solvers. the generated strategies are stored in a single pickle object. during model deployment, the monitor process buffers incoming jobs, 4 retrieves the generated modality plans, 5 uses gekko to finalize the modality plan for each job, and puts the job into a fifo queue shared with the worker process. to handle the gekko's overhead (which takes up to 80 ms), the monitor process enqueues enough jobs to compensate for the optimizer overhead. the worker process polls the fifo queue, executes 6 the jobs, and reports the latest 7 execution latency metrics back to the monitor process for accurate resource estimation. ## evaluation to evaluate our implementation, we conduct experiments using realistic workloads and address the following questions: q1: what are the benefits of modality-aware optimizations? (section 7.1) q2: is mosel resilient towards profiling error? (section 7.2) unless specified otherwise, our experiments use the following configurations. we explore mosel's compatibility with existing model optimization techniques in appendix c. experimental setup all measurements are conducted on real hardware using a nvidia tesla a100 gpu (80gb dram) and an intel xeon silver 4314 cpu (2.40ghz, 128gb dram). we used nvidia driver version 525.85, cuda 12.0, and pytorch 2.1.0. the operating system is ubuntu 22.04.1 lts with 5.15.0 kernel. models. ## conclusions we modulate the input to a model at inference time to achieve accuracy scaling. we show the benefits of this approach in multi-modal inference. we highlight the key challenges and present practical solutions within mosel. we believe that input data modulation, combined with model and system optimization, opens new possibilities in inference literature. modifying the input data can lead to significant benefits across the inference serving stack, including reduced network bandwidth, lower preprocessing costs, energy efficiency, and reduced operating costs. we envision mosel being applied to many scenarios with high input data variability that require adaptive optimizations.