Cost-effective Deployment of BERT Models in Serverless Environment (2021)

Abstract: In this study, we demonstrate the viability of deploying BERT-style models to AWS Lambda in a production environment. Since the freely available pre-trained models are too large to be deployed in this environment, we utilize knowledge distillation and fine-tune the models on proprietary datasets for two real-world tasks: sentiment analysis and semantic textual similarity. As a result, we obtain models that are tuned for a specific domain and deployable in the serverless environment. The subsequent performance analysis shows that this solution does not only report latency levels acceptable for production use but that it is also a cost-effective alternative to small-to-medium size deployments of BERT models, all without any infrastructure overhead.

## introduction machine learning models are notoriously hard to bring to production environments. one of the reasons behind is the large upfront infrastructure investment it usually requires. this is particularly the case with large pre-trained language models, such as bert @xcite or gpt @xcite whose size requirements make them difficult to deploy even when infrastructure investment is not of concern. at the same time, the serverless architecture with minimal maintenance requirements, automatic scaling and attractive cost, is becoming more and more popular in the industry. it is very well suited for stateless applications such as model predictions, especially in cases when the prediction load is unevenly distributed. since the serverless platforms have strict limits, especially on the size of the deployment package, it is not immediately obvious it may be a viable platform for deployment of models based on large pre-trained language models. ## related work despite a number of significant advances in various nlp approaches over the recent years, one of the limiting factors hampering their adoption is the large number of parameters that these models have, which leads to large model size and increased inference time. this may limit their use in resourceconstrained mobile devices or any other environment in which model size and inference time is the limiting factor, while negatively affecting the environmental costs of their use @xcite . this has led to a significant body of work focusing on lowering both the model size and inference time, while incurring minimal performance penalty. one of the most prominent approaches include knowledge distillation @xcite , in which a smaller model (the "student") is trained to reproduce the behavior of a larger model (the "teacher"). it was used to produce smaller bert alternatives, such as: when decreasing the model size leads to decreased latency, it can also have direct business impact. this has been demonstrated by google, which found out that increasing web search latency from 100 ms to 400 ms reduced the number of searches per user by 0.2 % to 0.6 % @xcite . a similar experiment done by booking.com has shown that an increase in latency of about 30 % results in about 0.5 percentage points decrease in conversion rates, which the authors report as a "relevant cost for our business" @xcite . each serverless platform has its specifics, which can have different impact on different use cases. various works, such as @xcite @xcite , provide a comparison of performance differences between the available platforms. in order to evaluate specific use cases, various benchmark suites have been introduced such as functionbench @xcite , which includes language generation as well as sentiment analysis test case. possibly the closest published work comparable to ours is @xcite , in which the authors demonstrate the deployment of neural network models, trained for short text classification and similarity tasks in a serverless context. since at the time of its publication the pytorch deployment ecosystem has been in its nascent stages, the authors had to build it from source, which complicates practical deployment. to the best of our knowledge, our work is the first to show the viability of deploying large pretrained language models (such as bert and its derivatives) in the serverless environment. aws azure gcp function size 250mb foot_0 -500mb execution time 15min -9min memory 10gb 14gb 8gb request size 6mb 100mb 10mb ## serverless environments serverless environments offer a convenient and affordable way of deploying a small piece of code. a survey by o'reilly media (o'reilly media, @xcite shows that the adoption of serverless was successful for the majority of the respondents' companies. they recognize reduced operational costs, automatic scaling with demand and elimination of concerns for server maintenance as the main benefits. since the functions deployed in a serverless environment share underlying hardware, os and runtime @xcite , there are naturally numerous limitations to what can be run in such environment. the most pronounced ones include: a more detailed list of the main limitations of the three most common serverless providers can be found in ## model training in the two case studies presented in this section, we first consider bert-provided classification token ([cls] token) an aggregate representation of a short text (up to 300 characters) for the sentiment analysis task. secondly, we utilize the embeddings produced by sentence-bert (sbert) @xcite for estimating the semantic similarity of a pair of short texts. since deploying even the smaller bert base with over 400mb in size is not possible in our setup, in the following cases studies we explore several alternative approaches, such as knowledge distillation into smaller models or training a smaller model directly. to do so, we use tinybert @xcite and mobilebert @xcite having about 56 mb and 98 mb in size, respectively. ## deployment as described in section 3, numerous limitations must be satisfied when deploying a model to a serverless environment, among which the size of the deployment package is usually the major one. the deployment package consists of the function code, runtime libraries and in our case a model. ## deployment evaluation we measured the performance of deployed models in scenarios with various amounts of allocated memory by making them predict on more than 5000 real-world examples. before recording measurements we let the deployed model evaluate a small subsample of data in order to keep the infrastructure in a "warm" state. this was done in order to estimate the real-life inference time, i.e. to avoid biasing the inference results by initialization time of the service itself. from the results described in the serverless deployments are also costeffective. the total costs of 1m predictions, taking 100 ms each and using 1 gb of ram, are around @xmath08 per month. ## conclusion we present a novel approach of deploying domainspecific bert-style models in a serverless environment. to fit the models within its limits, we use knowledge distillation and fine-tune them on domain-specific datasets. our experiments show that using this process we are able to produce much smaller models at the expense of a minor decrease in their performance. the evaluation of the deployment of these models shows that it can reach latency levels appropriate for production environments, while being cost-effective. although there certainly exist platforms and deployments that can handle much higher load (often times with smaller operational cost @xcite ), the presented solution requires minimal infrastructure effort, making the team that trained these models completely self-sufficient. this makes it ideal for smaller-scale deployments, which can be used to validate the model's value. the smaller, distilled models created in the process can then be used in more scalable solutions, should the cost or throughput prove inadequate during test deployments.