  Function-as-a-Service (FaaS) is one of the most promising directions for the
future of cloud services, and serverless functions have immediately become a
new middleware for building scalable and cost-efficient microservices and
applications. However, the quickly moving technology hinders reproducibility,
and the lack of a standardized benchmarking suite leads to ad-hoc solutions and
microbenchmarks being used in serverless research, further complicating
metaanalysis and comparison of research solutions. To address this challenge,
we propose the Serverless Benchmark Suite: the first benchmark for FaaS
computing that systematically covers a wide spectrum of cloud resources and
applications. Our benchmark consists of the specification of representative
workloads, the accompanying implementation and evaluation infrastructure, and
the evaluation methodology that facilitates reproducibility and enables
interpretability. We demonstrate that the abstract model of a FaaS execution
environment ensures the applicability of our benchmark to multiple commercial
providers such as AWS, Azure, and Google Cloud. Our work facilities
experimental evaluation of serverless systems, and delivers a standardized,
reliable and evolving evaluation methodology of performance, efficiency,
scalability and reliability of middleware FaaS platforms.
