Design and Evaluation of Decentralized Scaling Mechanisms for Stream Processing Over the last decade, stream processing (SP) gained momentum as an efficient paradigm dealing with the realtime processing of large streams of data. As mature as they are, mainstream stream processing engines are not ready to be used over more distributed platforms that are today emerging such as Fog platforms. Streams being produced at the edge of the platform, and edges being composed of less powerful compute nodes, a decentralized approach of Stream Processing is needed. This paper moves a step towards decentralized SP by proposing a decentralized autoscaling mechanism for SP applications. Autoscaling refers to the ability for a system to automatically scale in and out as the load varies. As a stream processing application is generally composed of a set of operators in a pipeline, our protocol lets each operatoru0027s replica take scaling decisions and enforce them independently. Assuming a fully decentralized setting, we consider each replica has only a partial view of the graph of operators. Then, without care, concurrent distributed scaling could lead to inconsistencies in replicasu0027 views and to data loss. To address this issue, we devise a maintenance protocol ensuring the consistency of replicasu0027 views, no data loss and no downtime during reconfigurations. Finally, the paper describes the early experimentations conducted with a software prototype of a decentralized SP engine over a computing cluster, showing the usability of the mechanism.

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced, e.g., in the domain of the Internet of Things. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enable SP systems to process these streams with continuous high quality of service. The current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes specific assumptions and focuses on particular aspects. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. Therefore, in this survey, we study the literature and develop a classification of current methods for both parallelization and elasticity in SP systems.