A gearbox model for processing large volumes of data by using pipeline systems encapsulated into virtual containers Abstract   Software pipelines enable organizations to chain applications for adding value to contents (e.g., confidentially, reliability, and integrity) before either sharing them with partners or sending them to the cloud. However, the pipeline components add overhead when processing large volumes of data, which can become critical in realworld scenarios. This paper presents a gearbox model for processing large volumes of data by using pipeline systems encapsulated into virtual containers. In this model, the gears represent applications, whereas gearboxes represent software pipelines. This model was implemented as a collaborative system that automatically performs Gear up (by using parallel patterns) andor Gear down (by using inmemory storage) until all gears produce uniform data processing velocities. This model reduces delays and bottlenecks produced by the heterogeneous performance of applications included in software pipelines. The new container tool     c  a  p  s  u  l  e     has been designed to encapsulate both the collaborative system and the software pipelines into a virtual container and deploy it on IT infrastructures. We conducted case studies to evaluate the performance of     c  a  p  s  u  l  e     when processing medical images and PDF repositories. The incorporation of a capsule to a cloud storage service for preprocessing medical imagery was also studied. The experimental evaluation revealed the feasibility of applying the gearbox model to the deployment of software pipelines in realworld scenarios as it can significantly improve the enduser service experience when preprocessing largescale data in comparison with stateoftheart solutions such as Sacbe and Parsl.

SocialEdge: Enabling Trusted Data Processing Workflow in Smart Communities In typical commercial cloud and edge providers data collection and processing nodes might be owned by a single trusted entity. In contrast, in a voluntary infrastructure, ownership is distributed. In such a setup, while individuals contributing data might demand constraints on how their data is accessed and used, volunteers of compute resources might want assurance that their resources are used only for intended purposes. These significant differences motivate SocialEdge, an edge orchestration framework that 1) leverages online social network for bootstrapping and permissioned block chain for information assurance, 2) exploits recursive containerization for enabling secure data access, ensuring only intended workloads are run and enforcing finegrained resource control, and 3) leverages components from Kubernetes to instantiate and manage an ephemeral cluster composed of volunteered resources. In this paper we describe the design and prototype implementation of SocialEdge and model the problem of orchestrating the ondemand data processing infrastructure under data access constraints as a variant of the multicommodity flow problem. We propose and evaluate mixed integer programming and greedy heuristic approaches, and characterize recursive containerized environment experimentally using benchmarks.