OctopusKing: A TCTAware Task Scheduling on Spark Platform The majority of largescale data intensive applications executed by data centers are based on MapReduce or its opensource implementation, Spark. However, the native Sparkxe2x80x99s data locality has significant side effects. Spark prioritizes highpriority tasks. But tasks at the next level of data locality are never started because Sparkxe2x80x99s delay scheduling mechanism. Additionally, the current scheduling approach on Spark only considers allocating reasonably computing resources for each task, but ignores the balance between computing resources and network resources. Networkintensive jobs will generate a large number of crossrack tasks on heterogeneous clusters. More extensively, geographically distributed big data analytics clusters will produce greater network latency due to network transfer, which will decrease the performance of the whole cluster. Therefore, we propose OctopusKing a scheduling optimization approach based on the task completion time awareness. OctopusKing employs a deep learning algorithm to predict task completion time(TCT), which comprehensively considers four influence factors including the data size of each task, data nodes heterogeneity, the computation complexity of each task and network transfers. Based on the prediction of task completion time, our approach balances nonlocal task scheduling and network transfer time, which decreases the overall computation time of a job and improves the utilization of cluster resources effectively. Finally, we have implemented OctopusKing on Spark cluster with a 10 nodes. We show the convincing evidence that OctopusKing reduces average job completion time by 21% in comparison to Sparkxe2x80x99s default scheduler with little overhead.

Deep Smart Scheduling: A Deep Learning Approach for Automated Big Data Scheduling Over the Cloud With the widespread adoption of Internet of Thing (IoT) and the exponential growth in the volumes of generated data, cloud providers tend to receive massive waves of demands on their storage and computing resources. To help providers deal with such demands without sacrificing performance, the concept of cloud automation had recently arisen to improve the performance and reduce the manual efforts related to the management of cloud computing workloads. In this context, we propose in this paper, Deep learning Smart Scheduling (DSS), an automated big data task scheduling approach in cloud computing environments. DSS combines Deep Reinforcement Learning (DRL) and Long ShortTerm Memory (LSTM) to automatically predict the Virtual Machines (VMs) to which each incoming big data task should be scheduled to so as to improve the performance of big data analytics and reduce their resource execution cost. Experiments conducted using realworld datasets from Google Cloud Platform show that our solution minimizes the CPU usage cost by 28.8% compared to the Shortest Job First (SJF), and by 14% compared to both the Round Robin (RR) and improved Particle Swarm Optimization (PSO) approaches. Moreover, our solution decreases the RAM memory usage cost by 31.25% compared to the SJF, by 25% compared to the RR, and by 18.78% compared to the improved PSO.