Heterogeneous Multi-Processor Systems-on-Chips (MPSoCs) containing CPU and GPU cores are typically required to execute applications concurrently. However, as will be shown in this paper, existing approaches are not well suited for concurrent applications as they are developed either by considering only a single application or they do not exploit both CPU and GPU cores at the same time. In this paper, we propose an energyefficient run-time mapping and thread partitioning approach for executing concurrent OpenCL applications on both GPU and GPU cores while satisfying performance requirements. Depending upon the performance requirements, for each concurrently executing application, the mapping process finds the appropriate number of CPU cores and operating frequencies of CPU and GPU cores, and the partitioning process identifies an efficient partitioning of the applications' threads between CPU and GPU cores. We validate the proposed approach experimentally on the Odroid-XU3 hardware platform with various mixes of applications from the Polybench benchmark suite. Additionally, a case-study is performed with a real-world application SLAMBench. Results show an average energy saving of 32% compared to existing approaches while still satisfying the performance requirements. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.  Modern embedded systems, e.g. mobile phones, rely on heterogeneous Multi-Processor Systems-on-Chips (MPSoCs) containing different types of cores. An example of a commercial heterogeneous MPSoC is the Samsung Exynos 5422 SoC [5] , which powers the popular Samsung Galaxy series of mobile phones. This SoC contains 4 ARM Cortex-A15 (big) CPU, 4 ARM Cortex-A7 (LITTLE) CPU and 6 ARM Mali-T628 GPU cores. Such an architecture provides opportunities to exploit distinct features of different types of cores in order to meet end-user demands in terms of performance and energy consumption. Additionally, the cores in such MPSoCs support dynamic voltage and frequency scaling (DVFS) that can be exploited to reduce dynamic power consumption (P ∝ V 2 f ). By employing DVFS, the energy consumption can be minimized if the power consumption is reduced sufficiently to account for the extra time taken to run the workload at a lower voltage and frequency. In some situations, dynamic power management (DPM) can lead to lower energy consumption than DVFS [20, 23] . For a given application, simultaneous exploitation of heterogeneous cores having different instruction set architectures (ISAs) such as CPU and GPU is challenging, as they handle instructions in different ways. Additionally, CPU cores typically handle task and thread level parallelisms, whereas GPU cores handle data level parallelism. OpenCL [7] provides an opportunity to write programs that can execute across heterogeneous cores including CPUs and GPUs [18, 19, 24, 30] . However, depending upon the kind of parallelism dominant in the application, the performance and energy consumption will vary when it is allocated onto only CPU, only GPU, or both CPU and GPU cores. Figure 1 shows execution time (bars) and energy consumption (lines) when executing individual OpenCL applications (SYR2K, SYRK, CORRELATION (CORR), and COVARIANCE (COVR)) from the Polybench benchmark [16] on the Exynos 5422 heterogeneous MPSoC while varying the fraction of application workload (threads) executed on CPU cores and remaining threads on GPU cores. All the cores are set to operate at maximum possible voltage-frequency. A fraction value of zero indicates that no threads are executed on CPU cores, i.e. all of them are executed only on the GPU cores. Similarly, when this value is 1, all the threads are executed only on CPU cores and none on GPU cores. It can be observed that some applications (specially having substantial sequential fraction) execute faster on CPU cores than GPU cores (e.g., CORR), whereas others (having extensive data parallelism) finish early on GPU cores (e.g., COVR). Further, all applications show a significant reduction in execution time and energy consumption when run on both the CPU and GPU cores, with an appropriate fraction value (between 0 and 1), defining the best partitioning of threads between CPU and GPU cores. These observations clearly indicate the advantages of simultaneously exploiting both CPU and GPU cores for each application. It is well known that multiple applications executing concurrently in a modern embedded system need to satisfy their performance requirements, and the overall energy consumption should be optimized [33] . For example, a mobile phone might need to execute JPEG and MP3 decoding concurrently, while satisfying the respective frames per second (fps) requirements when a user is viewing images and listening to music at the same time. However, existing run-time management works for concurrent applications consider only single-ISA heterogeneous MPSoCs, where cores have the same instruction set architecture [9, 10, 12, 13, 25, 34] . They usually consider Pthreads programming model and cannot be used to simultaneously exploit cores of different ISAs such as CPU and GPU. For a single application, simultaneous exploitation of CPU and GPU cores has been performed by employing OpenCL programming model [30] , but it leads to poor results when applied to concurrent applications (shown in the next section). Additionally, some works exploit either CPU or GPU for an application [37] , which is not efficient in terms of execution time and
Singh et al. REF proposed an energy-efficient runtime mapping and thread partitioning approach.
Energy-Efficient Run-Time Mapping and Thread Partitioning of Concurrent OpenCL Applications on CPU-GPU MPSoCs
TECS
Computer Science