Convolution Neural Networks (CNNs), a special subcategory of Deep Learning Neural Networks (DNNs), have become increasingly popular in industry and academia for their powerful capability in pattern classification, image processing, and speech recognition. Recently, they have been widely adopted in High Performance Computing (HPC) environments for solving complex problems related to modeling, runtime prediction, and big data analysis. Current state-of-the-art designs for DNNs on modern multi-and many-core CPU architectures, such as variants of Caffe, have reported promising performance in speedup and scalability, comparable with the GPU implementations. However, modern CPU architectures employ Non-Uniform Memory Access (NUMA) technique to integrate multiple sockets, which incurs unique challenges for designing highly efficient CNN frameworks. Without a careful design, DNN frameworks can easily suffer from long memory latency due to a large number of memory accesses to remote NUMA domains, resulting in poor scalability. To address this challenge, we propose NUMA-aware multi-solver-based CNN design, named NUMA-Caffe, for accelerating deep learning neural networks on multi-and many-core CPU architectures. NUMA-Caffe is independent of DNN topology, does not impact network convergence rates, and provides superior scalability to the existing Caffe variants. Through a thorough empirical study on four contemporary NUMA-based multi-and many-core architectures, our experimental results demonstrate that NUMA-Caffe significantly outperforms the state-of-the-art Caffe designs in terms of both throughput and scalability. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. With the prevalence of powerful multi-core-and many-core-based architectures in modern HPC systems, researchers have attempted to apply HPC resources to accelerate deep neural networks' (DNNs') long training processes. Existing approaches have focused on developing GPU-centric approaches [16, 21, 28 , 51] to speed up DNN training due to its massive parallelism and architecturespecific optimization libraries (e.g., cuBLAS and cuDNN [13] ). However, there are several limitations hindering GPU scalability as an effective training platform: (i) small resident memory that most real-world data input cannot fit in, (ii) lack of generality in optimizations, (iii) difficulty in porting across architectures, and (iv) strong bias towards certain applications [55] . This makes widely available CPU-based multi-and many-core architectures a strong alternative for training complex neural networks. This effort has been widely supported by both academia and industry, e.g., Microsoft's Project Adam [14] and spg-CNN [52] . The main reason to use CPU-based platforms is that they are easily accessible and can achieve several teraflops, perfect for training medium and moderately large networks within a reasonable time. Additionally, recent research on CPU architectures [19, 55] has shown significant training speedups, even at the level of performance as those achieved by GPU optimizations. Thus, tackling CPU-based machine learning challenges is both important and necessary. However, the state-of-the-art Convolution Neural Network (CNN) designs often lack sophistication in considering complex memory hierarchies of modern multi-core and many-core architectures and their impact on throughput and scalability. For instance, modern multi-core and many-core architectures have employed non-uniform memory access (NUMA) techniques to scale memory bandwidth. A NUMA architecture can scale to thousands of cores, such as SGI UV [53] and HP Integrity Superdome X [24] . A typical NUMA architecture integrates multiple microprocessors, and each has its memory physically attached. Any two microprocessors use a high-speed interconnect (e.g., Intel QuickPath [17] and AMD's HyperTransport [23] ) to exchange data. Cores in each microprocessor can access the local memory directly attached to the same microprocessor and also remote memory attached to other microprocessors. To simplify the discussion for the systems in which NUMA effects may exist in both main memory and last-level caches, we refer to cache/memory along with all CPU cores that can access a memory partition with uniform latency as a NUMA node. Local and remote NUMA nodes have significant differences in accessing latency and available bandwidth. Typically, a remote access has lower bandwidth and incurs more latency than a local access. In NUMA architectures, multithreaded programs may suffer from significant performance degradation if they access data in remote memory with high frequency. Since modern supercomputers are commonly built on a large amount of multi-socket-based computing nodes, identifying and resolving scaling issues originating from remote NUMA communication is evident for scaling up deep learning algorithms. Through a detailed performance scalability study, we find that current multi-and many-core CPU-based CNN designs do not take NUMA effects into consideration, causing significant scalability issues on today's NUMA architectures. Take two variants of a popular CNN framework-Caffe as an example, the original Berkley Vision Lab version [34] and the newly released Intel-Caffe [19] , which is highly accelerated by Intel Math Kernel Library 2017 [32] and batch-level parallelization [55]-have largely focused on the layer-level parallelism and memory optimizations inside a NUMA node. However, according to our observations on four contemporary architectures, neither of the two designs scales well across NUMA nodes (Section 3). This is because worker threads spawned for both batch-level and BLAS-level parallelization run on cores in different NUMA
REF is proposed as a novel NUMA-aware framework for training CNNs on modern CPU-based multiand many-core-based architectures.
NUMA-Caffe: NUMA-Aware Deep Learning Neural Networks
TACO
Computer Science