GPU Acceleration of Communication Avoiding Chebyshev Basis Conjugate Gradient Solver for Multiphase CFD Simulations Iterative methods for solving large linear systems are common parts of computational fluid dynamics (CFD) codes. The Preconditioned Conjugate Gradient (PCG) method is one of the most widely used iterative methods. However, in the PCG method, global collective communication is a crucial bottleneck especially on accelerated computing platforms. To resolve this issue, communication avoiding (CA) variants of the PCG method are becoming increasingly important. In this paper, the PCG and Preconditioned Chebyshev Basis CA CG (PCBCG) solvers in the multiphase CFD code JUPITER are ported to the latest V100 GPUs. All GPU kernels are highly optimized to achieve about 90% of the roofline performance, the block Jacobi preconditioner is redesigned to extract high computing power of GPUs, and the remaining bottleneck of halo data communication is avoided by overlapping communication and computation. The overall performance of the PCG and PCBCG solvers is determined by the competition between the CA properties of the global collective communication and the halo data communication, indicating an importance of the internode interconnect bandwidth per GPU. The developed GPU solvers are accelerated up to 2x compared with the former CPU solvers on KNLs, and excellent strong scaling is achieved up to 7,680 GPUs on the Summit.

HighPerformance Adaptive MPI Derived Datatype Communication for Modern MultiGPU Systems The recent advent of the NVLink interconnect and Peripheral Component Interconnect express (PCIe) switch has resulted in the creation of extremely dense Graphics Processing Unit (GPU) systems like Cray CSStorm and NVIDIA DGX. In addition to the extremely high computational capability and communication capacity within a single machine, these systems expose novel capabilities like performing loadstore operations from remote GPU memory across interconnects. While researchers have proposed solutions that take advantage of loadstore semantics to provide support for highperformance datatype processing on CPUs, there exists no scholarly work on how one can orchestrate such highperformance datatypebased communication for GPUresident data. In this paper, we take up this challenge and propose highperformance and architectureaware designs for GPUbased noncontiguous datatype processing that uses the loadstore semantics exposed by modern dense GPU systems. We demonstrate that the proposed solutions can reduce the overhead of datatype processing by up to 4.7X compared to the stateoftheart schemes for GPUbased MILC communication kernel on an NVLink2enabled dense GPU system. For a weather forecast application kernel, the proposed designs demonstrate up to 9.9X faster HaloExchange kernel among 64 GPUs over stateoftheart designs. The proposed adaptive scheme also reports 10% higher throughput than existing designs for a 2D Jacobi solver on 16 GPUs. To the best of our knowledge, this is the first scholarly work that takes advantage of the zerocopy based loadstore semantics to perform highperformance GPU to GPU derived datatype communication on modern dense GPU systems.