  PetscSF, the communication component of the Portable, Extensible Toolkit for
Scientific Computation (PETSc), is being used to gradually replace the direct
MPI calls in the PETSc library. PetscSF provides a simple application
programming interface (API) for managing common communication patterns in
scientific computations by using a star-forest graph representation. PetscSF
supports several implementations whose selection is based on the
characteristics of the application or the target architecture. An efficient and
portable model for network and intra-node communication is essential for
implementing large-scale applications. The Message Passing Interface, which has
been the de facto standard for distributed memory systems, has developed into a
large complex API that does not yet provide high performance on the emerging
heterogeneous CPU-GPU-based exascale systems. In this paper, we discuss the
design of PetscSF, how it can overcome some difficulties of working directly
with MPI with GPUs, and we demonstrate its performance, scalability, and novel
features.
