Abstract. This paper addresses the problem of one-dimensional partitioning of structurally unsymmetric square and rectangular sparse matrices for parallel matrix-vector and matrix-transposevector multiplies. The objective is to minimize the communication cost while maintaining the balance on computational loads of processors. Most of the existing partitioning models consider only the total message volume hoping that minimizing this communication-cost metric is likely to reduce other metrics. However, the total message latency (start-up time) may be more important than the total message volume. Furthermore, the maximum message volume and latency handled by a single processor are also important metrics. We propose a two-phase approach that encapsulates all these four communication-cost metrics. The objective in the first phase is to minimize the total message volume while maintaining the computational-load balance. The objective in the second phase is to encapsulate the remaining three communication-cost metrics. We propose communicationhypergraph and partitioning models for the second phase. We then present several methods for partitioning communication hypergraphs. Experiments on a wide range of test matrices show that the proposed approach yields very effective partitioning results. A parallel implementation on a PC cluster verifies that the theoretical improvements shown by partitioning results hold in practice. Key words. matrix partitioning, structurally unsymmetric matrix, rectangular matrix, iterative method, matrix-vector multiply, parallel computing, message volume, message latency, communication hypergraph, hypergraph partitioning AMS subject classifications. 05C50, 05C65, 65F10, 65F50, 65Y05 DOI. 10.1137/S1064827502410463 1. Introduction. Repeated matrix-vector and matrix-transpose-vector multiplies that involve the same large, sparse, structurally unsymmetric square or rectangular matrix are the kernel operations in various iterative algorithms. For example, iterative methods such as the conjugate gradient normal equation error and residual methods (CGNE and CGNR) [15, 34] and the standard quasi-minimal residual method (QMR) [14] , used for solving unsymmetric linear systems, require computations of the form y = Ax and w = A T z in each iteration, where A is an unsymmetric square coefficient matrix. The least squares (LSQR) method [31] , used for solving the least squares problem, and the Lanczos method [15] , used for computing the singular value decomposition, require frequent computations of the form y = Ax and w = A T z, where A is a rectangular matrix. Iterative methods used in solving the normal equations that arise in interior point methods for linear programming require repeated computations of the form y = AD 2 A T z, where A is a rectangular constraint matrix and D is a diagonal matrix. Rather than forming the coefficient matrix AD 2 A T , which may be quite dense, the above computation is performed as w = A T z, x = D 2 w, and y = Ax. The surrogate constraint method [29, 30, 39, 40] , which is used for solving the linear feasibility problem, requires decoupled matrix-vector and matrix-transpose-vector multiplies involving the same rectangular matrix.
Among these works, the one proposed in REF is a twophase approach which addresses the bandwidth cost in the first phase utilizing a 1D model and the latency cost in the second phase utilizing the communication hypergraph model.
Encapsulating multiple communication-cost metrics in partitioning sparse rectangular matrices for parallel matrix-vector multiplies
SIAM Journal on Scientific Computing
Computer Science