With the growing scale of HPC systems, fault tolerance is becoming a major concern. The two traditional approaches for message passing applications, coordinated checkpointing and message logging, have severe scalability issues. Coordinated checkpointing protocols make all processes roll back after a failure. Message logging protocols log a huge amount of data and can induce an overhead on communications performance. Hierarchical rollback-recovery protocols based on the combination of coordinated checkpointing and message logging are an alternative. These partial message logging protocols are based on process clustering: only messages between clusters are logged to limit the consequence of a failure to one cluster. These protocols would work eciently only if one can nd clusters of processes in the applications such that the ratio of logged messages is very low. We study the communication patterns of message passing HPC applications to show that partial message logging is suitable in most cases. We propose a partitioning algorithm to nd suitable clusters of processes, given the application execution communication pattern. Finally, we evaluate the eciency of partial message logging using two state of the art protocols on a set of representative applications.
Fortunately, several applications in HPC have a structured communication pattern with clusters of processes enclosing a high portion of the communication volume REF .
On the use of cluster-based partial message logging to improve fault tolerance for MPI HPC applications
Euro-Par
Computer Science