Using multicast to pre-load jobs on the ParPar cluster
The ParPar system is a high-performance cluster environment supporting a multiuser parallel workload. Its design follows a master-nodes structure, where the master controls all aspects of system activity using a dedicated control network. As nearly all control messages are multicast to a set of nodes, we implemented a reliable multicast protocol for this network based on UDP. This was then used to pre-load executable files to the nodes, rather than using demand paging via NFS. Such pre-loading leads to significant reductions in job startup times in most cases. It is also more scalable than an asymmetrical hardware approach giving the master higher bandwidth, which can be used for small clusters.
parpar cluster reliable multicast job startup file distribution job pre-loading