  The SPAdes assembler for metagenome assembly is a long-running application
commonly used at the NERSC supercomputing site. However, NERSC, like many other
sites, has a 48-hour limit on resource allocations. The solution is to chain
together multiple resource allocations in a single run, using
checkpoint-restart. This case study provides insights into the "pain points" in
applying a well-known checkpointing package (DMTCP: Distributed MultiThreaded
CheckPointing) to long-running production workloads of SPAdes. This work has
exposed several bugs and limitations of DMTCP, which were fixed to support the
large memory and fragmented intermediate files of SPAdes. But perhaps more
interesting for other applications, this work reveals a tension between the
transparency goals of DMTCP and performance concerns due to an I/O bottleneck
during the checkpointing process when supporting large memory and many files.
Suggestions are made for overcoming this I/O bottleneck, which provides
important "lessons learned" for similar applications.
