Mapping all the neurons in the brain requires automatic reconstruction of entire cells from volume electron microscopy data. The flood-filling networks (FFN) architecture can achieve leading performance. However, the training of the network is computationally very expensive. In order to reduce the training time, we implemented synchronous and data-parallel distributed training using the Horovod framework on top of the published FFN code. We demonstrated the scaling of FFN training up to 1024 Intel Knights Landing (KNL) nodes at Argonne Leadership Computing Facility. We investigated the training accuracy with different optimizers, learning rates, and optional warm-up periods. We discovered that square root scaling for learning rate works best beyond 16 nodes, which is contrary to the case of smaller number of nodes, where linear learning rate scaling with warm-up performs the best. Our distributed training reaches 95\% accuracy in approximately 4.5 hours on 1024 KNL nodes using Adam optimizer .
Mapping all the neurons in the brain requires automatic reconstruction of entire cells from volume electron microscopy data. The flood-filling networks (FFN) architecture can achieve leading performance. However, the training of the network is computationally expensive. In order to reduce the training time, we implemented synchronous and data-parallel distributed training using the Horovod framework , which is different from the asynchronous training scheme used in the published FFN code. We demonstrated that our distributed training scaled well up to 2048 Intel Knights Landing (KNL) nodes on the Theta supercomputer. Our trained models achieved similar level of inference performance but took less training time compared to previous methods. Our study on the effect of batch size on the FFN training suggests ways to further improve training efficiency. Our findings on the optimal learning rate and batch size agree with previous works .
D
very
R
on top of the
, which is different from the asynchronous training scheme used in the
R
the scaling of FFN training up to 1024
that our distributed training scaled well up to 2048
R
at Argonne Leadership Computing Facility. We investigated the training accuracy with different optimizers, learning rates, and optional warm-up periods. We discovered that square root scaling for learning rate works best beyond 16 nodes, which is contrary to the case of smaller number of nodes, where linear learning rate scaling with warm-up performs the best. Our distributed training reaches 95\% accuracy in approximately 4.5 hours on 1024 KNL nodes using Adam optimizer
on the Theta supercomputer. Our trained models achieved similar level of inference performance but took less training time compared to previous methods. Our study on the effect of batch size on the FFN training suggests ways to further improve training efficiency. Our findings on the optimal learning rate and batch size agree with previous works