  The training of deep neural networks (DNNs) is usually memory-hungry due to
the limited device memory capacity of DNN accelerators. Characterizing the
memory behaviors of DNN training is critical to optimize the device memory
pressures. In this work, we pinpoint the memory behaviors of each device memory
block of GPU during training by instrumenting the memory allocators of the
runtime system. Our results show that the memory access patterns of device
memory blocks are stable and follow an iterative fashion. These observations
are useful for the future optimization of memory-efficient training from the
perspective of raw memory access patterns.
