  Noisy labels are very common in deep supervised learning. Although many
studies tend to improve the robustness of deep training for noisy labels, rare
works focus on theoretically explaining the training behaviors of learning with
noisily labeled data, which is a fundamental principle in understanding its
generalization. In this draft, we study its two phenomena, clean data first and
phase transition, by explaining them from a theoretical viewpoint.
Specifically, we first show that in the first epoch training, the examples with
clean labels will be learned first. We then show that after the learning from
clean data stage, continuously training model can achieve further improvement
in testing error when the rate of corrupted class labels is smaller than a
certain threshold; otherwise, extensively training could lead to an increasing
testing error.
