Towards Making Deep Transfer Learning Never Hurt Transfer learning have been frequently used to improve deep neural network training through incorporating weights of pretrained networks as the startingpoint of optimization for regularization. While deep transfer learning can usually boost the performance with better accuracy and faster convergence, transferring weights from inappropriate networks hurts training procedure and may lead to even lower accuracy. In this paper, we consider deep transfer learning as minimizing a linear combination of empirical loss and regularizer based on pretrained weights, where the regularizer would restrict the training procedure from lowering the empirical loss, with conflicted descent directions (e.g., derivatives). Following the view, we propose a novel strategy making regularizationbased Deep Transfer learning Never Hurt (DTNH) that, for each iteration of training procedure, computes the derivatives of the two terms separately, then reestimates a new descent direction that does not hurt the empirical loss minimization while preserving the regularization affects from the pretrained weights. Extensive experiments have been done using common transfer learning regularizers, such as L2SP and knowledge distillation, on top of a wide range of deep transfer learning benchmarks including Caltech, MIT indoor 67, CIFAR10 and ImageNet. The empirical results show that the proposed descent direction estimation strategy DTNH can always improve the performance of deep transfer learning tasks based on all above regularizers, even when transferring pretrained weights from inappropriate networks. All in all, DTNH strategy can improve stateoftheart regularizers in all cases with 0.1%7% higher accuracy in all experiments.

Industrial Transfer Learning: Boosting Machine Learning in Production In the field of production, machine learning offers great potentials to develop innovative solutions for optimization or automation. However, it faces challenges with regard to the availability of data and the high training effort of the learning models in the event of changes in the production process. In this paper, we address these challenges by introducing deep transfer learning for production. We demonstrate its potentials and benefits in a real application for predictive quality in injection molding and propose a novel approach for the continual training of neural networks across manufactured products. By creating a neural network that leverages knowledge from previous products without forgetting them, the approach shows better learning rates and more accurate predictions while requiring much less data for training. Our code is publicly available1 to reproduce our results and build upon them.