Paper:
Overfitting is an ubiquitous problem in neural network training and usually mitigated using a holdout data set. Here we challenge this rationale and investigate criteria for overfitting without using a holdout data set. Specifically, we train a model for a fixed number of epochs multiple times with varying fractions of randomized labels and for a range of regularization strengths. A properly trained model should not be able to attain an accuracy greater than the fraction of properly labeled data points. Otherwise the model overfits. We introduce two criteria for detecting overfitting and one to detect underfitting. We analyze early stopping, the regularization factor, and network depth. In safety critical applications we are interested in models and parameter settings which perform well and are not likely to overfit. The methods of this paper allow characterizing and identifying such models.

Review:
Overview:
The authors aim at finding and investigating criteria that allow to determine whether a deep (convolutional) model overfits the training data without using a hold-out data set.  
Instead of using a hold-out set they propose to randomly flip the labels of certain amounts of training data and inspect the corresponding 'accuracy vs. randomization‘ curves. They propose three potential criteria based on the curves for determining when a model overfits and use those to determine the smallest l1-regularization parameter value that does not overfit. 
I have several issues with this work. Foremost, the presented criteria are actually not real criteria (expect maybe C1) but rather general guidelines to visually inspect 'accuracy over randomization‘ curves. The criteria remain very vague and seem be to applicable mainly to the evaluated data set (e.g. what defines a ’steep decrease’?). Because of that, the experimental evaluation remains vague as well, as the criteria are tested on one data set by visual inspection. Additionally, only one type of regularization was assumed, namely l1-regularization, though other types are arguably more common in the deep (convolutional) learning literature.  
Overall, I think this paper is not fit for publication, because the contributions of the paper seem very vague and are neither thoroughly defined nor tested.
Detailed remarks:
General:
A proper definition or at least a somewhat better notion of overfitting would have benefitted the paper. In the current version, you seem to define overfitting on-the-fly while defining your criteria. 
You mention complexity of data and model several times in the paper but never define what you mean by that.
Detailed:
Page 3, last paragraph: Why did you not use bias terms in your model?
Page 4, Assumption. 
- What do you mean by the data being independent? Independent and identically distributed?  
- "As in that case correlation in the data can be destroyed by the introduction of randomness making the data easier to learn.“ What do you mean by "easier to learn"? Better generalization? Better training error? 
- I don’t understand the assumptions. You state that the regularization parameter should decrease complexity of the model. Is that an assumption? And how do you use that later?
- What does "similar scale“ mean? 
Page 4, Monotony. 
- You state two assumptions or claims, 'the accuracy curve is strictly monotonically decreasing for increasing randomness‘ and 'we also expect that accuracy drops if the regularization of the model is increased’, and then state that 'This shows that the accuracy is strictly monotonically decreasing as a function of randomness and regularization.‘ Although you didn’t show anything but only state assumptions or claims (which may be reasonable but are not backed up here). 
I actually don’t understand the purpose of this paragraph.
- Section 3.3 is confusing to me. What you actually do here is you present 3 different general criteria that could potentially detect overfitting on label-randomized  training sets. But you state it as if those measures are actually correct, which you didn’t show yet.
My main concern here, besides the motivations that I did not fully understand (s.b.), is the lack of measurable criteria. While for criterion 1 you define overfitting as 'above the diagonal line‘ and underfitting as ‚below the line‘, which is at least measurable depending on sample density of the randomization, such criteria are missing for C2 and C3.       Instead, you present vague of ’sharp drops’ and two modes but do not present rigorous definitions. You present a number for C2 in Section 5, but that is only applicable to the present data set (i.e. assuming that training accuracy is 1). 
Criterion 2 (b) is not clear.  
- I neither understand "As the accuracy curve is also monotone decreasing with increasing regularization we will also detect the convexity by a steep drop in accuracy as depicted by the marked point in the Figure 1(b)" 
nor do I understand "accuracy over regularization curve (plotted in log-log space) is constant"?
Does that mean that you assume that whenever the training accuracy drops lower than that of the model without regularization, it starts to underfit?
Due to the lack of numerical measures, the experimental evaluation necessarily remains vague by showing some graphs that show that all criteria are roughly met by regularization parameter \lambda=0.00011 on the cifar data set.  In my view, this evaluation of the (vague) criteria is not fit for showing their possible merit.