Paper:
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major "pain point" for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.

Review:
* Note: I highlight I did not assess the model design, which is the main contribution of the paper, and  did not know the background of prior work of system design to really assess the novelty of the work, my score is solely based on the experiments. 
 I am an expert in machine learning/computer vision, so I could assess the experiments in terms of their validity and relevance from the machine learning/vision perspective, however, I may not be the best person to evaluate the design choices of the system. Therefore I choose the low experience for my background.
* Paper summary: The paper proposes a framework to evaluate machine learning models in a hardware-agnostic way.
To evaluate the models using this framework, the user needs to specify the pre-processing, inference, and post-processing steps and the required software/hardware stack. The authors argue that this is important to consider the  HW/SW stack to allow a fair evaluation and reproducibility. Models are specified using a model specification called manifest.
* The authors assume that SW/HW stack change the results of deep learning models a lot, and this is the main assumption in this work, however, normally in practice HW/SW stack wont change the results.
* I found the experiments either not related to the point of the paper or being very trivial not helping to backing up the arguments of the paper. 
* In section 4.1, the authors consider different preprocessing operations and study their impact on the model performance, however, the fact that preprocessing impact the results is trivial in machine learning. In the same section, color layout and data layout, cropping and resizing, where the authors discussed about for instance how changing the data representation from NCHW or NHWC change the results, this is also trivial, because if you change the dimensions, you need to also change the model in a way that handles this change of dimension, therefore, this is clear that the results will change accordingly as well. Such experiments does not back up the main argument of the paper, which argues for fair evaluation between neural models, nor provides informative information to the reader.
On section 4.1, the experiment of type conversion and normalization, again this is mathematically clear that the order would change the results,  let's call imgByte=x, then by substituting given
values for mean and standard evaluation, equation (b) is simplified to (b) = (x-127.5)/((127.5)*(255)) 
however simplification of (c) results in (x/255-0.5)/0.5 = (x-0.5*255)/(0.5*255)=(x-127.5)/(0.5*255) 
the dominator of (b) and (c) are not equal, therefore, this is trivial that the results of these two 
the expression would not be the same. The author posed it as a new finding, but this is trivial that mathematically
these two equations would not be equal. Again, this experiment does not add any value to the paper.
In section 4.2, in Figure 9, the authors show a plot of the CPU latency for different batch sizes and instances,
together with GPU throughput for different batch sizes, i.e., images/seconds. The authors show latency for CPU
instances, versus throughput for GPU instance, since these two measures are not shown for both instances, this is 
not supported from the text, how actually authors compare this two instance and draw the conclusion that which instance is more efficient since there is no value shown for CPU throughput. Apart from that, I don't see how this section and determining if GPU or CPU instances of  Amazon compute cloud is more cost-efficient is related to the point of this paper which is on reproducibility. Also please have a look at Amazon webpage:
https://docs.aws.amazon.com/dlami/latest/devguide/gpu.html
Here, they explicitly mention that "A GPU instance is recommended for most deep learning purposes. Training new models will be faster on a GPU instance than a CPU instance. You can scale sub-linearly when you have multi-GPU instances or if you use distributed training across many instances with GPUs.",  so having this experiment again neither back up the arguments in the paper, nor add value to the paper.
* The major issue with this submission is that the experiments are not related to the arguments of the paper, and are not conveying any message towards backing up the arguments of the paper.
* Another crucial problem is that to allow a fair comparison especially in neural models, as shown in several studies(see [1] as a sample), this is important to account for random seeds and study how it impacts the model performance, to allow a fair evaluation of the models this is important to consider this factor, fair evaluation of models is argued to be the main point of this paper, however, the authors does not consider this factor in the paper, nor study it in the experiments.
[1] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Nils Reimers and Iryna Gurevych