Paper:
Published as a conference paper at ICLR 2018 TOWARDS IMAGE UNDERSTANDING FROM DEEP COMPRESSION WITHOUT DECODING
Motivated by recent work on deep neural network (DNN)-based image compression methods showing potential improvements in image quality, savings in storage, and bandwidth reduction, we propose to perform image understanding tasks such as classification and segmentation directly on the compressed representations produced by these compression methods. Since the encoders and decoders in DNN-based compression methods are neural networks with feature-maps as internal representations of the images, we directly integrate these with architectures for image understanding. This bypasses decoding of the compressed representation into RGB space and reduces computational cost. Our study shows that accuracies comparable to networks that operate on compressed RGB images can be achieved while reducing the computational complexity up to 2×. Furthermore, we show that synergies are obtained by jointly training compression networks with classification networks on the compressed representations, improving image quality, classification accuracy, and segmentation performance. We find that inference from compressed representations is particularly advantageous compared to inference from compressed RGB images for aggressive compression rates.

Review:
Thanks for addressing most of the issues. I changed my given score from 3 to 6.
Summary:
This work explores the use of learned compressed image representation for solving 2 computer vision tasks without employing a decoding step. 
The paper claims to be more computationally and memory efficient compared to the use of original or the decompressed images. Results are presented on 2 datasets "Imagenet" and "PASCAL VOC 2012". They also jointly train the compression and classification together and empirically shows it can improve both classification and compression together.
Pros:
+ The idea of learning from a compressed representation is a very interesting and beneficial idea for large-scale image understanding tasks. 
Cons:
- The paper is too long (13 pages + 2 pages of references). The suggested standard number of pages is 8 pages + 1 page of references. There are many parts that are unnecessary in the paper and can be summarized. Summarizing and rewording them makes the paper more consistent and easier to read:
  ( 1. A very long introduction about the benefits of inferring from the compressed images and examples.
    2. A large part of the intro and Related work can get merged.  
    3. Experimental setup part is long but not well-explained and is not self-contained particularly for the evaluation metrics. 
    “Please briefly explain what MS-SSIM, SSIM, and PSNR stand for”. There is a reference to the Agustsson et al 2017 paper 
     “scalar quantization”, which is not well explained in the paper. It is better to remove this part if it is not an important part or just briefly but clearly explain it.
     4. Fig. 4 is not necessary. 4.3 contains extra information and could be summarized in a more consistent way.
     5. Hyperparameters that are applied can be summarized in a small table or just explain the difference between the 
      architectures that are used.)
- There are parts of the papers which are confusing or not well-written. It is better to keep the sentences short and consistent:
E.g: subsection 3.2, page 5: “To adapt the ResNet … where k is the number of … layers of the network” can be changed to 3 shorter sentences, which is easier to follow.
There are some typos: e.g: part 3.1, fever ---> fewer, 
- As it is mentioned in the paper, solving a Vision problem directly from a compressed image, is not a novel method (e.g: DCT coefficients were used for both vision and audio data to solve a task without any decompression). However, applying a deep representation for the compression and then directly solving a vision task (classification and segmentation) can be considered as a novel idea.
- In the last part of the paper, both compression and classification parts are jointly trained, and it is empirically presented that both results improved by jointly training them. However, to me, it is not clear if the trained compression model on this specific dataset and for the task of classification can work well for other datasets or other tasks.  
The experimental setup and the figures are not well explained and well written.