Paper:
Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community. To address it and to establish baselines and a common evaluation protocol in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form. Specifically, we investigate in-domain representation learning to develop generic remote sensing representations and explore which characteristics are important for a dataset to be a good source for remote sensing representation learning. The established baselines achieve state-of-the-art performance on these datasets using in-domain representations.

Review:
### Summary
This paper discusses transfer learning in remote sensing image-classification tasks.
The key finding is: while transfer learning from ImageNet dataset for remote sensing, performance can be slightly improved if the model weights are fine-tuned on an auxiliary remote sensing dataset (hence in-domain transfer).
​
​
### Strengths
- The paper is well motivated for the need of good representation learning for remote sensing, and is written well making it easy to follow.
- It compiles a nice list of five datasets available for remote sensing, along with their characteristics. Also, the list of common properties of such datasets (e.g. rotation invariant) is good for data pre-processing for engineering applications in remote sensing.
- The paper provides baseline accuracies on five important datasets in remote sensing.
​
​
### Weaknesses
Apart from the contributions of compilation of different datasets which are already present, and classification results on them, there is not much novelty in the paper. There are several major weaknesses:
​
1. **About datasets**: The paper emphasizes a lot on the lack of standard datasets and metrics in remote sensing. However, the contribution is limited to compiling existing datasets and defining a 60-20-20 train-validation-test split on them. There is no new data collection, no insight in how these existing diverse datasets could be combined together, or why this particular definition of split is used. Hence, the reviewer cannot count this as a contribution.
2. **About representation learning**: Since the source domain (ImageNet), in-domain source domain and the target domain all solve classification tasks, a better term to use would be *transfer learning* instead of representation learning. The reviewer recommends to demonstrate how these learned "representations" are used for other downstream tasks or otherwise show some other representation learning methods (semi-supervised or unsupervised), which is not same as the downstream task. Hence, it would be inappopriate to sell it as representation learning, in its current form.
3. **About in-domain learning**: Since the in-domain transfer is also done by fine-tuning an existing model which was trained on ImageNet, it is expected to do somewhat better than Imagenet only. The paper just clarifies this fact in practice, and the performance gains are also statistically insignificant. Again, this is not a significant contribution.
4. **About hyperparameter sweeping**: The minor improvements observed due to in-domain transfer is also unreliable, since very hyperparameters are tried across datasets. E.g. only 2 values of learning rates (0.1 and 0.01) are tested. This could also explain why BigEarthNet and So2Sat had poor training accuracies. The reviewer also disagrees with using only 1k samples for selecting best dataset to transfer from - explained in minor weaknesses.
5. **About experiments**: In-domain transfer is tested by fine-tuning on one dataset and then again fine-tuning on another dataset. It was observed that the dataset RESISC-45 does the best in this scenario. The reviewer disagrees with paper's hypothesis about noisy labels resulting in this artifact. The reviewer suspects that this is because RESISC-45 contains labels which are most appropriate for classification transfer in other datasets (and also it is very balanced). Can the authors comment on this? If another method of representation learning than classification is employed, then it is very possible that other larger datasets result in better representations. The comparison across datasets currently is strange, since all it tells is which dataset has most similar classification labels to other datasets on average. To remove this problem in comparison, the representation learning task and testing task should be different.
6. **About conclusions and contributions**: The experimental setup in this paper is insufficient to claim that it investigates *what characteristics should a dataset have to be a good source for remote sensing representation learning*. Transfer from ImageNet works because it contains a large number of classes, along with data diversity and its sheer size. This cannot be said for the datasets analyzed in this paper, since these factors are not analyzed individually, and the representation learning method is not standardized across all datasets (classification is not standardized since different datasets have different classes), making the comparison unfair.
​
​
​
#### Minor weaknesses:
- Table 4: Since existing BigEarthNet result contains precision/recall, that is what the authors should provide for their method.
- The best dataset to transfer from is evaluated only on 1k samples, which is too small. The accuracy changes ~25% to ~70% accuracy for BigEarthNet when going from 1k samples to full dataset. Hence the trend is unpredictable, and some other dataset could have potentially performed better on full dataset.
- Remove blank page at the end
​
### Suggestions to Improve
For this to be a comprehensive survey on in-domain representation learning in remote sensing, the reviewer recommends to add:
​
1. Standardized representation learning methods to compare across datasets.
2. Results on more downstream tasks (other than classification; examples mentioned in the paper introduction) to evaluate representation learning.
​
The reviewer understands these could be out of scope of this paper. However in its current state, the contributions are not significant enough.
​