Deep learning models using satellite data
Published: 2022-10-25 19:00:02

## Introduction
In this post, I'd like to share my experience developing computer vision models using remote sensing techniques and some neat characteristics that make them unique in the machine learning world. I don't claim to be a remote sensing expert, just an enthusiast.

First of all, there are some things I need to address so we are on the same page. Images combine RGB (Red, Green, and Blue) bands. For each pixel, there is an RGB vector indicating the magnitudes for each band. See the picture:

![RGB matrix in an image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l59pt7n8n8qktjsl8a28.png)

For the first pixel top left, red/green/blue intensities are 253, 14, and 165, respectively. 

Satellite imagery extends this concept beyond the RGB frequency intervals. So there are values for infra-red and other intervals outside the visible spectrum. See this [link](https://eos.com/make-an-analysis/) for more info.
&nbsp;

![Bands in remote sensing data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yn2lqrs20dnr0h5n8hgz.png)

The number of available bands depends on the satellite from which you're getting the data. Also, most of them have an 8-bit or 16-bit discretization. The number itself doesn't mean much, thus making it remarkably hard to combine images from different satellites. That comes with limitations.
&nbsp;
&nbsp;

## Challenges
&nbsp;

### Clouds. Lots of clouds. And they matter

![Cloud example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i0x0acqi2d3k3w321043.png)

Good luck trying to tell what's going on behind these clouds. With optical imagery, that's nearly impossible. There has to be a cloud filter to improve your Signal-to-Noise Ratio (SNR).
&nbsp;

### Spatial and Temporal resolution

Each satellite has a spatial resolution that indicates the distance represented by a pixel in an image (see the picture below). Furthermore, the spatial resolution can vary between bands - so if you want to work with all of them, you'll have to interpolate some of them.

![Spatial Resolution examples](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d7c4025h6078fk5bit51.png)

Also, each satellite has its own re-visit time (or temporal resolution). That is the period in time to produce an image from the same area in sequence.

![Temporal resolution table](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xrn68hz1roj77rreo3bv.png)

These things are inversely proportional - satellites with better temporal resolution have a worse spatial resolution and vice-versa (see image below). There is a way to circumvent this by launching a large constellation (4+) of equivalent high-spatial-resolution satellites, which is incredible but very expensive.

![Spatial vs Temporal Resolution comparison](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w0hz6o348e1osqdbevri.png)

So when you combine clouds with the temporal resolution? Sometimes it takes 20~30 days to have a decent image of a region which is incredibly bad for time-sensitive models. Also, smaller areas are a big problem if you don't have high enough spatial resolution.  
&nbsp;
&nbsp;

## Lessons learned
&nbsp;

### 1. Get good data

First of all, **good data is fundamental**. You won't get anywhere without it (even if you change your architecture 1000x times). Your job as a data scientist will be much easier. Remove outliers. Balance your dataset. Remove noise. Standardize. Do the most basic data scientist work.
&nbsp;

### 2. Removing clouds improves your model
This lesson should be intuitive, given the first one, but clouds are just noise. It is safer (and more efficient) to remove clouds by filtering them before feeding your dataset to a neural network. 

You can let the neural network learn that a gigantic blank portion of an image is noise, but in my experience, that'll give you slightly worse results.
&nbsp;

### 3. Learn how to deal with GeoTIFF and GeoJSON
Forget PNG, JPG, BMP, etc. They are not appropriate to carry geographical information or an arbitrary number of bands. With the GeoTIFF, you can have all of that /o/ GeoJSON is easy; you can deal with it quite well using the python `geojson` and `shapely` packages. You're going to need a specialized package to deal with GeoTIFF (I recommend the [`rasterio`](https://rasterio.readthedocs.io/en/latest/) package - it helped me create masks for segmentation models).
&nbsp;

### 4. Most pre-trained models are useless
Most machine learning applications, especially those that deal with images, benefit from **transfer learning**, i.e., using a backbone network (like ResNet50 trained on ImageNet) to extract features and then adding a few linear layers to adjust for your problem.

Even though satellite images are images (duh), their features aren't shapes but band values over time and their relationships. Often there's no predictive power in a portion of land being round-shaped, square-shaped, or other (see an [exception](https://en.wikipedia.org/wiki/Center-pivot_irrigation)). What models learn is if a band value or an index such as the [NDVI index](https://gisgeography.com/ndvi-normalized-difference-vegetation-index/) is high, for instance.

Thankfully, most ML tasks in this area don't require a complex model to work; **your dataset quality is much more important than the architecture**.
&nbsp;

### 5. The DataLoader should use tensors
The most common transformations are
- RandomRotation
- RandomCrop
- RandomVerticalFlip
- RandomHorizontalFlip

They only work for PIL Images or PyTorch Tensors with shape [..., H, W]. Often you can't transform multi-band images to PIL; you have to work with tensors. Some transformations like [ColorJitter](https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html) might not work, so you'll have to code them yourself.
&nbsp;

### 6. You'll need to code the Dataset logic yourself
Even for classification tasks, you can't rely on splitting the classes into folders and using the nice [ImageFolder](https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html#torchvision.datasets.ImageFolder) wrapper, because they also use PIL Images only. You'll need to create your custom Dataset and deal with all the image transformations. 
&nbsp;

### 7. Tensor manipulation can be tricky
Sometimes you'll have to work with four or 5-dimensional tensors.
For example, my input tensors normally look like this:
`input_tensor.shape = (batch_size, n_bands, height, width)`

To create dynamic-sized patches of images, for testing purposes, I use PyTorch's  [Unfold](https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html) function). 

A new dimension emerges, fold_size. Then you have to manipulate this to put it into the model's input shape and reshape it again to get the whole image. It's cumbersome to make it work.

Ex: My input image is (234,221), but the model accepts only (128,128). So I pad my image with zeros to make it (256,256), unfold it to make it four (128,128) images, pass through the model, fold the outputs to get to (256,256) again and then return it to the original shape. Phew!
&nbsp;

## Conclusion
This article is a sneak peek into computer vision models made with satellite images. Of course, there's much more to it that I couldn't cover here. The most important is assuring data quality - I dare say about all machine learning models. Hopefully, you have a better understanding now of the challenges I face every day and found a helpful tip :)