Large GeoTiff (raster) files on GeoServer using COG
Tags: geoserver, cloudoptimizedgeotiff, cog, raster
Published: 2021-09-04 19:17:30

### Publish large geotiff (raster) files on GeoServer and be able to access them in no time with COG (Cloud Optimized Geotiff)

The challenge here is that you might have a really large geotiff file (or a normal one). Publishing that file on geoserver would require any client to download the whole file before rendering it (not a good thing if you have a 500MB geotiff!).

The solution is fairly simple with **COG** (Cloud Optimized Geotiff). You can check all about COG on the [official website](https://www.cogeo.org/), or you can simply read their explanation of COG as:
> A Cloud Optimized GeoTIFF (COG) is a regular GeoTIFF file, aimed at being hosted on a HTTP file server, with an internal organization that enables more efficient workflows on the cloud. It does this by leveraging the ability of clients issuing ​HTTP GET range requests to ask for just the parts of a file they need.

Let's do it through these steps together:

- Download and use COG validation script.

Thanks to [rouault](https://github.com/rouault/) for creating this awesome [script](https://github.com/rouault/cog_validator/blob/master/validate_cloud_optimized_geotiff.py). Save it to your local machine, we will use it in a second.

- Get your large (or normal) geotiff file in the same directory as the validation [script](https://github.com/rouault/cog_validator/blob/master/validate_cloud_optimized_geotiff.py).

For the sake of this demo, I will use this [geotiff](http://leoworks.terrasigna.com/files/Envisat_ASAR_2003-08-04.tif), download it, rename it to `envisat.tif` and save it next to the validation [script](https://github.com/rouault/cog_validator/blob/master/validate_cloud_optimized_geotiff.py).

- Validate the file that it is a geotiff, but not yet a COG.

run the following command:
```
python validate_cloud_optimized_geotiff.py envisat.tif
```
you should see an output telling you that it is `NOT a valid cloud optimized geotiff` like this screenshot.
![not a valid cog file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lwn8w72tz30c02etitpl.png)

- Convert the geotiff file into a cloud optimized geotiff file (COG).

In order to do that, we will use GDAL, make sure you have it installed on your machine. for GDAL 3.2 and later, it's a simple one line command, for older versions than 3.2, it's two separate commands. We will cover both cases.

* GDAL >= 3.2
run this command on the geotiff file
```
gdal_translate envisat.tif envisat_cog.tif -of COG -co COMPRESS=LZW
```
The larger the file, the more time it will take to convert it, at the end you should see an output like this.
![convert to cog gdal > 3.2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gzzvhrqxfhc38z9dpmao.png)

Now let's try and validate the file again, run the validation command again, but this time on the **output** file:
```
python validate_cloud_optimized_geotiff.py envisat_cog.tif
```
you should see and output message telling you that `envisat_cog.tif is a valid cloud optimized GeoTIFF` like this one.
![valid cog file 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eiko44bdr4zlnnkzqsog.png)

* GDAL < 3.2
We have to make this manually by creating the internal overviews, then create the tiles in the files.

Run the following command to create the internal overviews:
```
gdaladdo -r average envisat.tif 2 4 8 16
```
It can take some time depending on the file size, but when it completes, we are ready for the next step which is creating the tiles. Run the following command on the file:
```
gdal_translate envisat.tif envisat_cog.tif -co COMPRESS=LZW -co TILED=YES -co INTERLEAVE=BAND
```
This command also can take some time depending on the file size, but when it completes, you should have a valid COG file. We can easily validate that, by running the validation script on the **output** file:
```
python validate_cloud_optimized_geotiff.py envisat_cog.tif
```
It should tell you that `envisat_cog.tif is a valid cloud optimized GeoTIFF`

## Halfway done!

You can now have a break and bring in a cup of coffee ☕ (or tea) since we made some good progress here. We have our COG file ready to be served by GeoServer.

Let's continue.

GeoServer doesn't support COG files out of the box, luckily there is a community module that adds support to that.

- Go ahead and open GeoServer's [nightly build](https://build.geoserver.org/geoserver/) directory.

- Choose your version of GeoServer `2.16.x` `2.17.x` `2.18.x` `2.19.x`

- Open `community-latest`

- Search for the word `cog`, you should have only **1** search hit, which is a module named `geoserver-2.x-SNAPSHOT-cog-plugin.zip` where `x` is GeoServer minor version.

- Download the module's zip file and extract the content to GeoServer's `WEB-INF/lib` directory then **restart** GeoServer.

### Congratulations! GeoServer now supports COG

- Open up GeoServer's homepage (usually at localhost:8080/geoserver) and login with your credentials (usually admin:geoserver)

- From the left side menu, under `Data`, select `Workspaces`, then click on `Add new workspace`
![GeoServer Data menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kn8zi6sa1cwxh1dj7wun.png)
![Add new workspace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hw4agfdayi7we3qsyrj6.png)

- Create a new workspace called `cog`
![cog workspace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/not8cyvf0d2dqoct064u.png)

- Navigate to GeoServer's data directory, create a folder called `cogs` then copy and paste or cog file `envisat_cog.tif` into that directory.

- Back to GeoServer's console, from the left side menu, under `Data`, select `Stores`. Every geotiff (COG) file you publish is a complete store. click on `Add new store`.
![GeoServer Data menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kn8zi6sa1cwxh1dj7wun.png)
![Add new store](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t0ffru7oc3c3okiezuti.png)

- From the long menu of GeoServer's data sources, under `Raster Data Sources`, click on `GeoTIFF`.
![New GeoTIFF](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/46dwr55zmcy3fwmcdkqk.png)

- A wizard will appear, for `Workspace` select our newly created workspace called `cog`.
- For the `Data source name`, I will type `envisat`.
- For the `Description`, I will type `envisat_store`.
- Next to the `URL` field, click on the `Browse` button and select the `envisat_cog.tif` file.
![Browse for Tif file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ewlgfilvautwe6kfxic0.png)
- Mark the checkbox `Cloud Optimized GeoTIFF (COG)`
Leave all the other settings as is, you should have something like this.
![Add new Raster Data Source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9p33d405yln9eg8onrfi.png)
- Click `Save`

- You'll find yourself in the wizard that creates a new layer (that is a good thing!), and your new layer is ready to be published.
![New Layer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6jy5tftxwjis3szmaymp.png)
- Click on `Publish`

- The wizard for `Edit Layer` is opened automatically, I will change the layer name to `envisat` and the title to `Envisat`, then I will click `Save`.
![Save Layer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o0lppooiur0csk9w95m5.png)

- The layer is created and added to the layers list.
![Layer List](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dr8otzio88txjwbcekfb.png)

- Preview the layer from the `Layer Preview` menu item under `Data` from the left menu.
![Layer Preview Menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5tcsjaygh8gs1f04a0kr.png)

- Click on `OpenLayers`

It Does not matter how large your layer is, it will load up in seconds (or less), and when you zoom in/out it will load up almost instantly thanks to **COG**.

![Preview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4puhsgwepmayw1wetn7y.png)