(almost) Everything I know about building an open source web application
Tags: gis, mapping
Published: 2019-12-29 02:51:01

I have been building web mapping apps for some time and thought I'd jot down some notes on the subject.

I'll cover:
- Javascript libraries used to create a map on the web
- Geospatial data formats
- Getting data from point A (server / cloud) to point B (the web app / internet browser)


### Software Resources
First you're going to want to select a front end mapping API. There are many available, but I'll just focus on a few.

- 2D
    - [MapBox-GL](https://docs.mapbox.com/mapbox-gl-js/api/)
    - [Leaflet] (https://leafletjs.com/)
    - [Openlayers] (https://openlayers.org/)
    - [Deck-GL](https://deck.gl/) (2.5D)


- 3D
    - [CesiumJs] (https://cesium.com/cesiumjs/)
    - [Vadstena] (https://www.melowntech.com/products/vadstena/#content) by Melown tech
    - [Harp](https://www.harp.gl)

- Geospatial DataBase
    - [PostGIS] (https://postgis.net/) an extension of [PostgreSQL] (https://www.postgresql.org/)
    - [GeoCouch] (https://github.com/couchbase/geocouch/) (an extension of [CouchDB] (https://couchdb.apache.org/))
    - [MongoDB] (https://www.mongodb.com/)

- Middleware
    - [GeoServer] (http://geoserver.org/)
    - [GRASS GIS](https://grass.osgeo.org/)
    - [MapServer](https://mapserver.org/)

- Online Geospatial Tools
    - [Turf.js] (https://turfjs.org/)
    - [GeoJson.io] (https://geojson.org/)
    - [Maputnik] (https://maputnik.github.io/editor/#0.87/0/0)


## 2D

##### Pick Your API

Ok, so you are after a 2D web mapping application, let's get started. Each of the above open source 2D web mapping apis are great and would probably fit the bill for most applications, but there are a few differences that might be worth considering. Leaflet might be the best bet for a basic project, while OpenLayers is probably better suited for  advanced applications. I tend to use MapBox-GL for most apps. I'll get into some reasons why a bit later in the post.

##### Basemap

You are going to need a basemap. A basemap provides a background of geographical information for the map, it can be a satellite and or street map or a hybrid of the two. For most situations it is going to make sense to get a basemap from some kind of mapping provider. A few include: [OpenStreetMaps](https://www.openstreetmap.org/), [Bing Maps] (https://www.bing.com/maps), some Googling will get you the rest of the way. [MapBox studio] (https://www.mapbox.com/mapbox-studio/) will allow you to create a custom styled basemap (based on OpenStreetMap data).

##### Scalability

Before we get too far, we need to think for a moment about the data that you want to render in the map. If a lot of data needs to be rendered in the map, we can run into scalability issues if we are not careful. Most data formats that we will deal with in 2D maps fall into two categories: vector and raster data. Vector data can be points, lines and polygons. Raster data is generally served to the front end mapping environment as tiled images. If your application does not require scaling tricks, you can render vector data directly from [GeoJSON] (https://geojson.org/). **Tip** [online GeoJson editor] (http://geojson.io/) is a hand tool for creating and editing GeoJSON.

In the early days of web mapping, it was easy to encounter performance issues when rendering too many vector features in a web browser. A solution was to burn the vector features into a raster on the server side, and then render the images in the web application. This is still a common approach with middleware solutions like [GeoServer] (http://geoserver.org/) or [MapServer] (https://mapserver.org/).

Another (more modern) approach is to use [MapBox vector tiles (MVT)] (https://github.com/mapbox/vector-tile-spec). With MVTs you can render vast amounts of vector data to the web. MVTs can be generated (commonly on the server side) and served by a cloud provider or you can generate and serve them yourself.

MapBox-GL and OpenLayers can render MVT, but Leaflet cannot without an extension. The MapBox-GL API provides integrated client side filtering and styling directly on the MVT spec. This can be a significant advantage if you plan to use MVT extensively in your app. **Tip** [T-Rex](https://github.com/t-rex-tileserver/t-rex), [Tegola](https://github.com/go-spatial/tegola), [Tippecanoe] (https://github.com/mapbox/tippecanoe) and [Geojson-VT](https://github.com/mapbox/geojson-vt) can be used to create MVTs.

#### Navigation

I have never build a navigation app, but there are a lot of navigation tools out there:

- [OSRM](http://project-osrm.org/) (open source)
- [Open Source Routing Machine](https://github.com/Project-OSRM/osrm-backend) (open source)
- [Graphhopper](https://github.com/graphhopper/graphhopper) (open source)
- [Valhalla] (https://www.mapzen.com/blog/valhalla-intro/) (open source)
- [MapBox](https://docs.mapbox.com/api/)
- [Google Maps](https://developers.google.com/maps/documentation/directions/intro?hl=en)
- [Bing](https://docs.microsoft.com/en-us/bingmaps/v8-web-control/map-control-concepts/directions-module-examples/?redirectedfrom=MSDN)
- [ESRI](https://developers.arcgis.com/features/directions/)
- [Mapquest](https://business.mapquest.com/products/routing-directions-api/)  

##### Data Manipulation

You may want to allow a user to tinker with the data, for example modify the location of a vector point. In this case you cannot modify raster images or MVT data directly from the font end application. One solution is to identify the modification in the front end, then send the change server side and then update the data, you will then need to re-render your data front end, however this could be troublesome, let me explain why: 

Suppose you are burning your vector data into rasters, you may want to cache these images (caching is a really good idea, we have not talked about rendering raster tiles in detail, but in a nut shell you will need to render a pyramid of tiles, each layer of the pyramid will be needed for all of the zoom levels available to your web mapping application). Now you have just changed a feature in your data, in order to make the data change visible in the map you will need to blow away your cache and start burning new images, this cache management can be a painful and expensive.

If you are using MVT, a solution could be to first request the GeoJson data for a feature from the server or geospatial middleware (even if you are rending the bulk of the data using raster images or MVT). Then, you can manipulate that GeoJson feature on the front-end with [Turf.js] (https://turfjs.org/), then pass your newly updated GeoJSON feature back to the server for a database update. Next, you will need to update the MVT and manage the cache, be carful, often times the MVT are cached client side. The best solution I have found is to isolate the layers that may get frequent updates from layers that are not often getting updated. Then modify the MVT headers such that the cache refresh rate is very low. This is not a very elegant solution, so if someone has better ideas I'm all ears.

## 3D

Because 3D web rendering is kind of new, I have only had the opportunity to deep dive into CesiumJS, so I'm going to talk mostly about that.

Raster tiles can be rendered in a 3D environment. Vector formats like GeoJSON can also be rendered in a 3D environment, however, similar to the 2D environment, if you render a lot of vector features you can run into performance issue quickly.

When viewing a map in 2D, you are looking at the data from a top down - birds eye view. Raster and vector formats are well suited for this perspective. However, when we start looking at the map in 3D it is interesting to explore different ways of rendering data. For this reason, Cesium has developed [3D tiles] (https://github.com/AnalyticalGraphicsInc/3d tiles) and [quantized mesh terrain format] (https://github.com/AnalyticalGraphicsInc/quantized-mesh) formats that can be used to render point-cloud, 3D models and terrain data.
 
#### 3D Tiles

In [CesiumJS](https://cesium.com/cesiumjs/) or [Deck-GL](https://deck.gl/#/) ecosystems 3D tiles can be used to render data in the map. 3D tiles are a spatial data structure that enable hierarchical level of detail (HLOD) so only visible tiles are streamed to the front-end map.

With 3D tiles you can encode a bunch of data formats into 3D tiles and then render 3D tiles in CesiumJS. The formats include:

- glTF (.gltf, .glb)
- CityGML (.citygml, .xml, .gml)
- KML/COLLADA (.kml, .kmz)
- LASer (.las, .laz)
- COLLADA (.dae)
- Wavefront OBJ (.obj)

When the above data formats are wrapped into the 3D tiles spec and rendered in the web, CesiumJS will "define a spatial hierarchy for fast streaming and precision rendering, balancing performance and visual quality at any scale from global to building interiors". This is a necessity when rendering very large data sets like point clouds. It's worth mentioning that ESRI has a similar open source spec called [i3s] (https://github.com/Esri/i3s-spec), shoot me a note if you know of an open source i3s web consumer.

Cesium also developed a terrain spec, with it you can take a digital elevation model (DEM) and convert it into a quantized mesh terrain format. You can create and serve 3D tiles and terrain in the cloud via [Cesium ion] (https://cesium.com/cesium-ion/) or you can create them using open source tools like [Entwine] (https://github.com/connormanning/entwine), for point cloud -> 3D tiles and [cesium terrain builder docker] (https://github.com/tum-gis/cesium-terrain-builder-docker) for DEM -> terrain. **Side note** if you are after a simple point cloud web render Cesium might be over kill and you might want to check out [Potree] (http://potree.org/). **Tip** [lopocs] (https://github.com/Oslandia/lopocs) is an open source point-cloud server. **Pro-tip** check out [PDAL] (https://pdal.io/) for all your point cloud translating and manipulating needs.

## Back-End

The data that you want to render on the map is going to be saved somewhere, the two conventional solutions for this are a database or a geospatial cloud service. There are a number of geospatial hosting / serving solutions out there, MapBox, Here, Esri, Carto, MapZen to name a few. This could be a good solution if you want to pay a monthly fee and be done. But this post is on open source solutions so I am going to focus on that. You might need a geospatial database depending on the complexity and size of your data and your application requirements. **Tip** [Geo-Stackexchange] (https://gis.stackexchange.com/) is a great resource for geospatial software development questions.

#### Vector Data

A geospatial database will allow you to geo-index your data, this allows the data to be accessed much more efficiently. Geospatial databases also allow you to do geospatial operations on the data. For example you might want to find all of the points that are within a given distance from another point. The most feature rich open source geospatial database is [PostGIS] (https://postgis.net/) which is an extension of [PostgreSQL] (https://www.postgresql.org/). If you are looking for a noSQL option then [GeoCouch] (https://github.com/couchbase/geocouch/) (an extension of [CouchDB] (https://couchdb.apache.org/)) may be a good fit, or if you just need to serve up some GeoJSON features [MongoDB] (https://www.mongodb.com/) might be all you need. **Pro-tip** [GDAL] (https://gdal.org/) is a translator library for raster and vector geospatial data formats.

#### Raster Data

PostGIS is probably the best option for hosting raster geospatial data. In order to get those raster to the web, you are going to need a geospatial middleware, see below for details. **Tips** [rasterio](https://github.com/mapbox/rasterio) is a nifty little raster project that can read and write geospatial raster data and [QGIS](https://qgis.org/en/site/) is an open source GIS tool I use for pre-processing geospatial data.

## Geospatial Middleware

A geospatial middleware is used to translate or transform data from the database or from a directory into a format that is easily interpreted by the mapping API in the front-end. For example, if you have some vector data in your database or in a file directory, the middleware can generate and serve MVT to the front end web application. Middleware can also generate and serve raster tiles from a raster source file. [GeoServer] (http://geoserver.org/), [GRASS GIS](https://grass.osgeo.org/) and [MapServer](https://mapserver.org/) are a few great middleware tools. A geospatial middleware is also going to help you by moving geospatial data around in an [OGC] (https://www.opengeospatial.org/standards) standard format (which is a good idea!). **Pro-tip** [GeoNode](http://geonode.org/) is a full-stack open source geospatial content management solution.

## Wish List

If I could snap my fingers and change anything in the open source web mapping world, I would love to see a 3D vector tiles spec developed for the CesiumJS ecosystem.

Also, I would change the way that the MapBox-GL API manages the basemap layer. If you render a basemap and then add some raster or vector layers on top of the basemap and then change the basemap for example from a street map to a satellite map, you will need re-order all of your layers to get back to the map state as it was before the basemap was changed.

## Community

If you're like me and get hooked on this stuff, you'll also want to check out [FOSS4G](https://foss4g.org/) a yearly open geospatial technology and business conference organized by the [Open Source Geospatial Foundation](https://www.osgeo.org/). They also have regional flavors of the conference. Hope to see you there!