SamGIS - Segment Anything applied to GIS
Tags: python, machinelearning, computervision, maps
Published: 2024-05-24 12:07:11

## SamGIS - Segment Anything applied to Geographic Information Systems (GIS)

This project, composed by a [SPA](https://en.wikipedia.org/wiki/Single-page_application) frontend and by a backend ([samgis-be](https://github.com/trincadev/samgis-be/)), is an attempt to perform machine learning image segmentation on geo-spatial data even without the use of dedicated graphics cards. I recently added the ability to start from [natural language text prompts](https://trinca.tornidor.com/projects/lisa-adapted-for-samgis).

I wrote the backend adapting [SAM Exporter](https://github.com/vietanhdev/samexporter) with the aim to use the machine learning [segment anything](https://segment-anything.com/) project for the purpose to improve polygons recognition in GIS web applications.

Starting from version 1.5.1 the backend integrates changes borrowed from [sam_onnx_full_export](https://github.com/AndreyGermanov/sam_onnx_full_export), to support OnnxRuntime 1.17.x and later versions. Please note that on MacOS directly running the project from the command line suffers from [memory leaks](https://github.com/microsoft/onnxruntime/issues/14455), making inference operations slower than normal. It's best therefore running the project inside a docker container, unless in case of development or debugging activities.

These are the project resource links:

- a public [HuggingFace space demo](https://huggingface.co/spaces/aletrn/samgis)
- the [documentation and AWS lambda OpenAPI specs](https://docs.ml-trinca.tornidor.com)

Note that the HuggingFace space demo has its static frontend files under the /static folder.

Ask me for a user via [linkedin](https://www.linkedin.com/in/trincatornidor/) if you would like to test the authenticated demo or an explanation about the code.

### LISA and SamGIS

I also added support for image segmentation via natural language text prompts: here are the details about the [SamGIS integration with LISA](https://trinca.tornidor.com/projects/lisa-adapted-for-samgis).

## SamGIS also replaces SurferDTM

[SurferDTM](https://trinca.tornidor.com/projects/surferdtm-docs) was an attempt to find [stratovolcanoes](https://volcano.oregonstate.edu/stratovolcanoes) within digital terrain models (DEM/DTM). As a new feature SamGIS now can now use also input DEM images (like the first image in this gallery):

![samgis-dem-mapzen](https://res.cloudinary.com/aletrinok/image/upload/v1701206785/samgis/samgis-dem-1.png)

![samgis-dem-bw](https://res.cloudinary.com/aletrinok/image/upload/v1701206785/samgis/samgis-dem-2.png)

Note that the "terrarium" source provider data encode the image as RGB, so the backend processes it to obtain the elevation data (like the second image in this gallery) using this formula (from the <a href="https://www.mapzen.com/blog/terrain-tile-service/">map tile provider page</a>):

> (red * 256) + green + (blue / 256) - 32768

I also added a custom elaboration step to create an intermediate RGB image composed by channels:

- RED - normalized DEM image
- GREEN - normalized 2d composed image by
  - normalized DEM image
  - slope
  - curvature
- BLUE - curvature

![samgis-dem-rgb](https://res.cloudinary.com/aletrinok/image/upload/v1701206785/samgis/samgis-rgb2-1.png)

Segment Anything will finally do the inference on this rgb image. This is the json payload request:

```json
{
   "bbox": {
         "ne": {"lat": 46.203706648934954, "lng": 9.401906739170105},
         "sw": {"lat": 46.12464369105346, "lng": 9.231103669101747}},
   "prompt": [
         {"id": 238, "type": "point", "data": {"lat": 46.158760519552146, "lng": 9.306795638666486}, "label": 1},
         {"id": 250, "type": "point", "data": {"lat": 46.152693546080975, "lng": 9.289288652508679}, "label": 0},
         {"id": 264, "type": "point", "data": {"lat": 46.16613515509541, "lng": 9.319840059725284}, "label": 0},
         {"id": 272, "type": "point", "data": {"lat": 46.161377438862836, "lng": 9.288087192674299}, "label": 0},
         {"id": 285, "type": "point", "data": {"lat": 46.174103408003454, "lng": 9.308340372739261}, "label": 0},
         {"id": 293, "type": "point", "data": {"lat": 46.17005996124035, "lng": 9.298042145587583}, "label": 0}
   ],
   "zoom": 13,
   "source_type": "nextzen.terrarium"
}
```

## Project architecture of AWS demo

![project-architecture](https://res.cloudinary.com/aletrinok/image/upload/v1701206785/samgis/photo1.png)

1. The user interact with a WebMap (in my example it's a leaflet map that use OpenStreetMap map data).
2. The user send a POST request to an [API](https://docs.ml-trinca.tornidor.com/openapi) using
   1. the current map extent
   2. the current map zoom
   3. a prompt array of objects; objects could be
      - an "include" marker position (a couple of float - latitude and longitude)
      - an "exclusionary" marker position (a couple of float - latitude and longitude)
      - a "include" rectangle (two latitude and longitude positions - north-east and south-west)
3. because of CORS constrain it's difficult to directly send requests to remote resources on different domains: to avoid this problems I prepared a simple Cloudflare proxy function that forward the user request to the remote API
4. The API Gateway proxies the request to the serverless backend
5. the serverless Lambda API elaborate the request:
   1. it parses the request and in particular translate the input latitude/longitude prompt into a x/y (pixel coordinates) one
   2. it downloads the geo-referenced tiles then merge and crop them
   3. it instantiates a [MobileSam](https://github.com/ChaoningZhang/MobileSAM) model instance if not already created using OnnxRuntime as runtime
   4. it prepares the [image embedding](https://datasciencedojo.com/blog/embeddings-and-llm/#) and...
   5. ...it starts the image inference process
   6. it transforms the recognition masks into a single binary mask then it [vectorizes](https://gisgeography.com/rasterization-vectorization/) the recognized binary mask:
      1. first it converts the mask into a GeoPandas GeoDataframe using [`rasterio.features.shapes`](https://rasterio.readthedocs.io/en/stable/api/rasterio.features.html#rasterio.features.shapes) and [`GeoDataFrame.from_features`](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.from_features.html)
      2. then it convert the new Geodataframe into a geojson using [`GeoDataFrame.to_json`](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.to_json.html)
6. the Cloudflare proxy function parses the response and forward it to the frontend
7. the frontend parse the geojson to load within the WebMap and shows some response info like the [non-relativistic response duration time](https://xkcd.com/2867/):

   ![datetime-diff](https://imgs.xkcd.com/comics/datetime.png)

To avoid abuse the [prompt input web map that I host on my personal site](https://ml-trinca.tornidor.com/prediction-map) is under authentication (I use [auth0.com](https://auth0.com)).

## Fastapi-based demo architecture

This project variant works almost as [this](https://trinca.tornidor.com/projects/samgis-segment-anything-applied-to-GIS#project-architecture-of-aws-demo). There are these differences:

- `opencv-python` is not anymore a SamGIS project dependency
- the server now saves and re-uses image _embeddings_ to avoid resource waste and speed up the inference process
- SPA frontend now is integrated directly within the Fastapi backend

Frontend and backend now are on the same domain. Therefore there is no longer use for a proxy (the AWS demo used a [Cloudflare Pages function](https://developers.cloudflare.com/pages/functions/)) to make requests to APIs on a different domain.

Additionally, since I can [host this Fastapi demo](https://huggingface.co/spaces/aletrn/samgis) for [free on HuggingFace](https://huggingface.co/pricing#hub), I don't need anymore an authentication system.

## AWS lambda, a serverless solution: some pros...

- It can work also with docker images for complex or big projects (like this)
- Almost easy, working out-of-the-box integration with other AWS services and third-parts authentication systems
- Pay-as-you-go service

## ...and some cons

- Setup not always easy (I used [Powertools for AWS Lambda (Python)](https://docs.powertools.aws.dev/lambda/python/latest/) to improve my logger setup)
- The serverless model suggested by AWS [not always it's the best solution](https://offbynone.io/issues/233/)
- Arm graviton EC2 instances that run AWS lambda are greatly limited ([OnnxRuntime at the moment cannot run on Arm AWS lambda and probably will never run](https://github.com/microsoft/onnxruntime/issues/10038)).