Developed Python CLI to easily download Dutch point cloud data
Tags: python, pointcloud, gis, las
Published: 2024-05-04 09:41:53

# Building a CLI Tool for Downloading AHN Point Cloud Data

Hey there, fellow developers! I recently had the opportunity to work on a really cool project—developing a Command-Line Interface (CLI) tool that makes it a breeze to download Actueel Hoogtebestand Nederland (AHN) point cloud data for specific cities. Here is the GitHub link. https://github.com/HideBa/ahn_cli I thought I'd share my experience with you all, so buckle up and let's dive in!

## The Objective

The main goal was to create a tool that would streamline the process of getting geospatial data, making it easier for various GIS applications like urban planning and environmental analysis. I mean, who doesn't love a good time-saver, right?

## The Journey

### Step 1: Simplifying City Polygons

First things first, I had to create simplified polygons of Dutch cities to make processing smoother. I found a comprehensive list of cities from the Nationaal Georegister [GeoNetwork](https://www.nationaalgeoregister.nl/geonetwork/srv/dut/catalog.search#/metadata/208bc283-7c66-4ce7-8ad3-1cf3e8933fb5). But here's the catch—the original polygons were so detailed that the software couldn't handle them efficiently. So, I put on my problem-solving hat and used the Douglas-Peucker Algorithm in QGIS to simplify those polygons. Voila! Optimized polygons that didn't compromise much on geographic accuracy.

### Step 2: GeoJSON Conversion

Next up, I converted the original dataset from a Web Feature Service (WFS) to GeoJSON format. This made it way easier to work with the data as standalone files and boosted the CLI tool's data handling capabilities.

### Step 3: GeoTiles Grid Data Prep

I downloaded the GeoTiles grid data from [fwrite.org](https://weblog.fwrite.org/kaartbladen/), which is crucial for identifying relevant point cloud data segments. Then, I converted the shapefile to GeoJSON to keep everything consistent and user-friendly.

## The CLI Magic

Now, let's talk about how this CLI tool actually works its magic:

1. **User Input:** The CLI tool starts by receiving parameters from the user, like the city name or geographic coordinates.
2. **Tile Detection:** It uses the simplified city polygons and GeoTiles grid data to find overlapping tiles by checking for intersections. This step figures out which point cloud data segments the user needs.
3. **Data Download:** Once the relevant tiles are identified, the CLI tool downloads the corresponding LAZ formatted point cloud data from the GeoTiles source.
4. **Data Manipulation:** This is where the real fun begins! I created a data manipulation process with numpy and geopandas. It can filter by AHN classification classes, clip to the city polygon, apply user-defined polygonal clips, decimate, and even do radius-based clipping. Talk about flexibility!
5. **Output Generation:** Finally, the processed data is saved in the user's preferred format (LAS or LAZ), ready to be used in further GIS analyses or applications.

## Challenges and Solutions

Of course, no project is complete without a few hurdles. Here are some of the issues I faced and how I tackled them:

1. **Slow AHN download:** Even for the smallest city in the Netherlands, the download took forever! I implemented multi-threading to download data simultaneously, reducing the total download time by around 8x.
2. **Slow point cloud data manipulation:** Initially, I simply used laspy and numpy, but it was taking way too long. To speed up polygon intersection test, I employed rasterisation to make time complexity log(O). That results in way faster than before.
3. **Runtime errors galore:** Dynamic typing in Python led to many runtime errors. Introducing a type system helped validate the program and improved productivity.
4. **Invalid user parameters:** Validating parameters on the fly wasn't user-friendly, so I implemented validation that runs immediately after the CLI is invoked.
5. **Large LAS/LAZ file sizes:** To help users deal with large file sizes, I added options for decimation and radius clipping, allowing them to reduce the number of points and focus on specific areas.

## Wrapping Up

There you have it, folks! Developing this CLI tool was an incredible learning experience, and I hope sharing my journey has given you some insights and inspiration for your own projects. Remember, every challenge is an opportunity to grow and find creative solutions. Happy coding!