nintynine-boundaries 0.3.1
Generate administrative boundaries with OpenStreetMap and Overpass
License: MIT
Requires Python: <4.0,>=3.10
Keywords: osm, openstreetmap, boundaries, geospatial, gis, overpass

# [99boundaries](https://99boundaries.com)

[![PyPI version](https://badge.fury.io/py/nintynine-boundaries.svg)](https://badge.fury.io/py/nintynine-boundaries)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

<img width="920" alt="Indonesia Maritime and Land Boundaries" src="https://user-images.githubusercontent.com/10322094/169665541-701d6de6-7bf9-4e8b-b098-c5a17206eb0b.png">

A simple command line utility to generate administrative boundary polygons from [OpenStreetMap](http://openstreetmap.org) via [Overpass](http://overpass-api.de/) and [OSM Land Polygons](https://osmdata.openstreetmap.de/data/land-polygons.html).

## Features

- Generate maritime and land boundaries for any country
- Support for multiple administrative levels (country, state/province, county, etc.)
- Multiple output formats: GeoJSON, ESRI Shapefile, GeoPackage, GeoParquet, CSV (WKT), MapInfo, KML, FlatGeobuf, PostgreSQL dump
- Batch processing for multiple countries at once
- Uses OpenStreetMap data via Overpass API
- High-quality land boundary generation with OSM land polygons
- Optional data directory cleaning for incremental processing
- Automatic filtering of neighboring country boundaries (50% overlap threshold for admin levels 3+)

## Installation

### For Users

Install via pip:

```bash
pip install nintynine-boundaries
```

### For Development

Please make sure you have installed [poetry](https://python-poetry.org/docs/) locally. Then clone this repository and run `poetry install` from the root folder.

To activate the virtual environment:

```bash
eval $(poetry env activate)
```

**Note:** Maritime boundaries represent the administrative boundaries as defined in OpenStreetMap, which may extend into water bodies. *Land boundaries are generated by intersecting these maritime boundaries with OSM land polygons to create more detailed coastlines*. For landlocked regions or administrative areas without coastal access, the maritime and land boundaries will be almost identical.

### Prerequisites

- Python 3.10 or higher
- GDAL/GEOS libraries (installed automatically with geopandas on most systems)
- Optional: [OSM Land Polygons](https://osmdata.openstreetmap.de/data/land-polygons.html) for generating land boundaries
  - Download the [WGS84 split variant](https://osmdata.openstreetmap.de/download/land-polygons-split-4326.zip) (~600MB) for faster processing
  - Unzip to a local directory and reference with `--land_data_dir` flag 


## Usage & Examples

```sh
make_boundary --help
usage: make_boundary [-h] -a ALPHA2 [ALPHA2 ...] -f FORMATS [FORMATS ...] [-l MAX_ADMIN_LEVEL] [-d LAND_DATA_DIR]
                     [--debug] [--no-debug] [--clean] [--no-clean] [--suppress-warnings] [--no-suppress-warnings]
                     [-o OUTPUT_PATH]

required arguments:
  -a ALPHA2 [ALPHA2 ...], --alpha2 ALPHA2 [ALPHA2 ...]
                        List of ISO-3166-1 alpha2 country codes, e.g. ES FR DE
  -f FORMATS [FORMATS ...], --formats FORMATS [FORMATS ...]
                        Output formats, one or multiple of shp, gpkg, csv, geojson, mapinfo, kml, fgb, pgdump, parquet

optional arguments:
  -l MAX_ADMIN_LEVEL, --max_admin_level MAX_ADMIN_LEVEL
                        Highest OSM administrative level to process. Processes levels 2 through max_admin_level.
                        Read more at https://wiki.openstreetmap.org/wiki/Key:admin_level (default: 2)
  -d LAND_DATA_DIR, --land_data_dir LAND_DATA_DIR
                        Path to the OSM land data polygons folder, read more and download from
                        https://osmdata.openstreetmap.de/data/land-polygons.html
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
                        Output directory path for generated boundary files (default: data/)
  --debug               Enable debug logging
  --no-debug            Disable debug logging (default)
  --clean               Clean the data directory before processing
  --no-clean            Skip cleaning the data directory (default)
  --suppress-warnings   Suppress PyGRIO/GDAL warnings (default)
  --no-suppress-warnings
                        Show PyGRIO/GDAL warnings
```

Depending on the selected country the processing time will vary. While the maritime boundaries will take a few seconds to be generated, the land boundaries will take longer due superior spatial detail. Land boundaries for countries with with greater coverage, e.g. the United States, Canada or France, can take up to 45 minutes to be generated and will require up to 8g of memory. The reasons are resource hungry geopandas overlay operations intersecting the maritime boundaries with the OSM land polygons dataset.

### Basic Examples

#### Generate the maritime boundary as GeoJSON for Australia (admin level 2 only):

```bash
make_boundary --alpha2 AU --formats geojson
```

#### Generate the maritime and land boundary as GeoJSON, GeoPackage and ESRI Shapefile for Spain:

```bash
make_boundary --alpha2 ES --formats geojson shp gpkg --land_data_dir /path/to/land-polygons-split-4326
```

#### Generate the maritime and land boundaries as CSV (geometry as WKT) for France and Canada:

```bash
make_boundary --alpha2 FR CA --formats csv --land_data_dir /path/to/land-polygons-split-4326
```

### Multi-Level Administrative Boundaries

Generate administrative boundaries up to level 11 (e.g., municipalities, districts). See the [OSM admin level documentation](https://wiki.openstreetmap.org/wiki/Tag:boundary%3Dadministrative#admin_level=*_Country_specific_values) for country-specific admin level meanings.

**Important:** Not all admin levels exist for every country. The available levels and their meanings vary by country - consult the [OSM admin level documentation](https://wiki.openstreetmap.org/wiki/Tag:boundary%3Dadministrative#admin_level=*_Country_specific_values) to understand which levels are defined for your region.

#### Generate admin levels 2-6 for the Netherlands with land boundaries:

```bash
make_boundary --alpha2 NL --formats geojson --max_admin_level 6 --land_data_dir /path/to/land-polygons-split-4326
```

This will generate:
- Admin level 2: Country boundary (Netherlands)
- Admin level 3: First-level subdivisions (provinces)
- Admin level 4: Second-level subdivisions (regions/corop)
- Admin level 5: Third-level subdivisions (municipalities)
- Admin level 6: Fourth-level subdivisions (districts/neighborhoods)

Each feature is saved separately with its attributes (name, tags, ID).

#### Generate admin levels 2-4 for Spain with maritime boundaries only:

```bash
make_boundary --alpha2 ES --formats geojson gpkg --max_admin_level 4
```

#### Process multiple countries with higher admin levels:

```bash
make_boundary --alpha2 BE LU --formats geojson --max_admin_level 6 --land_data_dir /path/to/land-polygons-split-4326
```

### Custom Output Directory

Specify a custom output directory instead of the default `data/` folder:

```bash
make_boundary --alpha2 DE --formats geojson --output_path /path/to/custom/output
```

### Incremental Processing

Skip cleaning the data directory to add new data without removing existing files:

```bash
# First run: Generate admin levels 2-4
make_boundary --alpha2 NL --formats geojson --max_admin_level 4

# Second run: Add admin levels 5-6 without removing previous data
make_boundary --alpha2 NL --formats geojson --max_admin_level 6 --no-clean
```

### Warning Suppression

By default, the tool suppresses PyGRIO/GDAL warnings (such as field truncation warnings when exporting to certain formats). To see all warnings during processing:

```bash
# Show all PyGRIO/GDAL warnings
make_boundary --alpha2 ES --formats shp --no-suppress-warnings
```

This can be helpful for debugging issues or understanding what's happening with data conversions.

The file output formats can be ESRI Shapefile, GeoJSON, CSV, GeoPackage, GeoParquet, MapInfo, KML, FlatGeobuf, or PostgreSQL dump and will be saved as archives into the `data` within this repository.

## Output

Generated files are saved as ZIP archives in the `data` folder (or custom output directory if specified) with the following structure:

### All Admin Levels
Each administrative feature is saved in its own directory:
- `{output_path}/{country_code}/{level}/{feature_name}/{feature_name}.{format}.zip` - Boundaries without land intersection
- `{output_path}/{country_code}/{level}/{feature_name}/{feature_name}_land.{format}.zip` - Boundaries intersected with OSM land polygons (if `--land_data_dir` is provided)

For example, with default settings:
- Admin level 2: `data/DE/2/relation_51477_deutschland/relation_51477_deutschland.geojson.zip`
- Admin level 3: `data/DE/3/relation_62761_bayern/relation_62761_bayern.geojson.zip`
- Admin level 4: `data/DE/4/relation_2145268_oberbayern/relation_2145268_oberbayern.geojson.zip`

### Boundary Types Explained

**Files without `_land` suffix** contain the administrative boundaries as returned directly from the OpenStreetMap query. For admin level 2 (countries), these typically include territorial waters and maritime zones. For higher admin levels (3+), these boundaries usually represent land-based administrative divisions but may occasionally include small maritime areas if defined in OSM.

**Files with `_land` suffix** contain boundaries that have been intersected with OSM land polygons. This intersection process:
- Creates precise coastline boundaries by clipping to actual land areas
- Removes maritime/water portions from the administrative boundaries
- Produces detailed shoreline geometry from the high-resolution OSM land polygon dataset
- For landlocked administrative regions, results in identical geometries to the non-land version

Each archive contains the boundary data in the requested format with attributes including:
- `id`: OSM relation ID
- `tags`: Dictionary of OSM tags (name, admin_level, etc.)
- `geometry`: Boundary geometry
- Additional OSM metadata

## Overlap Filtering

To prevent neighboring countries' administrative regions from being included in results, the tool automatically filters admin level 3+ features based on their spatial overlap with the parent country boundary (admin level 2).

**How it works:**
- Admin level 2 (country) boundaries are saved without filtering
- Admin levels 3 and higher are filtered to only include features with ≥50% overlap with the country boundary
- This filtering is applied to both maritime and land boundaries
- Features from neighboring countries that share borders are automatically excluded

**Example:** When querying Germany for admin level 3 (states), any Dutch or Belgian administrative regions that might appear in the Overpass results will be automatically filtered out if they have less than 50% overlap with Germany's boundary.

This ensures that only administrative regions that genuinely belong to the queried country are included in the output, while still capturing legitimate cross-border regions if they exist.

## Known Limitations

- Land boundary generation for countries with large territories (US, Canada, France, Russia) can take 30-45 minutes and require up to 8GB of memory
- Processing higher admin levels (3+) with land boundaries can be time-intensive for regions with many features
- Requires active internet connection to query Overpass API
- Admin level meanings vary by country - refer to the [OSM documentation](https://wiki.openstreetmap.org/wiki/Tag:boundary%3Dadministrative#admin_level=*_Country_specific_values) for country-specific details
- **Shapefile format limitations:** The tags field will be truncated to 254 characters when exporting to ESRI Shapefile format due to the DBF file format's field length restrictions. For complete tag data, use GeoJSON, GeoPackage, or FlatGeobuf formats instead.
- **Large GeoJSON files:** GeoJSON format is automatically skipped for geometries with more than 50,000 coordinates to avoid performance issues and excessive file sizes. For large/complex boundaries, use alternative formats such as GeoPackage (GPKG), FlatGeobuf (FGB), or ESRI Shapefile instead. When GeoJSON is skipped, you'll see a debug log message indicating the coordinate count.

## Troubleshooting

**Memory errors for large countries:**
- The land boundary intersection process is memory-intensive for countries with extensive coastlines
- Consider processing one country at a time
- Ensure you have at least 8GB of available RAM

**Missing GDAL/GEOS libraries:**
- On Ubuntu/Debian: `sudo apt-get install gdal-bin libgdal-dev libgeos-dev`
- On macOS: `brew install gdal geos`
- On Windows: Install via conda or use OSGeo4W

**Overpass API timeouts:**
- The Overpass API may be slow or timeout for very large queries
- Try again later or use the `--debug` flag to see detailed error messages

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request to the [GitHub repository](https://github.com/timmccauley/nintynine-boundaries).

## Future Work

- Performance optimizations for processing large numbers of sub-national boundaries
- Parallel processing support for faster multi-level boundary generation
- Caching of intermediate results to avoid re-processing

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.