gpsdecode and ogr2ogr for AIS messages
Tags: tutorial, bash, gdal, gnuplot
Published: 2024-02-14 19:00:35

## Access to AIS data

[Kystverket (Norwegian Coastal Administration)](https://www.kystverket.no/en/navigation-and-monitoring/ais/access-to-ais-data/)

> AIS data are also available at the following IP address: 153.44.253.27 port 5631

> The AIS data stream is in standard IEC format IEC 62320-1.

See https://www.confluent.io/blog/streaming-etl-and-analytics-for-real-time-location-tracking/

## OpenCPN

We start by viewing the AIS data from 153.44.253.27 port 5631 using [OpenCPN](https://opencpn.org/).

![AIS data from 153.44.253.27 port 5631 using OpenCPN](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/txfr6srjvx2zkcq1ksfa.png)

## AIS and NMEA message format

https://documentation.spire.com/tcp-stream-v2/

https://www.itu.int/rec/R-REC-M.1371-5-201402-I/en

[AIVDM/AIVDO protocol decoding. Eric S. Raymond. Version 1.58, 24 June 2023.](https://gpsd.gitlab.io/gpsd/AIVDM.html)

## ncat

[Ncat Users’ Guide. Nmap](https://nmap.org/ncat/guide/index.html)

> Ncat is a feature-packed networking utility which reads and writes data across networks from the command line. Ncat was written for the Nmap Project as a much-improved reimplementation of the venerable Netcat. It uses both TCP and UDP for communication and is designed to be a reliable back-end tool to instantly provide network connectivity to other applications and users. Ncat will not only work with IPv4 and IPv6 but provides the user with a virtually limitless number of potential uses.

```bash
timeout 3600s ncat 153.44.253.27 5631 > ais_2024_02_14_01_15_UTC
```

The messages look like this:

```bash
head -n 5 ais_2024_02_14_01_15_UTC
```

```
\s:2573555,c:1707873398*05\!BSVDM,1,1,,A,13mB7d001E1dDH0`KlKBjB?<0490,0*34
\s:2573105,c:1707873398*04\!B2VDM,2,1,4,B,53oHa402;QilhU1D00085=@v0TT000000000000t6`p>440Ht:j3lU1CcCCl,0*7A
\s:2573105,c:1707873398*04\!B2VDM,2,2,4,B,lh000000000,2*5F
\s:2573415,c:1707873398*00\!BSVDM,1,1,,B,13m9ERPP00Pp5opUivG8P?w:0<12,0*6F
\s:2573210,c:1707873398*03\!BSVDM,1,1,,B,13oQg60000PJMu`QhUB6D26v0D18,0*40
```

## Inspecting format of messages with xxd

We use [xxd](https://linux.die.net/man/1/xxd) and [sed](https://www.gnu.org/software/sed/manual/sed.html).

```bash
ncat 153.44.253.27 5631 | xxd -p -c 1024 | sed 's/0d0a/&\n/g'
```

1. Uses `ncat` to receive the data.
2. Pipes the data into `xxd -p -c 1024` to convert it into a plain hex dump. The `-c 1024` sets the number of columns to 1024, ensuring that each message up to 1024 bytes is processed as a single unit.
3. Pipes to `sed 's/0d0a/&\n\n/g'` to search for `0d0a` (the hex representation of `\r\n`) in the hex output and replace it with itself followed by a newline.

```bash
ncat 153.44.253.27 5631 | xxd -p -c 1024 | sed 's/0d0a/&\n/g'
```

```
5c733a323537333438352c633a313730383138383437352a30465c21425356444d2c312c312c2c412c42315777743f5030354843733d3939756565416b57777634335030362c302a36460d0a
21425356444d2c312c312c2c422c31336d486751303030305049554f6a5264305250493251563044303b2c302a34330d0a
5c733a323537333333352c633a313730383138383437352a30335c21425356444d2c312c312c2c412c31336d3a6e6e3730303c30614256465477736d623d706d68303850652c302a34380d0a
5c733a323537333134352c633a313730383138383437352a30365c21425356444d2c312c312c2c412c31334d43306b4030303d303a446f4c504e44733864563f68303850652c302a33310d0a
5c733a323537333532352c633a313730383138383437352a30345c21425356444d2c312c312c2c412c48315777743f50606e3038754556333e32323232323167373e456c2c322a35360d0a
```

This is a fun roundabout way of seeing that the messages are terminated with `\r\n`. We could use `ncat -x` to directly output the hex dump, but I didn't figure out a way nicely to parse the separate messages.

## gpsdecode

[gpsdecode](https://gpsd.gitlab.io/gpsd/gpsdecode.html)

> **gpsdecode** tool is a batch-mode decoder for NMEA and various binary packet formats associated with GPS, AIS, and differential-correction services. It produces a JSON dump on standard output from binary on standard input. The JSON is the same format documented by **gpsd**; this tool uses the same decoding logic as **gpsd**, but with a simpler interface intended for batch processing of data files.
>
> All sensor-input formats known to the GPSD project can be decoded by this tool. These include: NMEA, AIVDM (the NMEA-derived sentence format used by AIS, the marine Automatic Identification System), RTCM2, and all supported GPS binary formats (notably including SiRF). See **gpsd**(8) for applicable standards and known limitations of the decoding logic.
>
> You can use this tool with **nc**(1) to examine AIS feeds from AIS pooling services, RTCM feeds from RTCM receivers or NTRIP broadcasters.

The messages parsed using `gpsdecode` look like this:

```bash
head -n 5 ais_2024_02_14_01_15_UTC | gpsdecode
```

```json
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257198000,"scaled":true,"status":0,"status_text":"Under way using engine","turn":0,"speed":8.5,"accuracy":false,"lon":23.662507,"lat":70.664715,"course":71.3,"heading":71,"second":38,"maneuver":0,"raim":false,"radio":16960}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257054090,"scaled":true,"status":0,"status_text":"Under way using engine","turn":"nan","speed":0.0,"accuracy":true,"lon":12.253433,"lat":66.026820,"course":217.6,"heading":511,"second":37,"maneuver":0,"raim":false,"radio":49218}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":259551000,"scaled":true,"status":0,"status_text":"Under way using engine","turn":0,"speed":0.0,"accuracy":true,"lon":5.782060,"lat":58.998307,"course":161.6,"heading":67,"second":31,"maneuver":0,"raim":false,"radio":81992}
```

## jq

[jq](https://devdocs.io/jq/)

```bash
cat ais_2024_02_14_01_15_UTC_decoded_with_timestamps| jq --slurp '.' > ais_2024_02_14_01_15_UTC_decoded_with_timestamps.json
```

```bash
jq '.[0]' ais_2024_02_14_01_15_UTC_decoded_with_timestamps.json
```

```json
{
  "class": "AIS",
  "device": "stdin",
  "type": 1,
  "repeat": 0,
  "mmsi": 257198000,
  "scaled": true,
  "status": 0,
  "status_text": "Under way using engine",
  "turn": 0,
  "speed": 8.5,
  "accuracy": false,
  "lon": 23.662507,
  "lat": 70.664715,
  "course": 71.3,
  "heading": 71,
  "second": 38,
  "maneuver": 0,
  "raim": false,
  "radio": 16960,
  "timestamp": ""
}
```

## Longitude and latitude

```bash
head -n 5 ais_2024_02_14_01_15_UTC |
gpsdecode |
jq -r '. | select(.lon != null and .lat != null) | "\(.lon) \(.lat)"'
```

```
23.662507 70.664715
12.253433 66.02682
5.78206 58.998307
```

```bash
cat ais_2024_02_14_01_15_UTC |
gpsdecode |
jq -r '. | select(.lon != null and .lat != null) | "\(.lon) \(.lat)"' > \
ais_2024_02_14_01_15_UTC.dat
```

It is worth noting that `|` does not require `\` to split across lines while `>` does.

## Bounding box with awk

```bash
awk '$1 >= 0 && $1 <= 35 && $2 >= 55 && $2 <= 80' ais_2024_02_14_01_15_UTC.dat > ais_2024_02_14_01_15_UTC_box.dat
```

```bash
wc -l ais_2024_02_14_01_15_UTC.dat
72999 ais_2024_02_14_01_15_UTC.dat
wc -l ais_2024_02_14_01_15_UTC_box.dat
72585 ais_2024_02_14_01_15_UTC_box.dat
```

## Gnuplot

```
gnuplot> plot 'ais_2024_02_14_01_15_UTC_box.dat' using 1:2 with points
```

![Longitude is between 0 and 35 and latitude is between 55 and 80](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wfwfssxvjaad229ama56.png)

## GDAL ogr2ogr

```bash
echo "Longitude,Latitude" > ais_2024_02_14_01_15_UTC_box.csv

awk '{print $1 "," $2}' ais_2024_02_14_01_15_UTC_box.dat >> ais_2024_02_14_01_15_UTC_box.csv
```

```bash
head -n 5 ais_2024_02_14_01_15_UTC_box.csv
```

```csv
Longitude,Latitude
23.662507,70.664715
12.253433,66.02682
5.78206,58.998307
2.966662,62.485987
```

[ogr2ogr](https://gdal.org/programs/ogr2ogr.html)

```bash
ogr2ogr -f "CSV" -lco GEOMETRY=AS_XY -t_srs "EPSG:32633" -s_srs "EPSG:4326" projected_output/ ais_2024_02_14_01_15_UTC_box.csv -oo X_POSSIBLE_NAMES=Longitude -oo Y_POSSIBLE_NAMES=Latitude -overwrite
```

```bash
head -n 5 projected_output/ais_2024_02_14_01_15_UTC_box.csv
X,Y,Longitude,Latitude
```

```csv
819151.812467396,7862821.57947704,23.662507,70.664715
375505.625008375,7325626.69347061,12.253433,66.02682
-28538.7387103818,6576427.56565985,5.78206,58.998307
-117577.293012299,6986142.22730701,2.966662,62.485987
```

`plot_ais.gp`

```
set terminal pngcairo enhanced
set output 'ais_plot.png'
set title 'Ship Positions in Norwegian Waters'
set xlabel 'UTM Easting (kilometers)'
set ylabel 'UTM Northing (kilometers)'
set grid
set datafile separator ','
plot 'projected_output/ais_2024_02_14_01_15_UTC_box.csv' using ($1/1000):($2/1000) with points pt 7 ps 1 lc rgb 'blue' title 'Ships'
```

```bash
gnuplot plot_ais.gp
```

![Ship locations using the UTM 33N projection system, with distances measured in kilometers](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c8fdzt178s85xwsaq7li.png)

## Including ISO 8601 timestamps

https://www.w3.org/TR/NOTE-datetime

ISO 8601 UTC timestamp format YYYY-MM-DDThh:mm:ssZ

To pipe the output of `ncat`, we do the following:

```bash
ncat 153.44.253.27 5631 | tr -d '\r'
```

Now we can compose this pipe with `gpsdecode`:

```bash
ncat 153.44.253.27 5631 | tr -d '\r' | gpsdecode
```

```json
{"class":"AIS","device":"stdin","type":3,"repeat":0,"mmsi":257878000,"scaled":true,"status":0,"status_text":"Under way using engine","turn":"fastright","speed":6.0,"accuracy":false,"lon":9.639210,"lat":59.107997,"course":278.4,"heading":284,"second":30,"maneuver":0,"raim":false,"radio":7792}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257021970,"scaled":true,"status":7,"status_text":"Engaged in fishing","turn":"nan","speed":0.0,"accuracy":false,"lon":17.902577,"lat":69.509332,"course":360.0,"heading":511,"second":32,"maneuver":0,"raim":false,"radio":27224}
{"class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":235103057,"scaled":true,"status":0,"status_text":"Under way using engine","turn":0,"speed":11.3,"accuracy":false,"lon":5.716548,"lat":63.704728,"course":206.7,"heading":204,"second":30,"maneuver":0,"raim":false,"radio":27224}
```

Now we compose with `awk` and `date` (https://www.gnu.org/software/coreutils/manual/html_node/Examples-of-date.html):

```bash
ncat 153.44.253.27 5631 | tr -d '\r' | gpsdecode | awk -v date="$(date -u +%Y-%m-%dT%H:%M:%SZ)" '{print "{\"timestamp_ISO_8601\":\"" date "\"," substr($0, 2)}'
```

```json
{"timestamp_ISO_8601":"2024-02-16T21:38:38Z","class":"AIS","device":"stdin","type":18,"repeat":0,"mmsi":257129980,"scaled":true,"reserved":0,"speed":0.0,"accuracy":true,"lon":23.329870,"lat":69.977463,"course":360.0,"heading":511,"second":39,"regional":0,"cs":false,"display":false,"dsc":true,"band":true,"msg22":true,"raim":true,"radio":609914}
{"timestamp_ISO_8601":"2024-02-16T21:38:38Z","class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257482600,"scaled":true,"status":0,"status_text":"Under way using engine","turn":"nan","speed":0.1,"accuracy":false,"lon":5.209840,"lat":59.298867,"course":27.4,"heading":511,"second":39,"maneuver":0,"raim":false,"radio":49221}
{"timestamp_ISO_8601":"2024-02-16T21:38:38Z","class":"AIS","device":"stdin","type":5,"repeat":0,"mmsi":258033200,"scaled":true,"imo":0,"ais_version":1,"callsign":"JWCH","shipname":"NORDNES","shiptype":0,"shiptype_text":"Not available","to_bow":9,"to_stern":6,"to_port":2,"to_starboard":2,"epfd":1,"epfd_text":"GPS","eta":"02-14T20:00Z","draught":0.9,"destination":"NAVIGATION TRAINING","dte":0}
```

```bash
timeout 3600s ncat 153.44.253.27 5631 | tr -d '\r' | gpsdecode | awk -v date="$(date -u +%Y-%m-%dT%H:%M:%SZ)" '{print "{\"timestamp_ISO_8601\":\"" date "\"," substr($0, 2)}' > ais_2024_02_16T21_42_57Z
```

We examine the contents of the file we created above:

```bash
head -n 2 ais_2024_02_16T21_42_57Z
```

```json
{"timestamp_ISO_8601":"2024-02-16T21:43:35Z","class":"AIS","device":"stdin","type":1,"repeat":0,"mmsi":257827700,"scaled":true,"status":15,"status_text":"Not defined","turn":0,"speed":0.0,"accuracy":false,"lon":14.376283,"lat":67.284670,"course":342.6,"heading":48,"second":35,"maneuver":0,"raim":false,"radio":81943}
{"timestamp_ISO_8601":"2024-02-16T21:43:35Z","class":"AIS","device":"stdin","type":21,"repeat":1,"mmsi":992576156,"scaled":true,"aid_type":30,"aid_type_text":"Special Mark","name":"AQUACULTURE 4","lon":18.480670,"lat":69.895130,"accuracy":false,"to_bow":1,"to_stern":1,"to_port":1,"to_starboard":1,"epfd":7,"epfd_text":"Surveyed","second":60,"regional":0,"off_position":false,"raim":false,"virtual_aid":true}
```

Because the interrupt from `timeout 3600s` did not coincide with a complete message from `ncat`, the tail of the file looks like:

```bash
tail -n 1 ais_2024_02_16T21_42_57Z
```

```bash
{"timestamp_ISO_8601":"2024-02-16T21:43:35Z","class":"AIS","devi
```

Thus we omit the last line of the file, and then parse as JSON using `jq --slurp`:

```bash
head -n -1 ais_2024_02_16T21_42_57Z | jq --slurp '.' > ais_2024_02_16T21_42_57Z.json
```

We inspect the first element:

```bash
jq '.[0]' ais_2024_02_16T21_42_57Z.json
```

```json
{
  "timestamp_ISO_8601": "2024-02-16T21:43:35Z",
  "class": "AIS",
  "device": "stdin",
  "type": 1,
  "repeat": 0,
  "mmsi": 257827700,
  "scaled": true,
  "status": 15,
  "status_text": "Not defined",
  "turn": 0,
  "speed": 0,
  "accuracy": false,
  "lon": 14.376283,
  "lat": 67.28467,
  "course": 342.6,
  "heading": 48,
  "second": 35,
  "maneuver": 0,
  "raim": false,
  "radio": 81943
}
```

Now we use `jq` to convert this to the GeoJSON format ([RFC 7946](https://datatracker.ietf.org/doc/html/rfc7946)).

```bash
jq '{type: "FeatureCollection", features: map({type: "Feature", geometry: {type: "Point", coordinates: [.lon, .lat]}, properties: .})}' ais_2024_02_16T21_42_57Z.json > ais_2024_02_16T21_42_57Z.geojson
```

[ais_2024_02_16T21_42_57Z.geojson on ArcGIS Online](https://arcg.is/5q8v02)

[Map view on ArcGIS Online](https://arcg.is/1ur5nD)

[ais_2024_02_16T21_42_57Z.geojson on GitHub](https://github.com/jordanbell2357/how-to/blob/main/ais_2024_02_16T21_42_57Z.geojson)

## Ais Decoder by Neal Arundale

[Ais Decoder by Neal Arundale](https://arundaleais.github.io/docs/ais/ais_decoder.html)

Ais Decoder Options:

![Ais Decoder Options](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z56us65rgqfnovurc779.png)

Ais Decoder main menu:

![Ais Decoder main menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q33q19b8tbo2ot9yp4nj.png)

Ais Decoder Nmea input:

![Ais Decoder Nmea input](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kfacbnyz9lh62b636i73.png)

Ais Decoder parsed Summary:

![Ais Decoder parsed Summary](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vu8v2dskk2kwf2o372k7.png)

Ais Decoder NMEA sentence Detail:

![Ais Decoder NMEA sentence Detail](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/47f7ewj2ns9d63kqhbsc.png)

Compare to gpsdecode:

```bash
echo '!BSVDM,1,1,,B,13mL5v0001PEjC6Rsu:Pca`L0@9W,0*59,2/14/2024 6:36:17 PM' | gpsdecode | jq --slurp '.' | jq '.[0]'
```

```json
{
  "class": "AIS",
  "device": "stdin",
  "type": 1,
  "repeat": 0,
  "mmsi": 257361400,
  "scaled": true,
  "status": 0,
  "status_text": "Under way using engine",
  "turn": 0,
  "speed": 0.1,
  "accuracy": true,
  "lon": 4.759205,
  "lat": 61.056497,
  "course": 17.4,
  "heading": 308,
  "second": 14,
  "maneuver": 0,
  "raim": false,
  "radio": 66151
}
```