voxOCR: tiny person detector for drone search and rescue

A CenterNet-style, anchor-free person detector for downward-looking drone cameras: 136,547 parameters, 197 KB full-int8 TFLite, built for ESP32-S3-class microcontrollers. It marks where people are in a 160×160 RGB tile so a drone can flag places worth a closer look for a rescue team.

Türkçe özet: Arama kurtarma dronları için, mikrodenetleyicide çalışacak kadar küçük (136 bin parametre, int8 dosyası 197 KB) bir insan tespit modeli. HERIDAL test setinde int8 hâli, eşik 0,2'de insanların %82,5'ini buluyor. CC BY 4.0.

Code and training details: ErdemWilkinson/voxOCR.

Files

File	What it is
voxocr_qat_int8.tflite	Recommended. Quantization-aware trained, full int8, input [1,160,160,3] int8, output [1,40,40,3] int8
voxocr_int8.tflite	Post-training quantized int8 of the float model
voxocr.keras	Float model
evaluation.json	Recall / precision / false alarms per threshold, val and test

Input and output

• Input: 160×160 RGB, pixel / 127.5 - 1. A 640×640 crop of a 4000×3000 aerial photo, resized to 160, matches the test setting.
• Output: 40×40 grid (stride 4).
  • Channel 0: person-centre heatmap (0 to 1).
  • Channels 1 and 2: log box width and height in grid cells.
• Detections: local maxima of channel 0 above a threshold. Lower thresholds find more people and give more false alarms.

import numpy as np
import tensorflow as tf
from PIL import Image

interpreter = tf.lite.Interpreter(model_path="voxocr_qat_int8.tflite")
interpreter.allocate_tensors()
inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0]
(si, zi), (so, zo) = inp["quantization"], out["quantization"]

tile = np.asarray(Image.open("tile.png").convert("RGB").resize((160, 160)), np.float32) / 127.5 - 1
interpreter.set_tensor(inp["index"], np.clip(np.round(tile / si + zi), -128, 127).astype(np.int8)[None])
interpreter.invoke()
grid = (interpreter.get_tensor(out["index"])[0].astype(np.float32) - zo) * so

heat = grid[..., 0]
peaks = heat == tf.nn.max_pool2d(heat[None, :, :, None], 3, 1, "SAME")[0, :, :, 0].numpy()
for y, x in zip(*np.nonzero(peaks & (heat >= 0.2))):
    w, h = np.exp(grid[y, x, 1:]) * 4
    print(f"person at ({(x + 0.5) * 4:.0f}, {(y + 0.5) * 4:.0f}) px, about {w:.0f}x{h:.0f}, score {heat[y, x]:.2f}")

Results (HERIDAL test set, 3,507 tiles, 372 people)

A detection is a find when its centre is inside a person's box grown by one grid cell.

Model, threshold	Recall	Precision	False alarms per tile
int8 QAT, 0.45 (best F1)	59.7%	52.5%	0.06
int8 QAT, 0.30	74.5%	29.8%	0.19
int8 QAT, 0.20	82.5%	15.8%	0.47
int8 QAT, 0.10	86.6%	5.5%	1.58
float, 0.45 (best F1)	57.0%	55.1%	0.05

The int8 file loses nothing measurable against the float model. v1 (an under-trained first run, recall 28.8% at best F1) is shown in the chart for comparison.

Limitations

• Trained and tested only on HERIDAL: summer, daytime, mostly Mediterranean terrain. Snow, forest canopy, water, night and thermal are untested.
• 90% recall is not reached on the test set.
• Not yet measured on an ESP32-S3 (speed, memory) or on real flight video.

Data and license

Trained on HERIDAL (CC BY 3.0): D. Božić-Štulić, Ž. Marušić, S. Gotovac, "Deep Learning Approach in Aerial Imagery for Supporting Land Search and Rescue Missions", IJCV 2019.

Weights: CC BY 4.0. Training code: MIT.

Intended use

Search and rescue: helping find people who are lost or injured. It is not intended for targeting or harming people.