models for IndyXe

Lightweight LightGBM classifiers used by the action_recognition ROS 2 package of the IndyXe robot system. Each model takes per-person features produced by the upstream pose / face estimation (recognition_msgs/Persons) and outputs a binary label for every person in a frame.

File	Task	Input features	Output
`hand_waving_model.joblib` / `.txt`	Is the person waving a hand?	255 = 5 frames × 17 keypoints × (x, y, z)	0: not waving, 1: waving
`looking_at_model.joblib` / `.txt`	Is the person looking toward the camera (robot)?	7 = face bbox [x, y, w, h] + face orientation [yaw, pitch, roll]	0: not looking, 1: looking
`wheelchair_model.joblib` / `.txt`	Is the person seated (e.g. in a wheelchair / on a chair)?	51 = 17 keypoints × (x, y, z)	0: not seated, 1: seated

*.joblib: pickled lightgbm.LGBMClassifier (scikit-learn API). Use with predict() / predict_proba().
*.txt: the same model in LightGBM's native text format (lightgbm.Booster(model_file=...)). It does not depend on pickle or on scikit-learn versions and gives identical probabilities.

Feature details

Pose keypoints (`hand_waving`, `wheelchair`)

17 3D keypoints in the sensor (camera) coordinate frame, taken from Person.pose_sensor.keypoints (flattened [x0, y0, z0, x1, y1, z1, ...], COCO-style keypoint order).
Normalization: keypoints are centered by the mean of the valid keypoints (valid = z > 0 and finite). Invalid keypoints are set to 0 after centering.

import numpy as np

def normalize_keypoints(data):
    p = np.asarray(data, dtype=float).reshape(-1, 3)
    valid = (p[:, 2] > 0) & np.all(np.isfinite(p), axis=1)
    norm = np.zeros_like(p)
    if np.any(valid):
        norm[valid] = p[valid] - p[valid].mean(axis=0)
    return norm.reshape(-1)

wheelchair: one normalized frame (51 dims).
hand_waving: the last 5 normalized frames of the same tracked person, concatenated oldest first (5 × 51 = 255 dims).

Face (`looking_at`)

Person.face_location (image coordinates [x, y, w, h]) concatenated with Person.face_orientation ([yaw, pitch, roll]).

Usage

import joblib
from huggingface_hub import hf_hub_download

path = hf_hub_download("<user>/models-for-IndyXe", "wheelchair_model.joblib")
model = joblib.load(path)
labels = model.predict(features)  # features: (n_persons, 51)

Native format (no pickle):

import lightgbm as lgb
from huggingface_hub import hf_hub_download

booster = lgb.Booster(model_file=hf_hub_download("<user>/models-for-IndyXe", "wheelchair_model.txt"))
prob = booster.predict(features)  # probability of class 1 (looking_at: (n, 2) array)

In the IndyXe action_recognition package the nodes download these files automatically when no local model file is found.

Training

lightgbm.LGBMClassifier with n_estimators=10000, learning_rate=0.1, num_leaves=31, random_state=0; the number of boosting rounds was limited by early stopping on a validation split (resulting trees: hand_waving 179, looking_at 67 iterations × 2 classes, wheelchair 174).
looking_at was trained with the multiclass objective (num_class=2); the others with binary.
Data: ROS 2 bags recorded with the IndyXe perception pipeline, one bag per class. The training data is not published.
Trained with scikit-learn 1.5.2 / LightGBM 4.x. Loading the .joblib files with other scikit-learn versions may emit InconsistentVersionWarning; use the .txt files if this is a concern.

Limitations

The models depend on the specific keypoint layout, coordinate frame and camera setup of the IndyXe perception pipeline; features from other pose estimators must match the layout above.
looking_at uses image-space face coordinates, so it depends on the camera resolution and mounting used for training.
They are small task-specific classifiers and were not evaluated for use outside the IndyXe system.

Security note

.joblib files are Python pickles. Only load them from a source you trust; the .txt files are plain-text LightGBM models and do not execute code when loaded.

Downloads last month: -; Downloads are not tracked for this model. How to track