kein5/vrct-vox-chatbox-detector

VRCT-VOX chatbox detector (YOLOX-Tiny)

Finds VRChat chatboxes (the speech bubbles above avatars) in a screenshot, so VRCT-VOX can read only the chat text and skip nameplates, signs and menus. VRCT-VOX downloads this file only after the user agrees, and checks its SHA-256 before loading it.

This is not the chat-bubble model from upstream VRCT. It was trained separately, without that model's weights or outputs.

File

File	Size	SHA-256
vrct_vox_chatbox_yolox_tiny.onnx	20,184,371 bytes	160c4a505bee87d101564c569dff7f0bacd416b9006ec5aa70f8c9d221470ffd

• Model: YOLOX-Tiny (depth 0.33, width 0.375), one class: chatbox.
• Input images: float32 [1, 3, H, W], BGR, 0–255 without normalization. H and W are dynamic and must be multiples of 32. Letterbox the frame so the long side is 1280, pad at the bottom and right with 114.
• Output output: [1, anchors, 6] = cx, cy, w, h, objectness, class score, in input pixels. Score = objectness × class score. NMS is not in the graph. VRCT-VOX keeps scores of 0.7 or higher and runs NMS at IoU 0.65.
• ONNX opset 17, float32.

Training

• Base weights: yolox_tiny.pth (COCO) by Megvii, YOLOX, Apache-2.0. Framework: YOLOX at commit 6ddff48.
• Data: 2,209 frames captured with the maintainer's own two VRChat accounts and no other people present: in a local test instance of a test world built for this purpose, and in an invite-only instance of the same world (uploaded as a private world) on VRChat's servers, where public avatars by other creators load too. The world has plain rooms and zones modelled on common VRChat worlds (Japanese room, night rooftop, club, cafe, mirror lounge, park, beach, sci-fi lobby, classroom, avatar gallery) with signs, menus and dark info panels as hard negatives, and per-zone post-processing. Chat text was sent over OSC, and the speaker switched to a random avatar every few messages (101 avatars in total; one capture favoured one-word messages, distant chatboxes and chatboxes at the edge of the view). Boxes were labeled automatically: PP-OCR text lines that match the sent message mark the chatbox, and where an earlier version of this detector boxed that text, its box becomes the label, so labels follow the panel rather than padded text bounds. The images are not published, because the avatars in them belong to their creators.
• Split: 1,880 frames for training (1,496 with a chatbox) and 329 for validation (263 with a chatbox). Earlier validation frames stay in validation. Validation takes whole one-minute blocks, so near-identical frames do not appear in both.
• Settings: 1280 input, 80 epochs, mosaic 0.3, no mixup, horizontal flip, HSV jitter. The training tools are in tools/chatbox_lab of the VRCT-VOX repository. They are adapted from upstream VRCT's training scripts (MIT).

Evaluation

COCO AP on the validation split: 0.941 (IoU 0.50:0.95), against labels that follow the chatbox panel; earlier versions were scored against padded text bounds, so the APs do not compare.

A detector-independent check, with VRCT-VOX's settings (score 0.7): a chatbox counts as found when a detection covers at least 80% of the PP-OCR text of the sent message. None of these frames were used to train any of the versions:

Frames	Chatboxes	This version	d3f89ea1	475e92e0
VRChat servers, validation part of two captures	108	107	105	104
Separate 10-minute capture in the local test instance	109	108	107	106
Capture of short messages and distant chatboxes, validation part	32	32	32	n/a

No version detected anything on the 65 frames without a chatbox. The check counts only chatboxes PP-OCR can read, so the smallest and most washed-out chatboxes are not in it; those remain the main source of misses.

Limitations

All frames come from one test world (eighteen zones), two clients, and the default chatbox style. Worlds with unusual light, post-processing, very distant chatboxes, or mods that change the chatbox may give worse results. Report problems in the VRCT-VOX repository.

License

MIT. See LICENSE. The base weights are from Megvii's YOLOX under Apache-2.0.