harness-race/pi-r3

DETR (ResNet-50) fine-tuned on Beyond Words (loc_beyond_words)

Object detection model detecting 7 types of visual content on World War I-era historical newspaper pages (Library of Congress, Chronicling America): photographs, illustrations, maps, comics/cartoons, editorial cartoons, headlines, and advertisements.

• Base model: facebook/detr-resnet-50 (Apache-2.0)
• Dataset: biglam/loc_beyond_words (CC0-1.0), 2,846 train / 712 validation images, COCO format
• Task: object detection (bounding boxes + class labels)

How to use

from transformers import AutoImageProcessor, DetrForObjectDetection
from PIL import Image
import requests

url = "https://tile.loc.gov/storage-services/service/amed/amedscd/2018603128/001.jpg"  # example page
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("harness-race/pi-r3")
model = DetrForObjectDetection.from_pretrained("harness-race/pi-r3")

inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

target_sizes = [(image.height, image.width)]
results = processor.post_process_object_detection(outputs, threshold=0.5,
                                                  target_sizes=target_sizes)[0]
for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
    print(score, model.config.id2label[label.item()], [round(x, 2) for x in box.tolist()])

Or with the pipeline:

from transformers import pipeline
detector = pipeline("object-detection", model="harness-race/pi-r3")
results = detector(image, threshold=0.5)

Training details

Fine-tuned from the COCO-pretrained checkpoint with the HF Transforms API (aspect-preserving resize to shortest edge 480px, max 768px, padding with pixel masks, random horizontal flip augmentation).

Hyperparameter	Value
Epochs (best)	13 of 14 requested
Batch size	4
Optimizer	AdamW
Learning rate (main / backbone)	5e-05 / 5e-06
Weight decay	0.0001
Warmup / schedule	2% linear warmup + cosine decay
Gradient clipping	10.0
Early stopping patience	3 (on validation loss)
Max input size	768px
Total training time	57 min

Class distribution in the training set is skewed (Headline ~58%, Advertisement ~28%, followed by Photograph / Comics / Illustration / Editorial Cartoon / Map).

Validation results (hold-out split, COCO protocol)

Metric	Value
mAP (IoU 0.5:0.95)	0.1130
mAP@0.5	0.2466
mAP@0.75	0.0954
mAR (max 100 dets)	0.2892

Per-class mAP (IoU 0.5:0.95, area=all, maxDets=100):

Class	mAP
Photograph	0.2316
Illustration	0.0531
Map	0.0189
Comics/Cartoon	0.1403
Editorial Cartoon	0.1589
Headline	0.0650
Advertisement	0.1236

Training/validation loss curves and full eval details are in training_log.json in this repository.

Limitations

• Trained on WWI-era (mostly 1910s) American newspaper scans; may transfer less well to other layouts, eras, or languages.
• Classes are heavily imbalanced (headlines and advertisements dominate).
• Input resolution is limited to 768px on the longest edge; re-running inference at higher resolution may improve small-object recall.
• Model weights are the DETR Apache-2.0-licensed architecture; dataset is CC0.