You need to agree to share your contact information to access this model

Terms of Use

This model is shared by the Institutional Data Initiative for research and public-interest use (the “Service”). These terms are intended to support experimentation while encouraging collaboration and feedback as we refine the model and work with contributing institutions to define shared, long-term norms for open data reuse. To share questions or feedback, contact us at contact@institutional.org.

By accessing or downloading the model or otherwise using the Service, you agree to the following:

Noncommercial Use Only
You may use the Service solely for noncommercial purposes. Open-source projects and other public-use efforts are welcome, even if they may indirectly support commercial use, so long as they are unaffiliated with commercial actors or intent.

If you are affiliated with a commercial organization or plan to use the Service for commercial purposes (including AI model training), you will contact us first at contact@institutional.org.
No Redistribution
You may not share or redistribute the Service or any of the data provided through the Service, in whole or in part, including through public repositories or aggregators. If you want others to access it, please direct them to the attribution link.
Derivative Works
You may create derivative works for noncommercial use, but you may not make available any such derivative works that substantially reproduce the original model. Only outputs that are significantly transformed and cannot substitute for the original—such as evaluations, summary statistics, or visualizations—may be shared, with attribution.
Attribution
If you use the model in public-facing work, you must include attribution substantially similar to:

Institutional Books provided by the Institutional Data Initiative with source material contributed by Harvard Library, available at https://institutional.org.

Minor modifications to fit citation style or formatting are permitted, provided the essential elements remain intact.
Provisional Terms
These terms apply only to this release and may change. We are actively working with contributing institutions to develop a long-term framework for responsible, open data sharing.
DISCLAIMER OF WARRANTY
TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, ACCESS TO “SERVICE IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND (EXPRESS, IMPLIED, OR OTHERWISE), INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. WE DO NOT WARRANT THAT THE SERVICE WILL OPERATE IN AN UNINTERRUPTED OR ERROR-FREE MANNER OR THAT THE SERVICE IS FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS. WITHOUT LIMITING THE FOREGOING, WE DO NOT WARRANT THAT (A) THE SERVICE WILL MEET YOUR REQUIREMENTS OR EXPECTATIONS OR ACHIEVE THE INTENDED PURPOSES; (B) THE SERVICE WILL NOT EXPERIENCE OUTAGES OR OTHERWISE WILL BE UNINTERRUPTED, TIMELY, OR SECURE; (C) THE INFORMATION OR SERVICES OBTAINED THROUGH OR FROM THE SERVICE WILL BE ACCURATE, COMPLETE, CURRENT, ERROR-FREE, OR RELIABLE; (D) ANY DEFECTS IN OR ON THE SERVICE WILL BE CORRECTED; OR (E) THAT ANY POLLS OR OTHER SOLICITATIONS OF INFORMATION POSTED THROUGH THE SERVICE BY YOU OR OTHER USERS ARE SAFE OR APPROPRIATE FOR YOUR OR OTHER USERS’ PARTICIPATION. WE MAKE NO REPRESENTATION OR WARRANTY REGARDING YOUR ABILITY TO TRANSMIT AND RECEIVE INFORMATION FROM OR THROUGH THE SERVICE, AND YOU AGREE AND ACKNOWLEDGE THAT YOUR ABILITY TO ACCESS THE SERVICE MAY BE IMPAIRED.
LIMITATION OF LIABILITY
EXCEPT INSOFAR AS THE FOLLOWING LIMITATION MAY BE PROHIBITED BY APPLICABLE LAW, WE SHALL NOT BE LIABLE TO YOU OR TO ANY THIRD PARTY FOR ANY DIRECT, CONSEQUENTIAL, INDIRECT, PUNITIVE, SPECIAL, OR INCIDENTAL DAMAGES, WHETHER FORESEEABLE OR UNFORESEEABLE (INCLUDING, BUT NOT LIMITED TO, LOSS OF PROFITS OR EARNING POWER, LOSS OF DATA, LOSSES DUE TO ERRORS OR INTERRUPTION IN AVAILABILITY OF THE SERVICE, UNAVAILABILITY OF ANY SERVICE, SERVER, OR COMMUNICATIONS FACILITY, OR DAMAGES DUE TO ACTS OR OMISSIONS OF OTHERS USING THE SERVICE), ARISING OUT OF OR RELATING TO THE SERVICE, INCLUDING WITHOUT LIMITATION YOUR AND OTHERS’ USE OF OR INABILITY TO USE THE SERVICE, OR YOUR RELIANCE UPON INFORMATION OBTAINED FROM OR THROUGH THE SERVICE, WHETHER BASED IN CONTRACT, TORT, STATUTORY, OR OTHER LAW. OUR TOTAL CUMULATIVE LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE SERVICE (INCLUDING, WITHOUT LIMITATION, IN THE WAYS DESCRIBED IN THE PRECEDING SENTENCE), WHETHER BASED IN CONTRACT, TORT, STATUTORY, OR OTHER LAW, WILL NOT EXCEED THE AMOUNT, IF ANY, THAT YOU PAID US TO USE THE SERVICE IN THE TWELVE MONTHS PRECEDING THE CLAIM. THE DISCLAIMERS AND LIMITATIONS SET FORTH IN THIS SECTION SHALL APPLY, TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, WHETHER OR NOT WE HAVE BEEN NEGLIGENT OR OTHERWISE AT FAULT.
YOU ACKNOWLEDGE THAT, FOR PURPOSES OF THE FOREGOING DISCLAIMERS AND LIMITATIONS, AS WELL AS THE INDEMNITY PROVISION IN SECTION 8 BELOW, THE TERMS “WE,” “OUR,” “US,” “INSTITUTIONAL DATA INITIATIVE,” AND “IDI” INCLUDE THE CORPORATE BODY PRESIDENT AND FELLOWS OF HARVARD COLLEGE, ALSO KNOWN AS HARVARD UNIVERSITY, AND ITS VARIOUS SCHOOLS, THE MEMBERS OF ITS GOVERNING BOARDS, AND ITS OFFICERS, FACULTY MEMBERS, EMPLOYEES, FELLOWS, AND TO THE EXTENT WORKING ON IDI, ITS STUDENTS, CONTRACTORS, AND REPRESENTATIVES.
Indemnification
You agree to indemnify us and hold us harmless from any and all claims, liabilities, damages, losses and expenses, including reasonable attorneys’ fees and costs, relating to or arising out of (a) your use or attempted use of the Service in violation of these Terms of Service; or (b) your violation of any law or rights of any third party in connection with your use of the Service.
Governing Law/ Jurisdiction
You agree that the Terms of Service and any claim or dispute arising out of or relating to the Service or Terms of Service will be governed by the laws of the Commonwealth of Massachusetts, excluding its conflicts of laws principles. You agree that all such claims and disputes will be heard and resolved exclusively in the federal or state courts located in and serving Middlesex or Suffolk County, Massachusetts, U.S.A. You consent to the personal jurisdiction of those courts over you for this purpose, and you waive and agree not to assert any objection to such proceedings in those courts (including any defense or objection of lack of proper jurisdiction or inconvenience of forum).
Whole Agreement/ Amendment
These Terms of Service constitute the entire agreement between you and Harvard with respect to your use of the Service. We reserve the right to amend these Terms of Service at any time. The Service will post notice of changes to the terms on this webpage, and by accessing the Service after modifications to these Terms of Service have been posted, you agree to be bound by all the modified terms. Accordingly, you should periodically revisit this page to review the then-current Terms of Service.

📚 Institutional Books — Visual Elements Classification Model (YOLOv26s-cls)

A multi-class image classification model that categorizes detected visual elements from digitized book page scans into five semantic classes. This model is the classification component of the Institutional Books Visual Elements Pipeline, operating on cropped regions produced by the upstream detection model.

More information:

📄 Full analysis available in our technical report
🌐 Website

See also:

The Institutional Data Initiative at Harvard Law School Library works with knowledge institutions—from libraries and museums to cultural groups and government agencies—to refine and publish their collections as data. Reach out to collaborate on your collections.

Model Description

Architecture: YOLOv26s-cls (small classification variant)
Parameters: 2.8 million
Classes: 5
Framework: Ultralytics

Classes

Class	Description
Image or Illustration	Photographs, drawings, engravings, and pictorial content
Music	Musical notation and scores
Artifact	Scanning errors, book clips, finders, and other elements not part of the published work
Ex Libris or Decorative	Bookplates, ornamental borders, and decorative elements
Chart or Graph	Statistical visualizations, diagrams, and data representations

Classes are mutually exclusive.

Intended Use

This model classifies visual elements that have already been detected and cropped from digitized book pages. It is designed to work as part of a multi-stage pipeline.

Primary use cases:

Categorizing detected visual elements from large-scale digitized book collections
Enabling filtered access to specific content types (e.g., retrieving only music scores or only charts)
Supporting downstream captioning by identifying which elements warrant description

Out of scope:

General-purpose image classification
Classification of full pages (expects pre-cropped regions)
Fine-grained sub-classification within categories (e.g., distinguishing engravings from photographs)

Training

Dataset

The training set consists of 6,042 crops from the Institutional Books dataset:

1,602 annotated fully manually
4,440 pre-annotated using an intermediary model and manually reviewed/corrected

Crops are derived from bounding boxes in the detection training dataset — each crop is one specific element on a specific page.

Training Configuration

Parameter	Value
Image size (training)	480×480 px
Batch size	32 per GPU
Max epochs	275
Early stopping patience	100
Best checkpoint epoch	140
Mixed precision	Enabled
Deterministic training	Enabled
Hardware	4× NVIDIA L40S
Training time	~30 minutes

Data Augmentation

Augmentation	Setting
HSV hue shift	0.015
HSV saturation	0.7
HSV value	0.4
Translation	0.1
Scale	0.5
Horizontal flip	p=0.5
Mosaic	1.0 (full strength)
RandAugment	Enabled
Random erasing	p=0.4

Evaluation

Per-class metrics on the held-out test split:

Class	Precision	Recall	Test Samples
Image/Illustration	0.98	0.97	325
Artifact	0.98	0.99	51
Ex Libris/Decorative	0.96	0.96	47
Music	0.99	0.99	141
Chart/Graph	0.87	0.91	44

Chart/Graph is the most challenging class, due to the visual diversity of diagrams, maps, and statistical visualizations grouped under this label.

Inference Configuration

Parameter	Value	Rationale
Image size	640 px	Increased throughput without compromising accuracy
Batch size	16 crops	Balances GPU utilization and memory
Confidence threshold	0.0 (at inference)	Full softmax vector stored; thresholding applied post-hoc
Post-hoc reclassification threshold	0.70	Predictions below 0.70 are relabeled `Other` when building the final dataset

Usage

With Ultralytics

from ultralytics import YOLO

model = YOLO("path/to/best.pt")

results = model.predict("crop.jpg", imgsz=640)

for result in results:
    top1_class = result.names[result.probs.top1]
    top1_conf = float(result.probs.top1conf)
    all_probs = result.probs.data.tolist()
    print(f"Class: {top1_class}, Confidence: {top1_conf:.3f}")

With HuggingFace Hub

from huggingface_hub import hf_hub_download
from ultralytics import YOLO

model_path = hf_hub_download(
    repo_id="institutional/institutional-books-visual-elements-classification",
    filename="weights/best.pt",
)

model = YOLO(model_path)
results = model.predict("crop.jpg", imgsz=640)

Class Index Mapping

The model's class indices correspond to the following labels (alphabetical order):

Index	Class
0	Artifact
1	Chart/Graph
2	Ex Libris/Decorative
3	Image/Illustration
4	Music

Production Results

When deployed against 43 million crops from the Institutional Books collection:

Total crops classified: ~43 million
Processing time: 3 days 17 hours (4× L40S GPUs)
Average per batch (1,000 volumes): 5 min 19 sec
Throughput: 140 crops/second end-to-end (75 crops/s inference per GPU)

Class Distribution (before post-processing)

Class	Percentage
Image or Illustration	70.2%
Artifact	9.7%
Ex Libris or Decorative	8.0%
Chart or Graph	6.8%
Music	5.3%

Class Distribution (after confidence thresholding at 0.70)

Class	Percentage
Image or Illustration	68.0%
Artifact	9.2%
Ex Libris or Decorative	7.0%
Chart or Graph	6.2%
Music	5.3%
Other (reclassified)	4.3%

Confidence Statistics

Statistic	Value
Mean confidence	0.966
Median confidence	1.000
% predictions in 0.95–1.00	85.9%
% predictions below 0.50	0.3%

Per-class average confidence:

Class	Avg. Confidence
Music	0.994
Image/Illustration	0.974
Ex Libris/Decorative	0.904

Reclassification Rates (to "Other")

Class	% Reclassified
Ex Libris/Decorative	12.9%
Chart/Graph	9.1%

These rates highlight which categories are most ambiguous — decorative bookplates, printer's marks, and schematic figures are hardest to separate cleanly.

Model Files

File	Description
`weights/best.pt`	Best checkpoint (epoch 140) — use this for inference
`weights/last.pt`	Final training checkpoint
`weights/best.torchscript`	TorchScript export of best checkpoint

Limitations

Trained exclusively on public domain books. Performance on other digitized collections may vary. Performance on other types of collections may vary.
Class boundaries can be ambiguous — particularly between Ex Libris/Decorative and Image/Illustration (e.g., images depicting Ex Libris designs are contextually ambiguous).
The "Artifact" class groups heterogeneous content (hands, clips, barcodes, fractured pages) that may not generalize to scanning artifacts from different digitization workflows.
Chart/Graph has the lowest precision (0.87), reflecting the visual diversity within this category.

Terms of Use

By accessing or downloading the model or otherwise using the Service, you agree to the following:

Noncommercial Use Only
You may use the Service solely for noncommercial purposes. Open-source projects and other public-use efforts are welcome, even if they may indirectly support commercial use, so long as they are unaffiliated with commercial actors or intent.

If you are affiliated with a commercial organization or plan to use the Service for commercial purposes (including AI model training), you will contact us first at contact@institutional.org.
No Redistribution
You may not share or redistribute the Service or any of the data provided through the Service, in whole or in part, including through public repositories or aggregators. If you want others to access it, please direct them to the attribution link.
Derivative Works
You may create derivative works for noncommercial use, but you may not make available any such derivative works that substantially reproduce the original model. Only outputs that are significantly transformed and cannot substitute for the original—such as evaluations, summary statistics, or visualizations—may be shared, with attribution.
Attribution
If you use the model in public-facing work, you must include attribution substantially similar to:

Institutional Books provided by the Institutional Data Initiative with source material contributed by Harvard Library, available at https://institutional.org.

Minor modifications to fit citation style or formatting are permitted, provided the essential elements remain intact.
Provisional Terms
These terms apply only to this release and may change. We are actively working with contributing institutions to develop a long-term framework for responsible, open data sharing.
DISCLAIMER OF WARRANTY
TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, ACCESS TO “SERVICE IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND (EXPRESS, IMPLIED, OR OTHERWISE), INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. WE DO NOT WARRANT THAT THE SERVICE WILL OPERATE IN AN UNINTERRUPTED OR ERROR-FREE MANNER OR THAT THE SERVICE IS FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS. WITHOUT LIMITING THE FOREGOING, WE DO NOT WARRANT THAT (A) THE SERVICE WILL MEET YOUR REQUIREMENTS OR EXPECTATIONS OR ACHIEVE THE INTENDED PURPOSES; (B) THE SERVICE WILL NOT EXPERIENCE OUTAGES OR OTHERWISE WILL BE UNINTERRUPTED, TIMELY, OR SECURE; (C) THE INFORMATION OR SERVICES OBTAINED THROUGH OR FROM THE SERVICE WILL BE ACCURATE, COMPLETE, CURRENT, ERROR-FREE, OR RELIABLE; (D) ANY DEFECTS IN OR ON THE SERVICE WILL BE CORRECTED; OR (E) THAT ANY POLLS OR OTHER SOLICITATIONS OF INFORMATION POSTED THROUGH THE SERVICE BY YOU OR OTHER USERS ARE SAFE OR APPROPRIATE FOR YOUR OR OTHER USERS’ PARTICIPATION. WE MAKE NO REPRESENTATION OR WARRANTY REGARDING YOUR ABILITY TO TRANSMIT AND RECEIVE INFORMATION FROM OR THROUGH THE SERVICE, AND YOU AGREE AND ACKNOWLEDGE THAT YOUR ABILITY TO ACCESS THE SERVICE MAY BE IMPAIRED.
LIMITATION OF LIABILITY
EXCEPT INSOFAR AS THE FOLLOWING LIMITATION MAY BE PROHIBITED BY APPLICABLE LAW, WE SHALL NOT BE LIABLE TO YOU OR TO ANY THIRD PARTY FOR ANY DIRECT, CONSEQUENTIAL, INDIRECT, PUNITIVE, SPECIAL, OR INCIDENTAL DAMAGES, WHETHER FORESEEABLE OR UNFORESEEABLE (INCLUDING, BUT NOT LIMITED TO, LOSS OF PROFITS OR EARNING POWER, LOSS OF DATA, LOSSES DUE TO ERRORS OR INTERRUPTION IN AVAILABILITY OF THE SERVICE, UNAVAILABILITY OF ANY SERVICE, SERVER, OR COMMUNICATIONS FACILITY, OR DAMAGES DUE TO ACTS OR OMISSIONS OF OTHERS USING THE SERVICE), ARISING OUT OF OR RELATING TO THE SERVICE, INCLUDING WITHOUT LIMITATION YOUR AND OTHERS’ USE OF OR INABILITY TO USE THE SERVICE, OR YOUR RELIANCE UPON INFORMATION OBTAINED FROM OR THROUGH THE SERVICE, WHETHER BASED IN CONTRACT, TORT, STATUTORY, OR OTHER LAW. OUR TOTAL CUMULATIVE LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE SERVICE (INCLUDING, WITHOUT LIMITATION, IN THE WAYS DESCRIBED IN THE PRECEDING SENTENCE), WHETHER BASED IN CONTRACT, TORT, STATUTORY, OR OTHER LAW, WILL NOT EXCEED THE AMOUNT, IF ANY, THAT YOU PAID US TO USE THE SERVICE IN THE TWELVE MONTHS PRECEDING THE CLAIM. THE DISCLAIMERS AND LIMITATIONS SET FORTH IN THIS SECTION SHALL APPLY, TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, WHETHER OR NOT WE HAVE BEEN NEGLIGENT OR OTHERWISE AT FAULT.
YOU ACKNOWLEDGE THAT, FOR PURPOSES OF THE FOREGOING DISCLAIMERS AND LIMITATIONS, AS WELL AS THE INDEMNITY PROVISION IN SECTION 8 BELOW, THE TERMS “WE,” “OUR,” “US,” “INSTITUTIONAL DATA INITIATIVE,” AND “IDI” INCLUDE THE CORPORATE BODY PRESIDENT AND FELLOWS OF HARVARD COLLEGE, ALSO KNOWN AS HARVARD UNIVERSITY, AND ITS VARIOUS SCHOOLS, THE MEMBERS OF ITS GOVERNING BOARDS, AND ITS OFFICERS, FACULTY MEMBERS, EMPLOYEES, FELLOWS, AND TO THE EXTENT WORKING ON IDI, ITS STUDENTS, CONTRACTORS, AND REPRESENTATIVES.
Indemnification
You agree to indemnify us and hold us harmless from any and all claims, liabilities, damages, losses and expenses, including reasonable attorneys’ fees and costs, relating to or arising out of (a) your use or attempted use of the Service in violation of these Terms of Service; or (b) your violation of any law or rights of any third party in connection with your use of the Service.
Governing Law/ Jurisdiction
You agree that the Terms of Service and any claim or dispute arising out of or relating to the Service or Terms of Service will be governed by the laws of the Commonwealth of Massachusetts, excluding its conflicts of laws principles. You agree that all such claims and disputes will be heard and resolved exclusively in the federal or state courts located in and serving Middlesex or Suffolk County, Massachusetts, U.S.A. You consent to the personal jurisdiction of those courts over you for this purpose, and you waive and agree not to assert any objection to such proceedings in those courts (including any defense or objection of lack of proper jurisdiction or inconvenience of forum).
Whole Agreement/ Amendment
These Terms of Service constitute the entire agreement between you and Harvard with respect to your use of the Service. We reserve the right to amend these Terms of Service at any time. The Service will post notice of changes to the terms on this webpage, and by accessing the Service after modifications to these Terms of Service have been posted, you agree to be bound by all the modified terms. Accordingly, you should periodically revisit this page to review the then-current Terms of Service.

Citation

@misc{mendez2026institutionalbooksvisual,
      title={Institutional Books - Visual Elements: An open-source pipeline for extracting, classifying, deduplicating, and captioning visual elements from digital book collections}, 
      author={Jimmy Mendez and Matteo Cargnelutti and David Lowry-Duda and Catherine Brobston and Salwa Ismail and Greg Leppert and Amanda Watson and Jonathan Zittrain},
      year={2026},
      eprint={2608.18957},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.18957}, 
}

Downloads last month: 34

Dataset used to train institutional/institutional-books-visual-elements-classification

Collection including institutional/institutional-books-visual-elements-classification

Institutional Books

Collection

A growing corpus of public domain books from library collections, seeded by Harvard Library. • 12 items • Updated about 21 hours ago • 8

Paper for institutional/institutional-books-visual-elements-classification

Institutional Books - Visual Elements: An open-source pipeline for extracting, classifying, deduplicating, and captioning visual elements from digital book collections

Paper • 2608.18957 • Published 3 days ago

Evaluation results

Precision (Image/Illustration) on Institutional Books Visual Elements (test split)
self-reported

0.980
Recall (Image/Illustration) on Institutional Books Visual Elements (test split)
self-reported

0.970
Precision (Artifact) on Institutional Books Visual Elements (test split)
self-reported

0.980
Recall (Artifact) on Institutional Books Visual Elements (test split)
self-reported

0.990
Precision (Ex Libris/Decorative) on Institutional Books Visual Elements (test split)
self-reported

0.960
Recall (Ex Libris/Decorative) on Institutional Books Visual Elements (test split)
self-reported

0.960
Precision (Music) on Institutional Books Visual Elements (test split)
self-reported

0.990
Recall (Music) on Institutional Books Visual Elements (test split)
self-reported

0.990