SteamBERTh

SteamBERTh is a sentence-embedding model for English scientific and technical texts of the seventeenth and eighteenth centuries. It is MacBERTh, a BERT model pre-trained on historical English (1450–1950), fine-tuned on the titles of scientific and technical books printed in Britain, 1600–1800, and on the short descriptions of English patents.

The purpose of the model is to serve for text-comparisons for scientific and technical text between 1600 and 1800: An embedding model for the age of steam.

The model was developed in Koschnick (2025), where it underlies text-based measures of innovation and of knowledge spillovers between fields. These measures are available in the Python package innovfromtext.

Usage

With sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("juliuskoschnick/SteamBERTh")
titles = [
    "A new and easy method of finding the longitude at sea",
    "The art of navigation improved, with tables of the sun's declination",
    "The compleat husbandman: or, a treatise of the improvement of land",
]
emb = model.encode(titles, normalize_embeddings=True)
print(emb @ emb.T)      # cosine similarities

Innovation and spillover indices with innovfromtext:

import innovfromtext as ift

df = ift.load_example()        # Philosophical Transactions titles 1665-1800, English patents 1700-1800
out = ift.compute(df, "innovation", model="juliuskoschnick/SteamBERTh")

Why use a historical sentence-transformer model?

Models trained on modern text carry anachronistic associations: they miss that a "fire engine" was a steam engine or that "philosophical instruments" were scientific instruments, or they place concepts such as "phlogiston" far from the science of the time. MacBERTh avoids this through its historical pre-training corpus, about 70 per cent of which dates from before 1800. Base BERT models are, however, not trained for sentence similarity, and their embedding spaces are strongly anisotropic. Contrastive fine-tuning on the texts of interest addresses both problems.

Training

• Base model: emanjavacas/MacBERTh, mean pooling over tokens.
• Data: 21,760 texts: all titles of the English Short Title Catalogue (ESTC) in applied physics, astronomy, chemistry, mathematics, encyclopedias and dictionaries, technical instructions in trades and agriculture, navigation, and scientific instruments, and the short descriptions of English patents from Woodcroft (1854); patent descriptions with fewer than three words are dropped.
• Objective: unsupervised SimCSE (Gao, Yao and Chen 2021): each text is paired with itself, dropout produces two different views, and the other texts of the batch serve as negatives (multiple negatives ranking loss, scale 15).
• Hyperparameters: 2 epochs, batch size 32, learning rate 2e-5, weight decay 0.01, linear warm-up over the first 10 per cent of steps, maximum sequence length 512.
• Reproducibility: fixed seeds (42), deterministic CUDA kernels, logged software versions (sentence-transformers 3.2.1, transformers 4.44.2, torch 2.2.1).

Evaluation

Koschnick (2025, appendix) documents the evaluation; in brief:

• The embedding space of SteamBERTh is much more isotropic than that of the base model, and fields and industries form more coherent clusters.
• An innovation index built on SteamBERTh embeddings predicts historical patent citations (within-year elasticity 0.18 for 1720–1799) and the number of editions of books (0.17). In a horse race for 1720–1800 it outperforms indices built on modern BERT models.

Limitations

The model is tuned for English scientific and technical texts of roughly 1600–1800, mostly book titles and patent descriptions. Koschnick (2025) finds that modern models perform slightly better on nineteenth-century patents. The model has not been evaluated on other genres or periods.

Citation

If you use SteamBERTh, please cite Koschnick (2025) that introduces it and the MacBERTh papers (the MacBERTh project asks users of MacBERTh to cite Manjavacas and Fonteyn 2022):

@misc{koschnick2025feedback,
  title         = {Did a feedback mechanism between propositional and prescriptive knowledge create modern growth?},
  author        = {Koschnick, Julius},
  year          = {2025},
  eprint        = {2512.16587},
  archivePrefix = {arXiv},
  primaryClass  = {econ.GN},
  url           = {https://arxiv.org/abs/2512.16587}
}

@article{manjavacas2022adapting,
  title   = {Adapting vs. Pre-training Language Models for Historical Languages},
  author  = {Manjavacas, Enrique and Fonteyn, Lauren},
  journal = {Journal of Data Mining \& Digital Humanities},
  volume  = {NLP4DH},
  year    = {2022},
  doi     = {10.46298/jdmdh.9152},
  url     = {https://doi.org/10.46298/jdmdh.9152}
}

@inproceedings{manjavacas-arevalo-fonteyn-2021-macberth,
  title     = {{M}ac{BERT}h: Development and Evaluation of a Historically Pre-trained Language Model for {E}nglish (1450-1950)},
  author    = {Manjavacas Arevalo, Enrique and Fonteyn, Lauren},
  booktitle = {Proceedings of the Workshop on Natural Language Processing for Digital Humanities},
  year      = {2021},
  address   = {NIT Silchar, India},
  publisher = {NLP Association of India (NLPAI)},
  pages     = {23--36},
  url       = {https://aclanthology.org/2021.nlp4dh-1.4/}
}

Base model: emanjavacas/MacBERTh.

License

MIT, as the base model MacBERTh.