dbcccc/IBEM-im2typst

Source code and browser application: GitHub

IBEM im2typst — Phase 10

An experimental printed-formula image-to-Typst recognizer trained from project seeded random initialization using only real IBEM formula crops. This repository contains the current best validation-selected Phase 10 step-2,000 weights.

Status

This model is not production-qualified.

• Parameters: 14,157,152
• FP32 validation exact match: 92.6268% over 16,899 document-disjoint images
• At-most-32-token exact match: 97.8519%
• 33–64-token exact match: 88.4282% (below the 90% production gate)
• Displayed-formula exact match: 70.1197%
• Test split and confidence calibration: unopened
• INT8 built-in export probes: exact token parity
• Full-domain INT8 evaluation: not completed

The displayed token score is uncalibrated and is not the probability that a formula is correct. Inspect and compile every generated Typst expression.

Files

File	Purpose
checkpoint-step-002000-transfer.pt	56.7 MB optimizer-free FP32 PyTorch transfer checkpoint
checkpoint-metadata.json	Checkpoint identity, lineage, hashes, and removed-state declaration
encoder.int8.onnx	Dynamic-INT8 browser encoder
decoder-step.int8.onnx	Cached autoregressive INT8 decoder step
model-config.json	Recognizer architecture
preprocess-config.json	Locked input preprocessing
vocabulary.json	528-token vocabulary with UTF-8 byte fallback
output-policy.json	Locked package-free Typst output surface
browser-deployment.json	Original browser export identity and artifact evidence
evaluation.json	Complete FP32 validation report
onnx-parity.json	FP32, FP16, and INT8 export-probe parity evidence

The PyTorch file retains model_state and the lineage metadata required for exact-model transfer. Optimizer state, RNG state, epoch/batch cursor, and cached metrics were deliberately removed. It cannot resume the original optimizer.

browser-deployment.json preserves the identity of the original multi-precision export and therefore names FP16/FP32 ONNX variants that are not copied here. This model repository contains the FP32 PyTorch state and the two small INT8 ONNX graphs; the unused FP16 and reference ONNX duplicates are intentionally omitted.

Using the weights

The model uses a project-specific cnn-transformer-v3 architecture rather than a Transformers architecture, so the hosted Hugging Face inference widget is disabled. Use the companion GitHub source repository for the browser app, Python model definition, preprocessing, training, evaluation, and export code.

After installing that Python package, the FP32 checkpoint can be loaded with:

import torch

from synthetic_im2typst.model import ImageToTypstRecognizer, ModelConfig

checkpoint = torch.load(
    "checkpoint-step-002000-transfer.pt",
    map_location="cpu",
    weights_only=True,
)
config = ModelConfig.from_data(checkpoint["model_config"])
model = ImageToTypstRecognizer(config)
model.load_state_dict(checkpoint["model_state"], strict=True)
model.eval()

The two ONNX graphs are a cached encoder/decoder pair, not independent single-call models. Use them together with vocabulary.json, preprocess-config.json, and the browser inference runtime from the companion source repository.

Architecture and input

The recognizer uses a compact convolutional encoder, depthwise axial 2-D context, a height-4 learned horizontal-preserving pool, and a three-layer autoregressive Transformer decoder. Input is foreground-cropped, polarity-normalized, aspect-preserving resized, and padded to a 256-pixel height with a maximum width of 2,048 pixels.

It expects one tightly cropped machine-typeset formula. It is not a page segmenter, general OCR system, handwriting model, or photograph recognizer.

Training data

• 129,303 real training crops from 480 IBEM documents.
• 16,899 validation crops from 60 different documents.
• Images are grayscale crops from original IBEM page JPEGs, not formulas rerendered in one standardized font.
• Targets are deterministic package-free Typst conversions of IBEM latex_norm annotations.
• No Fusion dataset, synthetic formula corpus, teacher-generated labels, external pretrained model, or fused samples were used.

IBEM is available from its official Zenodo record under CC BY 4.0. No IBEM image or annotation is included. The image under example/ is an original project-authored Typst render included only as a smoke-test sample.

Known limitations

• Small dot accents, top attachments, matrices, and alignment structures are common compact-formula errors.
• Long and multiline displayed formulas are substantially weaker than short inline formulas.
• Inputs with handwriting, perspective distortion, surrounding prose, or arbitrary screenshot styling are not qualified.
• Exact-match evaluation is strict: one wrong token makes an output non-exact.

Identity and license

• Original full checkpoint SHA-256: 29d4a51adc71526f395750bb12a1783fbfd9bfa5a4c68fc9bfb665b68c633d52
• Optimizer-free checkpoint SHA-256: 8b7972f440556f328828f9e90247f266d9fbc59266f5ff8b23f7bce21f717924
• Parameter-state hash: dcbf1a037fee205c82b7c52c9d0c0c2ee3d68d4a514f4593e15711f02bad31ec
• Training run ID: 64e092c2013ed43e35d3573e3e2a161c9948f12279ebb92513ab6b15c8811814

Project code, the identified weights, and the project-authored example are MIT licensed. IBEM is not redistributed and retains its source license. See LICENSE, LICENSES.md, MODEL_LICENSE.md, and DATA_PROVENANCE.md.