miwilliams9/contrastive-int8

DeiT for Contrastive

Overview

A research-oriented DeiT prototype targeting Contrastive. The included xlarge setup documents defaults and file formats without presenting unverified performance numbers.

Quick check

python main.py --help

Inspect the script's __main__ block for its generated smoke-test example. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.

Repository status

• The Python file contains the model and runnable example or training entry point.
• config.json records the generated architecture settings.
• training_args.json records the default experiment recipe.
• model.safetensors is a valid initialization checkpoint for smoke tests; it is not presented as a trained benchmark checkpoint.
• No benchmark score is claimed in this repository.

Default experiment recipe

The generated recipe uses adam with a exponential schedule.

These values exist so the script runs end-to-end. A meaningful experiment needs its own tuning budget and at least three seeds.

Architecture

• Architecture: DeiT
• Scale: xlarge
• Attention: sparse
• Fusion: co attention
• Activation: gelu tanh
• Normalization: scalenorm

Evaluation guidance

A first evaluation could use a task-specific held-out set. Report the metric across at least three seeds and include a matched-capacity baseline; keep logs and library versions with any published number.

Roadmap

Next steps, in no particular order: a cleaner entry point, input-shape docs, and a small evaluation script if time permits.

Limitations

The checkpoint is untrained and unaudited. Treat the repository as an experimental starting point; anything learned from a future trained version must be documented separately.

Citation

If this repository is useful for your work, a link back is appreciated. No formal citation is required yet.

Files

File	Purpose
main.py	primary artifact
README.md	this documentation
config.json	architecture configuration
training_args.json	default experiment settings
model.safetensors	initialization checkpoint
USE_POLICY.md	use and redistribution notes
NOTES.md	informal to-do notes
examples/sample_usage.py	usage sketch

License

Distributed under the mpl-2.0 license. External datasets keep their own terms — check them separately.