openmodelai/Terra-2.5-Vision

Terra 2.5 Vision

This code uses the existing Terra 2.5 350M text checkpoint from this repository as its language model and adds an image encoder plus a trainable visual projector. It does not replace or alter Terra's released text weights.

A text checkpoint has no image knowledge or image-processing layers. The released Terra ZIP is a GPT2LMHeadModel text model, so image understanding requires a separate pretrained vision encoder and training the connector on image/question/answer examples. The files here implement that real integration; the adapter checkpoint must be trained before image answers are meaningful.

1. Install and prepare checkpoints

Python 3.10+, PyTorch 2.2+, Transformers 4.44+, Pillow 10+:

python -m pip install -r terra_vision/requirements.txt

Extract terra-2.5-350m-hf.zip from this repository's GitHub Release on your machine. Also prepare a Transformers-compatible ViT checkpoint, e.g. google/vit-base-patch16-224-in21k; it must return last_hidden_state from AutoModel. You can pass local directories for offline use. The code defaults to local_files_only=True, and only downloads models when --allow-download is explicitly supplied.

2. Train the vision connector

Prepare licensed image/question/answer training examples as JSON Lines. Each image path is relative to --image-root:

{"image":"train/cat-001.jpg","question":"Что изображено на картинке?","answer":"На картинке кошка."}
{"image":"train/street-002.jpg","question":"How many cars are visible?","answer":"There are two cars."}

Then run from the repository root:

python -m terra_vision.train_adapter \
  --terra-dir /models/terra-2.5-350m \
  --vision-dir /models/vit-base-patch16-224-in21k \
  --manifest /data/train.jsonl \
  --image-root /data \
  --output /models/terra-vision-projector.pt \
  --epochs 3 --batch-size 1 --device cuda

The released Terra language model and ViT are frozen; only the visual projector is trained. This preserves the existing model weights and lowers optimizer memory, but still requires backpropagating through Terra. Use a properly varied, licensed dataset and held-out validation before claiming quality. The script writes a projector state dict and a companion JSON metadata file; neither is included in Git.

3. Ask a question about an image

python -m terra_vision.chat \
  --terra-dir /models/terra-2.5-350m \
  --vision-dir /models/vit-base-patch16-224-in21k \
  --adapter /models/terra-vision-projector.pt \
  --image /data/test.jpg \
  --question "Что изображено на картинке?" \
  --max-new-tokens 64 --device cuda

The inference path preprocesses the image with the ViT's own image processor, projects its visual features into 32 learned prefix embeddings, and calls the Terra model's generate() method. The prefix consumes 32 positions from Terra's 256-token context; generation is capped to the remaining context.

4. Build the Hugging Face upload folder and ZIP

After training, create a complete bundle containing Terra weights, ViT weights, the processor, trained projector, source, config, model card, and checksums:

python -m terra_vision.package_hf \
  --terra-dir /models/terra-2.5-350m \
  --vision-dir /models/vit-base-patch16-224-in21k \
  --adapter /models/terra-vision-projector.pt \
  --output-dir /models/terra-2.5-vision-hf \
  --zip /models/terra-2.5-vision-hf.zip

The output directory is suitable for uploading as a Hugging Face model repository. The script refuses to package a missing or mismatched trained adapter; it does not put random projector weights in a release.

Current scope and validation

This is the first multimodal training/inference and packaging path, not an officially released or benchmarked Terra Vision checkpoint. Image understanding quality depends on the encoder and adapter training data. No vision weights or image dataset are bundled. Generation uses Transformers' inputs_embeds path; run the included tests in the pinned environment and smoke-test against the actual extracted Terra checkpoint before deployment.

python -m pip install -r terra_vision/requirements-dev.txt
python -m pytest tests/test_terra_vision.py -q