yyilmazecrin/perceiver-contrastive-practice

Perceiver experiments: Contrastive

Overview

This is an experimental Perceiver codebase for Contrastive. It keeps the tiny setup intentionally manageable so architecture changes can be inspected before a full training run.

---

Default Experiment Recipe

Defaults: rmsprop optimizer, constant warmup schedule. Treat these as starting points; a real run should sweep learning rate and batch size before drawing conclusions.

---

Limitations

The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here.

---

Changelog

1. 2026-10-08 — initial upload
2. 2026-10-22 — config and docs sync

---

Repository Status

1. Single-file implementation; the __main__ block runs a smoke test.
2. config.json and training_args.json hold the generated settings.
3. model.safetensors is freshly initialized — plumbing, not results.
4. No benchmark numbers are reported on purpose.

---

Quick Check

Run python run.py --help for the built-in smoke test. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.

---

Evaluation Guidance

1. pick a task dataset such as a task-specific held-out set
2. fix a tuning budget in advance
3. compare against a capacity-matched baseline
4. publish raw logs next to any claim

---

Architecture

1. Architecture: Perceiver
2. Scale: tiny
3. Attention: multi query
4. Fusion: cross attention
5. Activation: gelu tanh
6. Normalization: batchnorm

---

Files

File	Purpose
run.py	primary artifact
README.md	this documentation
config.json	architecture configuration
training_args.json	default experiment settings
model.safetensors	initialization checkpoint
USE_POLICY.md	use and redistribution notes
NOTES.md	informal to-do notes

---

License

Released under cc-by-4.0. Review the source-data terms separately when this repository is used with external datasets.