Hugging Face's logo

opencontext
/
archaeo-clip

Safetensors
English
archaeology
CLIP
Model card Files Community
Edit model card

ABOUT ARCHAEO-CLIP

This model is the result of "fine-tuning" the openai/clip-vit-base-patch32 model using captioned images of archaeological artifacts published by Open Context. This model is the latest of several iterations in experiments to improve the captions, debug the training pipeline, and try different fine-tuning parameters. It seems a model with a relatively low training rate helps add some "archaeological knowledge" while still retain much of the general knowledge of out-of-the-box CLIP.

We'll use this fine-tuned model in the future to do more experiments, including further fine-tuning with captioned images from open access museum collections of archaeological materials. Below we itemize the specific training parameters used in the fine tuning of this model

The training (45,256 captioned archaeological artifact images from Open Context) and test data, as well as the specific Python code used to run the fine-tuning run can be found here:

https://github.com/opencontext/archaeology-images-ai

Fine Tuning

Below is the invocation and specific parameters I used for fine tuning:

python -W ignore finetune-clip-huggingface/huggingface_finetune_clip.py     --output_dir /home/ekansa/github/archaeology-images-ai/results     --model_name_or_path openai/clip-vit-base-patch32     --train_file /home/ekansa/github/archaeology-images-ai/files/train.json     --validation_file /home/ekansa/github/archaeology-images-ai/files/test.json     --image_column="image_path"     --overwrite_output_dir=True     --max_seq_length=77     --num_train_epochs=25     --caption_column="caption"     --overwrite_cache=True     --remove_unused_columns=False     --do_train=True     --per_device_train_batch_size=64     --per_device_eval_batch_size=64     --learning_rate="2e-5" --warmup_steps="2" --weight_decay 0.2

12/10/2023 21:35:43 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
Running tokenizer on train dataset: 100%|███████████████████████████████| 45256/45256 [00:02<00:00, 21481.25 examples/s]Parameter 'transform'=<function main.<locals>.transform_images at 0x7fe53504d9e0> of the transform datasets.arrow_dataset.Dataset.set_format couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
12/10/2023 21:35:47 - WARNING - datasets.fingerprint - Parameter 'transform'=<function main.<locals>.transform_images at 0x7fe53504d9e0> of the transform datasets.arrow_dataset.Dataset.set_format couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.


{'loss': 1.7174, 'learning_rate': 1.9437224545146346e-05, 'epoch': 0.71}
{'loss': 1.1706, 'learning_rate': 1.887218894790372e-05, 'epoch': 1.41}
{'loss': 0.9596, 'learning_rate': 1.8307153350661094e-05, 'epoch': 2.12}
{'loss': 0.7291, 'learning_rate': 1.7742117753418467e-05, 'epoch': 2.82}
{'loss': 0.5833, 'learning_rate': 1.717708215617584e-05, 'epoch': 3.53}
{'loss': 0.5094, 'learning_rate': 1.6612046558933215e-05, 'epoch': 4.24}
{'loss': 0.4368, 'learning_rate': 1.6047010961690588e-05, 'epoch': 4.94}
{'loss': 0.365, 'learning_rate': 1.548197536444796e-05, 'epoch': 5.65}
{'loss': 0.3394, 'learning_rate': 1.4916939767205336e-05, 'epoch': 6.36}
{'loss': 0.3159, 'learning_rate': 1.4351904169962709e-05, 'epoch': 7.06}
{'loss': 0.2776, 'learning_rate': 1.3786868572720083e-05, 'epoch': 7.77}
{'loss': 0.2584, 'learning_rate': 1.3221832975477456e-05, 'epoch': 8.47}
{'loss': 0.2464, 'learning_rate': 1.2656797378234832e-05, 'epoch': 9.18}
{'loss': 0.227, 'learning_rate': 1.2091761780992204e-05, 'epoch': 9.89}
{'loss': 0.2116, 'learning_rate': 1.1526726183749577e-05, 'epoch': 10.59}
{'loss': 0.2026, 'learning_rate': 1.0961690586506951e-05, 'epoch': 11.3}
{'loss': 0.1869, 'learning_rate': 1.0396654989264325e-05, 'epoch': 12.01}
{'loss': 0.1792, 'learning_rate': 9.831619392021698e-06, 'epoch': 12.71}
{'loss': 0.167, 'learning_rate': 9.266583794779072e-06, 'epoch': 13.42}
{'loss': 0.1671, 'learning_rate': 8.701548197536446e-06, 'epoch': 14.12}
{'loss': 0.154, 'learning_rate': 8.136512600293819e-06, 'epoch': 14.83}
{'loss': 0.1574, 'learning_rate': 7.571477003051193e-06, 'epoch': 15.54}
{'loss': 0.1496, 'learning_rate': 7.006441405808566e-06, 'epoch': 16.24}
{'loss': 0.1329, 'learning_rate': 5.876370211323313e-06, 'epoch': 17.66}
{'loss': 0.1316, 'learning_rate': 5.311334614080687e-06, 'epoch': 18.36}
{'loss': 0.1254, 'learning_rate': 4.746299016838062e-06, 'epoch': 19.07}
{'loss': 0.1266, 'learning_rate': 4.181263419595435e-06, 'epoch': 19.77}
{'loss': 0.1193, 'learning_rate': 3.6162278223528084e-06, 'epoch': 20.48}
{'loss': 0.1163, 'learning_rate': 3.0511922251101822e-06, 'epoch': 21.19}
{'loss': 0.1154, 'learning_rate': 2.486156627867556e-06, 'epoch': 21.89}
{'loss': 0.1125, 'learning_rate': 1.9211210306249294e-06, 'epoch': 22.6}
{'loss': 0.1063, 'learning_rate': 1.356085433382303e-06, 'epoch': 23.31}
{'loss': 0.1082, 'learning_rate': 7.91049836139677e-07, 'epoch': 24.01}
{'loss': 0.1032, 'learning_rate': 2.2601423889705053e-07, 'epoch': 24.72}

{'train_runtime': 78442.5601, 'train_samples_per_second': 14.423, 'train_steps_per_second': 0.226, 'train_loss': 0.31630637788503185, 'epoch': 25.0}
100%|██████████████████████████████████████████████████████████████████████████████| 17700/17700 [21:47:22<00:00,  4.43s/it]
***** train metrics *****
  epoch                    =        25.0
  train_loss               =      0.3163
  train_runtime            = 21:47:22.56
  train_samples_per_second =      14.423
  train_steps_per_second   =       0.226

license: gpl-3.0

Credit / Acknowledgements / Blame

Shawn Graham (https://huggingface.co/sgraham https://carleton.ca/history/people/shawn-graham/) figured out how to use available open source tooling to fine-tune the CLIP model.

Eric Kansa (https://github.com/ekansa) put together the captioned training data and further streamlined and debugged the open source tooling to fine-tune the CLIP model.

Many, many data contributors to Open Context did the hard work of excavating, surveying, and documenting the archaeological objects used in this training. Images and data used for captioning came from the following projects:

Finally, we're grateful to the many people (ancient and some modern) who made and used the physical objects that informed this model. Everything we do now is built on their legacy.

Downloads last month
0
Unable to determine this model's library. Check the docs .