---
license: gpl-3.0
language:
- en
tags:
- archaeology
- CLIP
---

## ABOUT ARCHAEO-CLIP


This model is the result of "fine-tuning" the openai/clip-vit-base-patch32 model using captioned images of archaeological artifacts published by Open Context. This model is the latest of several iterations in experiments to improve the captions, debug the training pipeline, and try different fine-tuning parameters. It seems a model with a relatively low training rate helps add some "archaeological knowledge" while still retain much of the general knowledge of out-of-the-box CLIP.

We'll use this fine-tuned model in the future to do more experiments, including further fine-tuning with captioned images from open access museum collections of archaeological materials. Below we itemize the specific training parameters used in the fine tuning of this model

The training (45,256 captioned archaeological artifact images from Open Context) and test data, as well as the specific Python code used to run the fine-tuning run can be found here:

https://github.com/opencontext/archaeology-images-ai







Fine Tuning
-----------
Below is the invocation and specific parameters I used for fine tuning:

```bash
python -W ignore finetune-clip-huggingface/huggingface_finetune_clip.py     --output_dir /home/ekansa/github/archaeology-images-ai/results     --model_name_or_path openai/clip-vit-base-patch32     --train_file /home/ekansa/github/archaeology-images-ai/files/train.json     --validation_file /home/ekansa/github/archaeology-images-ai/files/test.json     --image_column="image_path"     --overwrite_output_dir=True     --max_seq_length=77     --num_train_epochs=25     --caption_column="caption"     --overwrite_cache=True     --remove_unused_columns=False     --do_train=True     --per_device_train_batch_size=64     --per_device_eval_batch_size=64     --learning_rate="2e-5" --warmup_steps="2" --weight_decay 0.2

12/10/2023 21:35:43 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
Running tokenizer on train dataset: 100%|███████████████████████████████| 45256/45256 [00:02<00:00, 21481.25 examples/s]Parameter 'transform'=<function main.<locals>.transform_images at 0x7fe53504d9e0> of the transform datasets.arrow_dataset.Dataset.set_format couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
12/10/2023 21:35:47 - WARNING - datasets.fingerprint - Parameter 'transform'=<function main.<locals>.transform_images at 0x7fe53504d9e0> of the transform datasets.arrow_dataset.Dataset.set_format couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.


{'loss': 1.7174, 'learning_rate': 1.9437224545146346e-05, 'epoch': 0.71}
{'loss': 1.1706, 'learning_rate': 1.887218894790372e-05, 'epoch': 1.41}
{'loss': 0.9596, 'learning_rate': 1.8307153350661094e-05, 'epoch': 2.12}
{'loss': 0.7291, 'learning_rate': 1.7742117753418467e-05, 'epoch': 2.82}
{'loss': 0.5833, 'learning_rate': 1.717708215617584e-05, 'epoch': 3.53}
{'loss': 0.5094, 'learning_rate': 1.6612046558933215e-05, 'epoch': 4.24}
{'loss': 0.4368, 'learning_rate': 1.6047010961690588e-05, 'epoch': 4.94}
{'loss': 0.365, 'learning_rate': 1.548197536444796e-05, 'epoch': 5.65}
{'loss': 0.3394, 'learning_rate': 1.4916939767205336e-05, 'epoch': 6.36}
{'loss': 0.3159, 'learning_rate': 1.4351904169962709e-05, 'epoch': 7.06}
{'loss': 0.2776, 'learning_rate': 1.3786868572720083e-05, 'epoch': 7.77}
{'loss': 0.2584, 'learning_rate': 1.3221832975477456e-05, 'epoch': 8.47}
{'loss': 0.2464, 'learning_rate': 1.2656797378234832e-05, 'epoch': 9.18}
{'loss': 0.227, 'learning_rate': 1.2091761780992204e-05, 'epoch': 9.89}
{'loss': 0.2116, 'learning_rate': 1.1526726183749577e-05, 'epoch': 10.59}
{'loss': 0.2026, 'learning_rate': 1.0961690586506951e-05, 'epoch': 11.3}
{'loss': 0.1869, 'learning_rate': 1.0396654989264325e-05, 'epoch': 12.01}
{'loss': 0.1792, 'learning_rate': 9.831619392021698e-06, 'epoch': 12.71}
{'loss': 0.167, 'learning_rate': 9.266583794779072e-06, 'epoch': 13.42}
{'loss': 0.1671, 'learning_rate': 8.701548197536446e-06, 'epoch': 14.12}
{'loss': 0.154, 'learning_rate': 8.136512600293819e-06, 'epoch': 14.83}
{'loss': 0.1574, 'learning_rate': 7.571477003051193e-06, 'epoch': 15.54}
{'loss': 0.1496, 'learning_rate': 7.006441405808566e-06, 'epoch': 16.24}
{'loss': 0.1329, 'learning_rate': 5.876370211323313e-06, 'epoch': 17.66}
{'loss': 0.1316, 'learning_rate': 5.311334614080687e-06, 'epoch': 18.36}
{'loss': 0.1254, 'learning_rate': 4.746299016838062e-06, 'epoch': 19.07}
{'loss': 0.1266, 'learning_rate': 4.181263419595435e-06, 'epoch': 19.77}
{'loss': 0.1193, 'learning_rate': 3.6162278223528084e-06, 'epoch': 20.48}
{'loss': 0.1163, 'learning_rate': 3.0511922251101822e-06, 'epoch': 21.19}
{'loss': 0.1154, 'learning_rate': 2.486156627867556e-06, 'epoch': 21.89}
{'loss': 0.1125, 'learning_rate': 1.9211210306249294e-06, 'epoch': 22.6}
{'loss': 0.1063, 'learning_rate': 1.356085433382303e-06, 'epoch': 23.31}
{'loss': 0.1082, 'learning_rate': 7.91049836139677e-07, 'epoch': 24.01}
{'loss': 0.1032, 'learning_rate': 2.2601423889705053e-07, 'epoch': 24.72}

{'train_runtime': 78442.5601, 'train_samples_per_second': 14.423, 'train_steps_per_second': 0.226, 'train_loss': 0.31630637788503185, 'epoch': 25.0}
100%|██████████████████████████████████████████████████████████████████████████████| 17700/17700 [21:47:22<00:00,  4.43s/it]
***** train metrics *****
  epoch                    =        25.0
  train_loss               =      0.3163
  train_runtime            = 21:47:22.56
  train_samples_per_second =      14.423
  train_steps_per_second   =       0.226
```




---
license: gpl-3.0
---



Credit / Acknowledgements / Blame
---------------------------------

Shawn Graham (https://huggingface.co/sgraham https://carleton.ca/history/people/shawn-graham/) figured out how to use 
available open source tooling to fine-tune the CLIP model.

Eric Kansa (https://github.com/ekansa) put together the captioned training data and further streamlined and debugged the 
open source tooling to fine-tune the CLIP model.

Many, many data contributors to Open Context did the hard work of excavating, surveying, and documenting archaeology 
materials used in this training.