AI & ML interests

Merging models

The merge crew is the mergiest crew. 😎

Merge Crew Planning Document

Merging tutorial

Colab for merging

Lazy merge kit notebook for merging models.

Model Merging Discord

Discord channel for discussions of model mergin.

Merge methods

When working you use a YAML file like the following

    - model: timpal0l/BeagleCatMunin
      # No parameters necessary for base model
    - model: bineric/NorskGPT-Mistral-7b
        density: 0.53
        weight: 0.6
  merge_method: dare_ties
  base_model: timpal0l/BeagleCatMunin
    int8_mask: true
  dtype: bfloat16
  random_seed: 42

The dare_ties methods seems to perform better than other merging methods. Learn more about merge ties here.


None public yet