Magnolia-v3-12B / README.md
grimjim's picture
Initial release
abd98a2
|
raw
history blame
2.71 kB
---
base_model:
- grimjim/mistralai-Mistral-Nemo-Base-2407
- grimjim/magnum-consolidatum-v1-12b
- nbeerbower/Mistral-Nemo-Prism-12B
- TheDrummer/Rocinante-12B-v1.1
- grimjim/magnum-twilight-12b
- grimjim/mistralai-Mistral-Nemo-Instruct-2407
library_name: transformers
pipeline_tag: text-generation
tags:
- mergekit
- merge
license: apache-2.0
---
# Magnolia-v3-12B
This repo contains a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
This merge takes advantage of task arithmetic to incorporate the influence of two additional models at low weight in order to improve default creative outputs.
Tested at temperature 1.0 with minP 0.01; that was it. Mistral prompt formats for Nemo should work, although I successfully tested with a variant:
```
User message prefix: [INST]user
User message suffix: [/INST]
Assistant message prefix: [INST]assistant
Assistant message suffix: [/INST]
```
Testing with a blank sysprompt was not awful, though a good sysprompt will elicit more desired behavior.
## Merge Details
### Merge Method
This model was merged using the [task arithmetic](https://arxiv.org/abs/2212.04089) merge method using [grimjim/mistralai-Mistral-Nemo-Base-2407](https://huggingface.co/grimjim/mistralai-Mistral-Nemo-Base-2407) as a base.
### Models Merged
The following models were included in the merge:
* [grimjim/magnum-consolidatum-v1-12b](https://huggingface.co/grimjim/magnum-consolidatum-v1-12b)
* [nbeerbower/Mistral-Nemo-Prism-12B](https://huggingface.co/nbeerbower/Mistral-Nemo-Prism-12B)
* [TheDrummer/Rocinante-12B-v1.1](https://huggingface.co/TheDrummer/Rocinante-12B-v1.1)
* [grimjim/magnum-twilight-12b](https://huggingface.co/grimjim/magnum-twilight-12b)
* [grimjim/mistralai-Mistral-Nemo-Instruct-2407](https://huggingface.co/grimjim/mistralai-Mistral-Nemo-Instruct-2407)
### Configuration
The following YAML configuration was used to produce this model:
```yaml
base_model: grimjim/mistralai-Mistral-Nemo-Base-2407
dtype: bfloat16
merge_method: task_arithmetic
parameters:
normalize: true
slices:
- sources:
- layer_range: [0, 40]
model: grimjim/mistralai-Mistral-Nemo-Base-2407
- layer_range: [0, 40]
model: grimjim/mistralai-Mistral-Nemo-Instruct-2407
parameters:
weight: 0.9
- layer_range: [0, 40]
model: grimjim/magnum-consolidatum-v1-12b
parameters:
weight: 0.1
- layer_range: [0, 40]
model: grimjim/magnum-twilight-12b
parameters:
weight: 0.001
- layer_range: [0, 40]
model: TheDrummer/Rocinante-12B-v1.1
parameters:
weight: 0.001
- layer_range: [0, 40]
model: nbeerbower/Mistral-Nemo-Prism-12B
parameters:
weight: 0.05
```