Update README.md
Browse files
README.md
CHANGED
|
@@ -6,6 +6,9 @@ license: mit
|
|
| 6 |
|
| 7 |
The Phi-3-22b is a depth upsampled version of the 14b [Phi-3-medium-128k-instruct](https://huggingface.co/microsoft/Phi-3-medium-128k-instruct). We removed the bottom 8 layers of one copy of the 14b and the top 8 layers of another copy of the 14b model and stacked them. We plan to do continued pretraining to improve performance.
|
| 8 |
Since this model has not been continued pretrained, the quality may vary.
|
|
|
|
|
|
|
|
|
|
| 9 |
```
|
| 10 |
!pip install flash-attn --no-build-isolation
|
| 11 |
!pip install peft bitsandbytes accelerate transformers
|
|
|
|
| 6 |
|
| 7 |
The Phi-3-22b is a depth upsampled version of the 14b [Phi-3-medium-128k-instruct](https://huggingface.co/microsoft/Phi-3-medium-128k-instruct). We removed the bottom 8 layers of one copy of the 14b and the top 8 layers of another copy of the 14b model and stacked them. We plan to do continued pretraining to improve performance.
|
| 8 |
Since this model has not been continued pretrained, the quality may vary.
|
| 9 |
+
|
| 10 |
+
Some tests of the model in [colab](https://colab.research.google.com/drive/1eLoQXhysnBmN7DNNB6yElpELOSe6DHHH?usp=sharing).
|
| 11 |
+
|
| 12 |
```
|
| 13 |
!pip install flash-attn --no-build-isolation
|
| 14 |
!pip install peft bitsandbytes accelerate transformers
|