Refact 1.6B FIM GGUF
Introduction
The Refact 1.6B FIM GGUF model is a state-of-the-art AI-powered coding assistant developed by Small Magellanic Cloud AI Ltd. This model is designed to assist developers with code completion, refactoring, and chat-based interactions, excelling in code-related natural language understanding and generation tasks.
Quantized Model Files
The model comes in various quantized versions to suit different computational needs:
- refact-1.6B-fim-q4_0.gguf: A 4-bit quantized model with a file size of 878 MB.
- refact-1.6B-fim-q5_0.gguf: A 5-bit quantized model with a file size of 1.1 GB.
- refact-1.6B-fim-q8_0.gguf: An 8-bit quantized model with a file size of 1.6 GB.
Features and Usage
The model is versatile and can be employed for:
- Code completion
- Code refactoring
- Chat-based interactions
Example Usage
Here's a sample shell command to invoke the model:
# Sample shell command to use the model
./main -m models/smallcloudai/Refact-1_6B-fim/ggml-model-f16.gguf -n 300 -p "write a function to multiply two integers in python" --temp 1.0 --top-p 1.0 --top-k 1 --repeat_penalty 1.0
Performance Metrics
The model outperforms many existing models in both code completion and chat-based interactions, as evidenced by the HumanEval results.
| Model | Size | HumanEval pass@1 | HumanEval pass@10 |
|---|---|---|---|
| Refact-1.6-fim | 1.6b | 32.0% | 53.0% |
| StableCode | 3b | 20.2% | 33.8% |
| ReplitCode v1 | 3b | 21.9% | N/A |
Installation and Setup
The model can be integrated into your IDE via the Refact plugin. For self-hosting, an open-source Docker container is available.
Limitations and Bias
The model primarily focuses on English text, which may result in lower performance for non-English languages.
Technical Specifications
- Architecture: LLAMA-like model with multi-query attention
- Training Tokens: 1.2T for pretraining, 40B for fine-tuning
- Precision: bfloat16
- Training Time: 28 days
License
The model is licensed under the BigScience OpenRAIL-M v1 license agreement.
Citation
If you use this model in your work, please cite it by linking back to the following page for proper attribution:
Acknowledgments
Special thanks to ds5t5 for their contribution in implementing the source for converting the model's tensors from Hugging Face to GGUF format. Their work has been instrumental in enhancing the model's versatility.
Example Command for Testing
To test the model against Hugging Face, you can use the following command:
# Example command for testing against Hugging Face
python3 convert-refact-hf-to-gguf.py ./Refact-1_6B-fim 1
./main -m ./Refact-1_6B-fim/ggml-model-f16.gguf -n 300 -p "write a function to multiply two integers in python" --temp 1.0 --top-p 1.0 --top-k 1 --repeat_penalty 1.0
This resolves llama.cpp issue #3061.
- Downloads last month
- 153