Instructions to use hon9kon9ize/Cantonese-Llama-2-7B-preview20240625 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hon9kon9ize/Cantonese-Llama-2-7B-preview20240625 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="hon9kon9ize/Cantonese-Llama-2-7B-preview20240625")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("hon9kon9ize/Cantonese-Llama-2-7B-preview20240625")
model = AutoModelForCausalLM.from_pretrained("hon9kon9ize/Cantonese-Llama-2-7B-preview20240625")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hon9kon9ize/Cantonese-Llama-2-7B-preview20240625 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hon9kon9ize/Cantonese-Llama-2-7B-preview20240625"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hon9kon9ize/Cantonese-Llama-2-7B-preview20240625",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/hon9kon9ize/Cantonese-Llama-2-7B-preview20240625

SGLang

How to use hon9kon9ize/Cantonese-Llama-2-7B-preview20240625 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hon9kon9ize/Cantonese-Llama-2-7B-preview20240625" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hon9kon9ize/Cantonese-Llama-2-7B-preview20240625",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hon9kon9ize/Cantonese-Llama-2-7B-preview20240625" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hon9kon9ize/Cantonese-Llama-2-7B-preview20240625",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use hon9kon9ize/Cantonese-Llama-2-7B-preview20240625 with Docker Model Runner:
```
docker model run hf.co/hon9kon9ize/Cantonese-Llama-2-7B-preview20240625
```

Cantonese LLM using Llama-2 7B Architecture

Welcome to the preview of the Cantonese Language Model (LLM) built on the Llama-2 7B architecture. This model is designed to understand and generate text in Cantonese, including slangs, colloquials, and Internet terms.

License

This project is available under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). For more details, please visit the license page.

Preview Warning

Please be advised that this version of the Cantonese LLM is a preview. As such, the model's outputs may sometimes be inaccurate, hallucinatory, or potentially offensive to some individuals. We are continuously working to improve the model's accuracy and reduce such instances.

Training Infrastructure

The Cantonese LLM has been trained using Amazon HyperPod and AWS Trainium chips.

Training Credits

This model was trained by Votee AI Limited, and we contribute to hon9kon9ize, the Hong Kong AI Research Community.

Usage Guidelines

Ensure that you are aware of the potential for unexpected or offensive content.
Always review and assess the model's output before using it in any application.
Provide feedback on any issues you encounter to help us improve the model.

Contributions

We welcome contributions from the community. If you have suggestions or improvements, please submit a pull request or open an issue in the project repository.

Disclaimer

The developers of the Cantonese LLM are not responsible for any harm or offense caused by the model's outputs. Users are advised to exercise discretion and judgment when using the model.

Thank you for exploring the Cantonese LLM. We are excited to see the innovative ways in which it will be used!

Downloads last month: 8

Safetensors

Model size

7B params

Tensor type

F16

Model tree for hon9kon9ize/Cantonese-Llama-2-7B-preview20240625

Base model

meta-llama/Llama-2-7b

Finetuned

(41)

this model