Hugging Face's logo

mgoin
/
TinyLlama-1.1B-Chat-v1.0-pruned2.4-compressed

Text Generation
Transformers
Safetensors
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)
from sparseml.transformers import SparseAutoModelForCausalLM, SparseAutoTokenizer

ORIG_MODEL_PATH = "neuralmagic/TinyLlama-1.1B-Chat-v1.0-pruned2.4"
OUTPUT_PATH = "./compress_output"
HF_MODEL_ID = "mgoin/TinyLlama-1.1B-Chat-v1.0-pruned2.4-compressed"

# Compress and export the model
model = SparseAutoModelForCausalLM.from_pretrained(ORIG_MODEL_PATH, device_map="auto", torch_dtype="auto")
tokenizer = SparseAutoTokenizer.from_pretrained(ORIG_MODEL_PATH)

model.save_pretrained(OUTPUT_PATH, save_compressed=True)
tokenizer.save_pretrained(OUTPUT_PATH)

# Upload the checkpoint to Hugging Face
from huggingface_hub import HfApi
HfApi().upload_folder(
    folder_path=OUTPUT_PATH,
    repo_id=HF_MODEL_ID,
)
Downloads last month
3
Safetensors
Model size
753M params
Tensor type
I64
路
FP16
路
U8
路
Text Generation
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.