Hugging Face's logo

mesolitica
/
malaysian-llama-3-8b-262k

Text Generation
Transformers
Safetensors
llama
text-generation-inference
Inference Endpoints
Model card Files Community

Malaysian Llama-3 8B 262144 context length

262144 context length and 207100000 RoPE Theta.

WanDB, https://wandb.ai/huseinzol05/EasyContext-262144?nw=nwuserhuseinzol05

Source code, https://github.com/mesolitica/malaya/tree/master/session/llama3#extend-1m-context-length

Special thanks to https://github.com/jzhang38/EasyContext for wrapping https://github.com/zhuzilin/ring-flash-attention for distributed training!

Downloads last month
16
Safetensors
Model size
8.03B params
Tensor type
BF16
·
Inference Examples
Text Generation
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for mesolitica/malaysian-llama-3-8b-262k

Quantizations
1 model

Collection including mesolitica/malaysian-llama-3-8b-262k