🏗️ Building on HF
Sayali Pinge
saayoo
·
AI & ML interests
None yet
Recent Activity
repliedto serkanpeker's post about 16 hours ago
Generating the same text in Turkish took an estimated 1.45 to 1.62 times the GPU time of English on four open models (one H200 each, vLLM 0.33, 256 concurrent requests, prefix caching off).
Token count explains most of it. On 2,009 parallel FLORES+ sentences, six of seven tokenizers need 1.36 to 1.62 times as many tokens for Turkish as for English; Kumru-2B needs 0.64 times as many.
Dynamic FP8 in vLLM raised perplexity on FLORES+ by 0.30% to 0.81% in Turkish and 0.27% to 0.39% in English; per token, only Turkish-Gemma-9b-T1 showed a clearly larger rise in Turkish. For GGUF at Q4_K_M on two models, using an importance matrix at all mattered far more than calibrating it on Turkish.
Article (English): https://huggingface.co/blog/serkanpeker/turkish-tax
Data: https://huggingface.co/datasets/serkanpeker/turkish-tax
Code: https://github.com/speker/turkish-tax repliedto serkanpeker's post 1 day ago
Generating the same text in Turkish took an estimated 1.45 to 1.62 times the GPU time of English on four open models (one H200 each, vLLM 0.33, 256 concurrent requests, prefix caching off).
Token count explains most of it. On 2,009 parallel FLORES+ sentences, six of seven tokenizers need 1.36 to 1.62 times as many tokens for Turkish as for English; Kumru-2B needs 0.64 times as many.
Dynamic FP8 in vLLM raised perplexity on FLORES+ by 0.30% to 0.81% in Turkish and 0.27% to 0.39% in English; per token, only Turkish-Gemma-9b-T1 showed a clearly larger rise in Turkish. For GGUF at Q4_K_M on two models, using an importance matrix at all mattered far more than calibrating it on Turkish.
Article (English): https://huggingface.co/blog/serkanpeker/turkish-tax
Data: https://huggingface.co/datasets/serkanpeker/turkish-tax
Code: https://github.com/speker/turkish-taxOrganizations
None yet