Upload tokenizer.json
#28
by
ArthurZ
HF staff
- opened
No description provided.
ArthurZ
changed pull request status to
merged
Per IANA registry, iw was deprecated as the code for Hebrew in 1989 and the preferred code is he
Original PR was merged into whisper here - https://github.com/openai/whisper/pull/401
HuggingFace transformers PR here - https://github.com/huggingface/transformers/pull/21310
The correct subtag:
%%
Type: language
Subtag: he
Description: Hebrew
Added: 2005-10-16
Suppress-Script: Hebr
%%
And the deprecation
%%
Type: language
Subtag: iw
Description: Hebrew
Added: 2005-10-16
Deprecated: 1989-01-01
Preferred-Value: he
Suppress-Script: Hebr
%%