This repository contains KenLM models trained on the Jomleh dataset for the Farsi (Persian) language. Among the various use cases for KenLM language models, the models provided here are particularly useful for automatic speech recognition (ASR) tasks. They can be used in conjunction with CTC to select the most likely sequence of tokens extracted from a spectrogram.
The models in this repository are KenLM arpa files that have been converted to binary format. KenLM supports two binary formats: probing and trie. The models provided here are in the probing format, which KenLM claims are faster but have a larger memory footprint.
There are a total of 36 different KenLM models available in this repository. Unless you are conducting research, you will not need all of them. In that case, it is recommended that you download only the models you require rather than the entire repository since the total file size is over half a terabyte.
Unfortunately, I could not find an easy way to integrate the Python code that loads the models using Huggingface library. These are the steps that you have to take when you want to use any of the models provided here:
- Install KenLM package:
pip install https://github.com/kpu/kenlm/archive/master.zip
- Install the SentencePiece for the tokenization:
pip install sentencepiece
- Download the model that you are interested in from this repository along the Python code
model.py. Keep the model in the
filesfolder with the
model.pyby it (just like the file structure in the repository). Don't forget to download the SentencePiece files as well. For instance, if you were interested in 32000 vocabulary size tokenizer, 5-gram model with maximum pruning, these are the files you'll need:
model.py files/jomleh-sp-32000.model files/jomleh-sp-32000.vocab files/jomleh-sp-32000-o5-prune01111.probing
- In your script, instantiate a model and use it like this:
from model import KenlmModel # Load the model model = KenlmModel.from_pretrained("57218", "3", "011") # Get perplexity print(model.perplexity("من در را بستم")) # Outputs: 72.5 # Get score print(model.score("من در را بستم")) # Outputs: -11.160577774047852
The files you can find in this repository are either SentencePiece tokenizer models or KenLM binary models. For the tokenizers, this is the template their file name follows:
In this repository, all the models are based on the Jomleh dataset (
jomleh). And the only
tokenizer used is SentencePiece (
sp). Finally, the list of vocabulary sizes used is composed
of 2000, 4000, 8000, 16000, 32000, and 57218 tokens. Due to hardware limitations that I've
faced, I could only use 4 out of 60 jomleh's text file to train the tokenizer, namely: 10, 11,
12, and 13. Also, 57218 was the largest number that SentencePiece allowed to set as for the
Here's an example of the tokenizer files you can find in this repository:
Moving on to the KenLM binary models, their file names follow this template:
Just like with the tokenizers, the only available options for dataset and tokenizer type are
sp. The same applies to vocabulary sizes. There are two n-grams trained,
3-grams, and 5-grams. Additionally, there are three different pruning options available for
each configuration. To interpret the pruning numbers, add a space between each pair of digits.
0 1 1 was set during training of the KenLM model.
Here is a complete example: To train the binary model named
jomleh-sp-32000.model was used to encode (tokenize) the 95% Jomleh dataset,
resulting in a large text file holding space-separated tokens. Then, the file was fed into
lmplz program with the following input arguments:
lmplz -o 5 -T /tmp --vocab_estimate 32000 -S 80% --discount_fallback --prune "0 1 1 1 1" < enocoded.txt > jomleh-sp-32000-o5-prune01111.arpa
This command will produce the raw arpa file, which can then be converted into binary format
build_binary program, as shown below:
build_binary -T /tmp -S 80% probing jomleh-sp-32000-o5-prune01111.arpa jomleh-sp-32000-o5-prune01111.probing
Based on my personal evaluation, I recommend using the
It's the perfect balanced between file size (6GB) and accuracy (80%). But if you have no concern for file
size, then go for the largest model,
jomleh-sp-57218-o5-prune00011.probing (size: 36GB, accuracy: 82%).
- Downloads last month
Unable to determine this model’s pipeline type. Check the docs .