Hugging Face's logo

Spaces:
vialibre
/
edia_lmodels_es
Runtime error

App Files Community
nanom commited on
Commit
47c6ffb
·
1 Parent(s): f1ad590

Disabling parallelism to tring avoid deadlocks in the hf tokenizer

Browse files
Files changed (1) hide show
  1. modules/module_languageModel.py +2 -0
modules/module_languageModel.py CHANGED
@@ -1,4 +1,6 @@
1
  from transformers import AutoTokenizer, AutoModelForMaskedLM
 
 
2
 
3
  class LanguageModel:
4
  def __init__(
 
1
  from transformers import AutoTokenizer, AutoModelForMaskedLM
2
+ import os
3
+ os.environ["TOKENIZERS_PARALLELISM"] = "false"
4
 
5
  class LanguageModel:
6
  def __init__(