ModuleNotFoundError: No module named 'transformers.tokenization_bert'
State: closed | Pull request: no
Labels: name: bug

Hello. I am getting an error.
I was using the[ Sentiment analysis notebook](https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.0b2/tutorials/nlp/Text_Classification_Sentiment_Analysis.ipynb#scrollTo=dzqD2WDFOIN-) on colab by NVIDIA NeMo 
I followed the instructions as it said(run the first cell if you are on colab and vice versa).
In the second cell, it is said that restart runtime after running it. After restarting kernal, (or even if I don't) I am getting this error:
`ModuleNotFoundError: No module named 'transformers.tokenization_bert'`. It is from the first import of the 3rd cell,
 `from nemo.collections import nlp as nemo_nlp`

The full traceback is:

```---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-6-9dea72b2f46b> in <module>()
----> 1 from nemo.collections import nlp as nemo_nlp
      2 from nemo.utils.exp_manager import exp_manager
      3 
      4 import os
      5 import wget

3 frames
/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/__init__.py in <module>()
     13 # limitations under the License.
     14 
---> 15 from nemo.collections.nlp import data, models, modules
     16 from nemo.package_info import __version__
     17 

/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/data/__init__.py in <module>()
     25     NeuralMachineTranslationDataset,
     26 )
---> 27 from nemo.collections.nlp.data.question_answering_squad.qa_dataset import SquadDataset
     28 from nemo.collections.nlp.data.token_classification.token_classification_dataset import (
     29     BertTokenClassificationDataset,

/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/data/question_answering_squad/qa_dataset.py in <module>()
     26 
     27 from nemo.collections.common.parts.utils import _compute_softmax
---> 28 from nemo.collections.nlp.data.question_answering_squad.qa_squad_processing import (
     29     EVALUATION_MODE,
     30     INFERENCE_MODE,

/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/data/question_answering_squad/qa_squad_processing.py in <module>()
     20 
     21 from tqdm import tqdm
---> 22 from transformers.tokenization_bert import BasicTokenizer
     23 
     24 from nemo.collections.nlp.data.data_utils import DataProcessor, is_whitespace, normalize_answer

ModuleNotFoundError: No module named 'transformers.tokenization_bert'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
```
I tried searching stack overflow, but it didn't helped. And I don't want to tinker the main NeMo module myself cause I don't know what might go wrong if I change something. 
What shall I do?

I am also getting this error

It looks like the latest version of transformers (4.0.1) is being installed. I reverted to 3.5, and the code works.
Before importing nemo.collections, run the following and it should work:
```
!pip uninstall transformers
!pip install transformers==3.5
```


Thanks it worked for that cell, however [here](https://colab.research.google.com/github/NVIDIA/NeMo/blob/v1.0.0b2/tutorials/nlp/Text_Classification_Sentiment_Analysis.ipynb#scrollTo=NgsGLydWo-6-)  I am getting another error, traceback is:
```
Using bos_token, but it is not set yet.
Using eos_token, but it is not set yet.
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-31-543503bf2763> in <module>()
----> 1 model = nemo_nlp.models.TextClassificationModel(cfg=config.model, trainer=trainer)

4 frames
/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/models/text_classification/text_classification_model.py in __init__(self, cfg, trainer)
     58 
     59         # init superclass
---> 60         super().__init__(cfg=cfg, trainer=trainer)
     61 
     62         self.bert_model = get_lm_model(

/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/models/nlp_model.py in __init__(self, cfg, trainer)
     35 
     36     def __init__(self, cfg: DictConfig, trainer: Trainer = None):
---> 37         super().__init__(cfg, trainer)
     38         self.bert_model = None  # Pretrained BERT encoder
     39         self.set_world_size(trainer)

/usr/local/lib/python3.6/dist-packages/nemo/core/classes/modelPT.py in __init__(self, cfg, trainer)
     97         if self._cfg is not None and not self.__is_model_being_restored():
     98             if 'train_ds' in self._cfg and self._cfg.train_ds is not None:
---> 99                 self.setup_training_data(self._cfg.train_ds)
    100 
    101             if 'validation_ds' in self._cfg and self._cfg.validation_ds is not None:

/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/models/text_classification/text_classification_model.py in setup_training_data(self, train_data_config)
    206             self._test_dl = None
    207             return
--> 208         self._train_dl = self._setup_dataloader_from_config(cfg=train_data_config)
    209 
    210     def setup_validation_data(self, val_data_config: Optional[DictConfig]):

/usr/local/lib/python3.6/dist-packages/nemo/collections/nlp/models/text_classification/text_classification_model.py in _setup_dataloader_from_config(self, cfg)
    235                 The label of the example is separated with TAB at the end of each line. \n\
    236                 Each line of the files should follow the format: \n\
--> 237                 [WORD][SPACE][WORD][SPACE][WORD][...][TAB][LABEL]'
    238             )
    239 

FileNotFoundError: DATA_DIR/SST-2/train_nemo_format.tsv not found! The data should be be stored in TAB-separated files 
                "validation_ds.file_path" and "train_ds.file_path" for train and evaluation respectively. 
                Each line of the files contains text sequences, where words are separated with spaces. 
                The label of the example is separated with TAB at the end of each line. 
                Each line of the files should follow the format: 
                [WORD][SPACE][WORD][SPACE][WORD][...][TAB][LABEL]
```

Did you create DATA_DIR/SST-2/train_nemo_format.tsv?

I think we'd better use the new version to solve the problem , not to always use the older version 3.5 tranformers to solve it.

try ”from transformers.models.bert.tokenization_bert import BasicTokenizer“ in new version.