TUTORIAL BUG ONLY: Missing model.eval() in the SapBERT implementations
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

TUTORIAL BUG ONLY: Missing model.eval() in the SapBERT implementations.

There is a mistake causing inconsistent results in the NLP tutorials for at least two notebooks:
- In stable branch Nemo/tutorials/nlp/Entity_Linking_Medical.ipynb
- In v1.0.2 branch Nemo/tutorials/nlp/Entity_Linking_Medical.ipynb, which is referred by NeMo official tutorial https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/entity_linking.html and SapBERT author in https://github.com/cambridgeltl/sapbert?tab=readme-ov-file

and should also be in the stable branch NeMo/examples/nlp/entity_linking in every .py file who use *model.forward(...)* function.

After the training part the model is not set to eval mode and thus the Dropout remains active.
This could be quite misleading for anyone who whishes to implement those methods.

**Steps/Code to reproduce bug**

1) Run the tutorial code for the aforementionned notebooks in Colab for instance (any setting).
2) Run the evaluation part twice: you get different results.

**Wrong results printed in Notebook**

Once run the block Model Evaluation, will see: 
| Model| 1|5|
| ------ | ------ | ------ |
| BERT Base Baseline | 0.700000 |1.000000|
| BERT + SAP | 0.800000 |1.000000|

Run this block at twice, will see: 

| Model| 1|5|
| ------ | ------ | ------ |
| BERT Base Baseline | 0.700000 |1.000000|
| BERT + SAP | 0.700000 |0.800000|

The second line can be any other proportion possible.

In Entity Linking via Nearest Neighbor Search,

The similar score of SapBERT and most similar concepts is always changing even if the model is always the same one.

**Expected behavior**

The evaluation of the model should give consistent results.

**One solution suggestion**

Set model mode to *eval* each time at evaluation and restore the initial mode at the end of evaluation.

Replace the function *get_embeddings(model, dataloader)* by:

```
# Helper function to get data embeddings
def get_embeddings(model, dataloader):
    mode = model.training
    embeddings, cids = [], []
    with torch.no_grad():
        model.eval()
        for batch in tqdm(dataloader):
            input_ids, token_type_ids, attention_mask, batch_cids = batch
            batch_embeddings = model.forward(input_ids=input_ids.to(device),
                                             token_type_ids=token_type_ids.to(device),
                                             attention_mask=attention_mask.to(device))

            # Accumulate index embeddings and their corresponding IDs
            embeddings.extend(batch_embeddings.cpu().detach().numpy())
            cids.extend(batch_cids)
    # set mode back to its original value
    model.train(mode=mode)

    return embeddings, cids
``` 
It resolve the problem and the output of SapBERT is fixed. 

**Additional comment**

We think this bug always exist as this project is created. And the author has observed this mistaking result. But he wrote
````
This evaluation set contains very little data, and no serious conclusions should be drawn about model performance. Top 1 accuracy should be between 0.7 and 1.0 for both models and top 5 accuracy should be between 0.8 and 1.0. 
````
in the section of Model Evaluation.

Actually, after done the debug, the result (on our side) is

| Model| 1|5|
| ------ | ------ | ------ |
| BERT Base Baseline | 0.700000 |1.000000|
| BERT + SAP | 0.900000 |1.000000|

BERT + SAP acts always better than BERT Base Baseline even thought with the tiny example dataset.


**Environment overview **

 - Environment location: Colab
 - Method of NeMo install: pip install
```
BRANCH = 'r1.22.0'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]
```


**Environment details**

- OS version: Linux-6.1.58+-x86_64-with-glibc2.35
- PyTorch version: 2.1.0+cu121
- Python version: 3.10.12


**Additional context**

Add any other context about the problem here.
Example: GPU model

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.