Repository: EleutherAI/lm-evaluation-harness
Issue: Hang when evaluate nemo model with pipeline model parallel
Root cause: Interface Contract Mismatch

I am working on evaluating NeMo Model with lm_eval and my commands are as follows:
```bash
export CUDA_VISIBLE_DEVICES="0,1,2,3"
torchrun --nproc-per-node=4 --master-port=23333 --no-python lm_eval --model nemo_lm \
        --tasks $task \
        --model_args path=./llama-3.2-1b-instruct-nemo,devices=4,pipeline_model_parallel_size=4 \
        --batch_size 1
```
And it reports errors like:
```
[rank1]: Traceback (most recent call last):
[rank1]:   File "/usr/local/bin/lm_eval", line 8, in <module>
[rank1]:     sys.exit(cli_evaluate())
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 375, in cli_evaluate
[rank1]:     results = evaluator.simple_evaluate(
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 395, in _wrapper
[rank1]:     return fn(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 279, in simple_evaluate
[rank1]:     results = evaluate(
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 395, in _wrapper
[rank1]:     return fn(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 451, in evaluate
[rank1]:     resps = getattr(lm, reqtype)(cloned_reqs)
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/nemo_lm.py", line 380, in loglikelihood_rolling
[rank1]:     string_nll = self._loglikelihood_tokens(
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/nemo_lm.py", line 432, in _loglikelihood_tokens
[rank1]:     batch_token_ids = np.asarray(output["token_ids"])[:, :-1]
[rank1]: TypeError: 'NoneType' object is not subscriptable
[rank2]: Traceback (most recent call last):
[rank2]:   File "/usr/local/bin/lm_eval", line 8, in <module>
[rank2]:     sys.exit(cli_evaluate())
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 375, in cli_evaluate
[rank2]:     results = evaluator.simple_evaluate(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 395, in _wrapper
[rank2]:     return fn(*args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 279, in simple_evaluate
[rank2]:     results = evaluate(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 395, in _wrapper
[rank2]:     return fn(*args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 451, in evaluate
[rank2]:     resps = getattr(lm, reqtype)(cloned_reqs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/nemo_lm.py", line 380, in loglikelihood_rolling
[rank2]:     string_nll = self._loglikelihood_tokens(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/nemo_lm.py", line 432, in _loglikelihood_tokens
[rank2]:     batch_token_ids = np.asarray(output["token_ids"])[:, :-1]
[rank2]: TypeError: 'NoneType' object is not subscriptable

```

And when I used pipeline parallel word size=2 by runing the following scripts:
```bash
export CUDA_VISIBLE_DEVICES="0,1"
torchrun --nproc-per-node=2 --master-port=23333 --no-python lm_eval --model nemo_lm \
        --tasks $task \
        --model_args path=./llama-3.2-1b-instruct-nemo,devices=2,pipeline_model_parallel_size=2 \
        --batch_size 1
```
It will hang at the follwing line at [NeMo/nemo/collections/nlp/modules/common/text_generation_utils.py](https://github.com/NVIDIA/NeMo/blob/109562ba8bfd1d3a9abdf64a5446077b488ea4c5/nemo/collections/nlp/modules/common/text_generation_utils.py#L847)
```python
...
    if output is not None:
        decode_tokens, output_logits, full_logits = output
        resp_sentences = []
        resp_sentences_seg = []

        # Hang at this line.
        decode_tokens = decode_tokens.cpu().numpy().tolist()
        for decode_token in decode_tokens:
            sentence = tokenizer.ids_to_text(decode_token)
            resp_sentences.append(sentence)
            if not isinstance(tokenizer, TabularTokenizer):
                words = []
                for token in decode_token:
...
```

I used the nvidia official  NeMo docker: nvcr.io/nvidia/nemo:24.07, and list my lm_eval and nemo version:
```
lm_eval                       0.4.3
nemo_aligner                  0.4.0rc0                   /opt/NeMo-Aligner
nemo_curator                  0.4.0                      /opt/NeMo-Curator
nemo_text_processing          1.0.2
nemo_toolkit                  2.0.0rc1                   /opt/NeMo

```