Repository: EleutherAI/lm-evaluation-harness
Issue: error with .nemo model. invalid strategy name: `strategy=<nemo.collections.nlp.parts.nlp_overrides.NLPDDPStrategy object at 0x7fd7b9f79f60>`. It must be either a string or an instance of `lightning.pytorch.strategies.Strategy`
Root cause: External Dependency Breakage

torchrun --nproc-per-node=4 --no-python lm_eval \
    --model nemo_lm \
    --model_args path=/projects/llm/models/weight/Llama-3.1-8B.nemo,devices=4,tensor_model_parallel_size=2,pipeline_model_parallel_size=2 \
    --tasks ifeval \
    --batch_size 32



ValueError: You selected an invalid strategy name: `strategy=<nemo.collections.nlp.parts.nlp_overrides.NLPDDPStrategy object at 0x7fd7b9f79f60>`. It must be either a string or an instance of `lightning.pytorch.strategies.Strategy`. Example choices: auto, ddp, ddp_spawn, deepspeed, ... Find a complete list of options in our documentation at https://lightning.ai

Explanation with References:

The issue stems from a difference between pytorch_lightning (commonly used by NeMo during training) and the newer lightning.pytorch module (used in lm-evaluation-harness if Lightning 2.x is installed). While both provide a Trainer interface, their internal handling of strategy= differs in Lightning 2.x.

In Lightning 2.x, the Trainer constructor invokes _parse_strategy() internally to validate the strategy argument:

🔗 Source: [lightning.pytorch.trainer.connectors.accelerator_connector.py#L193](https://github.com/Lightning-AI/lightning/blob/master/src/lightning/pytorch/trainer/connectors/accelerator_connector.py#L193)
This function explicitly rejects any custom Strategy whose __module__ does not start with "lightning.pytorch":

if isinstance(strategy, Strategy):
    if not strategy.__module__.startswith("lightning.pytorch"):
        raise ValueError(...)
As a result, NeMo's NLPDDPStrategy—defined under nemo.collections.nlp.parts.nlp_overrides—is rejected by Lightning 2.x, even if it is a valid subclass of DDPStrategy.

However, in NeMo's own training flow, the strategy is typically instantiated before being passed into the Trainer (e.g., via MegatronTrainerBuilder.create_trainer()), and is used with pytorch_lightning (Lightning 1.x) or under execution contexts that bypass _parse_strategy()—for example, interactive sessions or CLI scripts that don't go through torchrun.

This explains why the same strategy works during NeMo training, but fails during lm-eval evaluation under torchrun, where lightning.pytorch.Trainer enforces stricter validation.

Suggested Fix:

To ensure compatibility, we recommend conditionally falling back to "ddp" or "auto" in evaluation contexts where Trainer(strategy=NLPDDPStrategy()) may fail. Alternatively, one can use pytorch_lightning.Trainer instead of lightning.pytorch.Trainer if Lightning 1.x behavior is desired.

Fix line: [from lightning.pytorch.trainer.trainer import Trainer#L193](https://github.com/EleutherAI/lm-evaluation-harness/blob/fc5019ead53c45119c522c62e8eea2daa837c56e/lm_eval/models/nemo_lm.py#L193)

References:

⚙️ Strategy Validation Logic:
https://github.com/Lightning-AI/lightning/blob/master/src/lightning/pytorch/trainer/connectors/accelerator_connector.py#L193
⚠️ Related Lightning issue on strategy validation:
https://github.com/Lightning-AI/lightning/issues/17668
🧪 NeMo strategy location (example):
https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/parts/nlp_overrides.py