Repository: huggingface/lighteval
Issue: [EVAL]
Root cause: External Dependency Breakage

## Evaluation short description

I've run nanotron script for training model with [my own fineweb dataset](https://huggingface.co/datasets/barney49/data-refine)
so I only got this result, model checkpoints
![image](https://github.com/user-attachments/assets/ae8570ca-ced4-47bf-a643-d488d2501ed7)
I'd like to evaluate this using lighteval
Please help this problem.
I've tried with this command 
```
accelerate launch --multi_gpu --num_processes=8 -m \
    lighteval accelerate \
    --model_args "pretrained=./checkpoints/10" \
    --tasks "lighteval|truthfulqa:mc|0|0" \
    --override_batch_size 1 \
    --output_dir="./evals/"
```
but how to use checkpoints for evaluating.
Python version: 3.10.12

Hi!
If you're using a nanotron model, you should not launch lighteval with accelerate, but you should instead use nanotron, with `lighteval nanotron` (You can do `lighteval nanotron --help` to get the arguments to use). 

Careful, most of the nanotron configuration should be provided with configuration files, you can look for examples [here](https://github.com/huggingface/lighteval/blob/main/examples/nanotron/lighteval_config_override_template.yaml))

hello @clefourrier 
Thank you for your answer.
I've tried this command `lighteval nanotron --checkpoint-config-path './checkpoint/10/config.yaml' --lighteval-override './examples/nanotron/lighteval_config_overrride_template.yaml'`
But getting this error
```
  File "/root/ben/lighteval/.venv/bin/lighteval", line 5, in <module>
    from lighteval.__main__ import cli_evaluate
  File "/root/ben/lighteval/src/lighteval/__main__.py", line 31, in <module>
    from lighteval.tasks.registry import Registry, taskinfo_selector
  File "/root/ben/lighteval/src/lighteval/tasks/registry.py", line 34, in <module>
    import lighteval.tasks.default_tasks as default_tasks
  File "/root/ben/lighteval/src/lighteval/tasks/default_tasks.py", line 22, in <module>
    import lighteval.tasks.tasks_prompt_formatting as prompt
  File "/root/ben/lighteval/src/lighteval/tasks/tasks_prompt_formatting.py", line 32, in <module>
    from lighteval.logging.hierarchical_logger import hlog_warn
  File "/root/ben/lighteval/src/lighteval/logging/hierarchical_logger.py", line 33, in <module>
    from nanotron.logging import get_logger
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/logging.py", line 38, in <module>
    from nanotron.config.config import LoggingArgs
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/__init__.py", line 2, in <module>
    from nanotron.config.config import *
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/config.py", line 13, in <module>
    from nanotron.config.lighteval_config import LightEvalConfig
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/lighteval_config.py", line 5, in <module>
    from nanotron.config.parallelism_config import ParallelismArgs
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/parallelism_config.py", line 4, in <module>
    from nanotron.config.utils_config import (
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/utils_config.py", line 8, in <module>
    from nanotron.parallel.pipeline_parallel.engine import (
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/parallel/pipeline_parallel/__init__.py", line 1, in <module>
    from nanotron.parallel.pipeline_parallel.engine import PipelineEngine
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/parallel/pipeline_parallel/engine.py", line 8, in <module>
    from nanotron.logging import log_rank
ImportError: cannot import name 'log_rank' from partially initialized module 'nanotron.logging' (most likely due to a circular import) (/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/logging.py)
```

I've tried to install by `pip install -e .`
but after that I am getting this error
```
Traceback (most recent call last):
  File "/root/ben/lighteval/.venv/bin/lighteval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/root/ben/lighteval/src/lighteval/__main__.py", line 61, in cli_evaluate
    from lighteval.main_nanotron import main as main_nanotron
  File "/root/ben/lighteval/src/lighteval/main_nanotron.py", line 35, in <module>
    from lighteval.models.nanotron_model import NanotronLightevalModel
  File "/root/ben/lighteval/src/lighteval/models/nanotron_model.py", line 32, in <module>
    from nanotron import distributed as dist
ModuleNotFoundError: No module named 'nanotron'
```
If I install nanotron by `pip install '.[nanotron]',` got above error again `ImportError: cannot import name 'log_rank' from partially initialized module 'nanotron.logging'`


Interesting, thanks for the report, we'll investigate this!

Thank you
Is there maybe some python version problem?

No, I think we might have added a small regression on the nanotron models. @zzhhjj has been working on adding unit tests for nanotron models in this PR #160, but since we don't have them yet we might be breaking things from time to time (we tend to focus on the accelerate models when testing, since we thought nanotron was only used internally, and internally people can fix the codebase easily).

Undertand, Thank you
So, which solution is good for me to evaluate nanotron model checkpoints?

Hi ! It seems to be an issue with the `nanotron` library. To replicate the issue, simply open a python interpreter and run `from nanotron.logging import get_logger`. We invite you to move this issue to the `nanotron` repo, closing it here, but feel free to open another one if you meet any issues with the lib :)

hello @NathanHB 
thank you for your message
if so, can I solve this problem using old lighteval version?


I am in huggingface discord community.
if available, can we solve problem in discord?

Hi @barney49 , no, you would need to use an old nanotron version, but I'm not sure which. 
As @NathanHB indicated, you'll need to open an issue on the nanotron repository, as we use github at HF to track problems and make sure no information is lost.

understand
I've tried using old nanotron version, maybe 2.0. There is 0.1, 0.2, 0.4
but it is also not working
I'd like to know about old lighteval version



Also I've opend issue on nanotron, but no response from team.

hello @clefourrier 
how are you
I think I solved problem by installing nanotron from githubrepo
but after done that, I am getting another error
```
WARNING:lighteval.logging.hierarchical_logger:main: (0, './checkpoints/10/config.yaml'), (1, 'template.yaml'), (2, None),  {
ef
WARNING:lighteval.logging.hierarchical_logger:  get config {
skip_unused_config_keys set
Skip_null_keys set
WARNING:lighteval.logging.hierarchical_logger:  } [0:00:00.019574]
WARNING:lighteval.logging.hierarchical_logger:} [0:00:00.029918]
Traceback (most recent call last):
  File "/root/ben/lighteval/.venv/bin/lighteval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/root/ben/lighteval/src/lighteval/__main__.py", line 63, in cli_evaluate
    main_nanotron(args.checkpoint_config_path, args.lighteval_override, args.cache_dir)
  File "/root/ben/lighteval/src/lighteval/logging/hierarchical_logger.py", line 175, in wrapper
    return fn(*args, **kwargs)
  File "/root/ben/lighteval/src/lighteval/main_nanotron.py", line 89, in main
    lighteval_config: config_cls = get_config_from_file(lighteval_config_path, config_class=LightEvalConfig)
  File "/root/ben/lighteval/.venv/src/nanotron/src/nanotron/config/config.py", line 463, in get_config_from_file
    config = get_config_from_dict(
  File "/root/ben/lighteval/.venv/src/nanotron/src/nanotron/config/config.py", line 424, in get_config_from_dict
    return from_dict(
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/dacite/core.py", line 58, in from_dict
    raise UnexpectedDataError(keys=extra_fields)
dacite.exceptions.UnexpectedDataError: can not match "lighteval" to any data class field
```
I am using this command `lighteval nanotron --checkpoint-config-path './checkpoints/10/config.yaml' --lighteval-override 'template.yaml'`
here is all config file
``` 
"config.yaml"
checkpoints:
  checkpoint_interval: 10
  checkpoints_path: checkpoints
  checkpoints_path_is_shared_file_system: false
  resume_checkpoint_path: null
  save_final_state: false
  save_initial_state: false
data_stages:
- data:
    dataset:
      dataset_overwrite_cache: false
      dataset_processing_num_proc_per_process: 1
      hf_dataset_config_name: null
      hf_dataset_or_datasets: barney49/data-refine
      hf_dataset_splits: train
      text_column_name: text
    num_loading_workers: 1
    seed: 42
  name: Stable Training Stage
  start_training_step: 1
- data:
    dataset:
      dataset_overwrite_cache: false
      dataset_processing_num_proc_per_process: 1
      hf_dataset_config_name: null
      hf_dataset_or_datasets: barney49/data-refine
      hf_dataset_splits: train
      text_column_name: text
    num_loading_workers: 1
    seed: 42
  name: Annealing Phase
  start_training_step: 10
general:
  benchmark_csv_path: null
  consumed_train_samples: 80
  ignore_sanity_checks: true
  project: debug
  run: tiny_llama_%date_%jobid
  seed: 42
  step: 10
lighteval: null
logging:
  iteration_step_info_interval: 1
  log_level: info
  log_level_replica: info
model:
  ddp_bucket_cap_mb: 25
  dtype: bfloat16
  init_method:
    std: 0.02
  make_vocab_size_divisible_by: 1
  model_config:
    bos_token_id: 1
    eos_token_id: 2
    hidden_act: silu
    hidden_size: 2048
    initializer_range: 0.02
    intermediate_size: 8192
    is_llama_config: true
    max_position_embeddings: 2048
    num_attention_heads: 32
    num_hidden_layers: 24
    num_key_value_heads: 32
    pad_token_id: null
    pretraining_tp: 1
    rms_norm_eps: 1.0e-05
    rope_scaling: null
    rope_theta: 10000.0
    tie_word_embeddings: true
    use_cache: true
    vocab_size: 50272
optimizer:
  accumulate_grad_in_fp32: true
  clip_grad: 1.0
  learning_rate_scheduler:
    learning_rate: 0.0003
    lr_decay_starting_step: null
    lr_decay_steps: 13
    lr_decay_style: cosine
    lr_warmup_steps: 2
    lr_warmup_style: linear
    min_decay_lr: 1.0e-05
  optimizer_factory:
    adam_beta1: 0.9
    adam_beta2: 0.95
    adam_eps: 1.0e-08
    name: adamW
    torch_adam_is_fused: true
  weight_decay: 0.01
  zero_stage: 0
parallelism:
  dp: 1
  expert_parallel_size: 1
  pp: 1
  pp_engine: 1f1b
  recompute_layer: false
  tp: 1
  tp_linear_async_communication: true
  tp_mode: REDUCE_SCATTER
  tp_recompute_allgather: true
profiler: null
tokenizer:
  tokenizer_max_length: null
  tokenizer_name_or_path: gpt2
  tokenizer_revision: null
tokens:
  batch_accumulation_per_replica: 4
  limit_test_batches: 0
  limit_val_batches: 0
  micro_batch_size: 2
  sequence_length: 2048
  train_steps: 15
  val_check_interval: 100
```
``` 
"template.yaml"
lighteval:
  batch_size: 16
  checkpoints_path: null
  generation: null
  logging:
    hub_repo_details: null
    hub_repo_results: null
    hub_repo_tensorboard: HuggingFaceBR4/thomwolf-webdata-std-two
    local_output_path: /scratch/thomwolf/lighteval/webdata-std-two-1p82G-wet_files_1-seed-5-698496
    push_details_to_hub: false
    push_results_to_hub: false
    push_results_to_tensorboard: true
    tensorboard_metric_prefix: e
  parallelism:
    dp: 1
    pp: 1
    pp_engine: 1f1b
    recompute_granularity: null
    tp: 1
    tp_linear_async_communication: false
    tp_mode: ALL_REDUCE
  tasks:
    custom_tasks: /fsx/thomwolf/github/lighteval/tasks_examples/custom_tasks/custom_evaluation_tasks.py
    dataset_loading_processes: 8
    max_samples: 10
    multichoice_continuations_start_space: null
    num_fewshot_seeds: null
    tasks: early-signal
    # tasks: custom|hellaswag|0
```
if possible, could you help this problem?


hello @clefourrier 
did you have a good weekend?

could you provide sample checkpoints config.yaml file for fineweb?