Repository: EleutherAI/lm-evaluation-harness
Issue: [huggingface.py: 1353] Failed to get model SHA for LLamaForCausalLM.....
Root cause: Validation Gap

Hiii, when I tried to run simple_evaluate on my finetuned peft model (codes below), the following bugs keep appeared: `2024-11-11:00:33:22,892 WARNING  [huggingface.py:1353] Failed to get model SHA for LlamaForCausalLM. Could you help me have a look on that? Thanks a lot!
   ###########Model Set Up######################################
   model_path = "meta-llama/Llama-3.2-3B"
   model = AutoModelForCausalLM.from_pretrained(
   model_path,
   device_map="auto",
   torch_dtype=torch.bfloat16
   )

   device = torch.device(f'cuda:{local_rank}')
   peft_model_id = "/home/wangz12/projects/RareDxGPT/checkpoints/RareDxGPT-orpo-3b50/checkpoint-19000"
   
   lora_model = PeftModel.from_pretrained(model,
                                   peft_model_id,
                                   torch_dtype=torch.bfloat16
                                   )
  
   model = lora_model.merge_and_unload()

   lm = HFLM(pretrained=model,
             tokenizer=tokenizer,
             dtype=torch.bfloat16,
             max_length=tokenizer.model_max_length,
             batch_size=args.batch_size,
             trust_remote_code=True,
             )


   task_manager = TaskManager()
   tasks = ["mmlu"]
   result = simple_evaluate(
       model=lm,
       tasks=tasks,
       num_fewshot=5,
       task_manager=task_manager
   )
`
`2024-11-11:00:33:22,892 WARNING  [huggingface.py:1353] Failed to get model SHA for LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 3072)
    (layers): ModuleList(
      (0-27): 28 x LlamaDecoderLayer(
        (self_attn): LlamaSdpaAttention(
          (q_proj): Linear(in_features=3072, out_features=3072, bias=False)
          (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
          (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
          (o_proj): Linear(in_features=3072, out_features=3072, bias=False)
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=3072, out_features=8192, bias=False)
          (up_proj): Linear(in_features=3072, out_features=8192, bias=False)
          (down_proj): Linear(in_features=8192, out_features=3072, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((3072,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((3072,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((3072,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=3072, out_features=128256, bias=False)
) at revision main. Error: Repo id must be a string, not <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>: 'LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 3072)
    (layers): ModuleList(
      (0-27): 28 x LlamaDecoderLayer(
        (self_attn): LlamaSdpaAttention(
          (q_proj): Linear(in_features=3072, out_features=3072, bias=False)
          (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
          (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
          (o_proj): Linear(in_features=3072, out_features=3072, bias=False)
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=3072, out_features=8192, bias=False)
          (up_proj): Linear(in_features=3072, out_features=8192, bias=False)
          (down_proj): Linear(in_features=8192, out_features=3072, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): LlamaRMSNorm((3072,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((3072,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((3072,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=3072, out_features=128256, bias=False)
)'.
fatal: not a git repository (or any parent up to mount point /mnt/isilon)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[rank0]:[W1111 00:33:26.906774961 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())




`

Hi! This is just for logging and reproducibility purposes. Should just raise a warning letting you know it couldn't find a the model hash, but the script should still finish properly

@baberabb Yeaaa! I solved it thanks a lot for your prompting and helpful response!!