Repository: EleutherAI/lm-evaluation-harness
Issue: how to load a local pruned model
Root cause: Configuration Error

@Mistobaan @richhankins @pminervini @jeffhsu3 @polm i use following manners to save a pruned model:
model.half()
torch.save({
            'model': model, 
            'tokenizer': tokenizer,
        }, logger.best_checkpoint_path)

the local saved model is following:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/96739096/da91b28d-8cfd-442e-b402-cfd6ccead3fb)
so i want to know how to load this model and its tokenizer by using lm-evaluation-harness, in orther words, i want to get where this project load a model。

Assuming you are using a huggingface code base for llama, you can export the model in huggingface format and use the huggingface interface.

Check this notebook for a working example: https://colab.research.google.com/drive/1XS9m9k0-5g5sJz5xQWP9HxyLunDs5yhG

Hi, the code in the notebook only solve the problem when i only want to load the existed model class. If I want to load a pruned model with the model architecture modified, how should i do?

Hi @qxpBlog , it's difficult to be more helpful without more info, but, 

If you have code to load your model, and its interface and tokenizer are compatible with Huggingface AutoModelForCausalLM, you can load the model yourself, and run lm-eval-harness externally using the `simple_evaluate()` or `evaluate()` functions--we allow for `pretrained` and `tokenizer` to be passed as already-initialized HF model objects already to `HFLM`'s initialization.

See https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage for an example of using the library's external API. your script would then look like initializing your model and tokenizer, passing them into the creation of HFLM, and running `simple_evaluate()`.

> Hi @qxpBlog , it's difficult to be more helpful without more info, but,
> 
> If you have code to load your model, and its interface and tokenizer are compatible with Huggingface AutoModelForCausalLM, you can load the model yourself, and run lm-eval-harness externally using the `simple_evaluate()` or `evaluate()` functions--we allow for `pretrained` and `tokenizer` to be passed as already-initialized HF model objects already to `HFLM`'s initialization.
> 
> See https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage for an example of using the library's external API. your script would then look like initializing your model and tokenizer, passing them into the creation of HFLM, and running `simple_evaluate()`.

thanks i have solved this problem 

Hello @haileyschoelkopf ,

I'm have pruned and further finetuned model of Llama-7B. 

Original model contains the 32 layers but my pruned model contains just 11 layers. 

when I run the `simple_evaluate`, I'm using issues. 


```py

tokenizer = AutoTokenizer.from_pretrained(
                 model_name, 
                cache_dir = self._cfg.llm.cache_dir)
 lm_eval_model = LMEvalAdaptor(pruned_llama7b_model, tokenizer)
results = evaluator.simple_evaluate(
            model=lm_eval_model,
            tasks=[dataset_name],
            batch_size=128,
            no_cache=True,
        )
```


```py

# taken from https://github.com/mit-han-lab/offsite-tuning/blob/main/offsite_tuning/eval_harness.py
class LMEvalAdaptor(BaseLM):

    def __init__(self, model, tokenizer, batch_size=1):
        super().__init__()

        assert isinstance(batch_size, int)

        self.model = model
        self.model.eval()

        self.tokenizer = tokenizer

        self.vocab_size = self.tokenizer.vocab_size

        self._batch_size = batch_size
        self._max_length = -1

    @property
    def eot_token_id(self):
        return self.tokenizer.eos_token_id

    # @property
    # def max_length(self):
    #     if hasattr(self.model.config, 'n_ctx'):
    #         return self.model.config.n_ctx
    #     elif hasattr(self.model.config, 'max_position_embeddings'):
    #         return self.model.config.max_position_embeddings
    #     else:
    #         return 2048
        

    @property
    def max_length(self):
        if self._max_length != -1:
            return self._max_length
        if hasattr(self.model.config, 'n_ctx'):
            return self.model.config.n_ctx
        elif hasattr(self.model.config, 'max_position_embeddings'):
            return self.model.config.max_position_embeddings
        elif hasattr(self.model.config, 'n_positions'):
            return self.model.config.n_positions
        # Jerry: need to check these defaults
        # elif 'bloom' in self.model_name:
        #     return 2048
        # elif 'llama' in self.model_name:
        #     return 2048  # TODO: did not check this
        # elif 'mpt' in self.model_name:
        #     return 2048
        # elif 'falcon' in self.model_name:
        #     return 2048
        else:
            print(self.model.config)
            raise NotImplementedError

    @property
    def max_gen_toks(self):
        return 256

    @property
    def batch_size(self):
        return self._batch_size

    @property
    def device(self):
        return "cuda"

    def tok_encode(self, string: str):
        return self.tokenizer.encode(string, add_special_tokens=False)

    def tok_decode(self, tokens):
        return self.tokenizer.decode(tokens)

    def _model_call(self, inps):
        """
        inps: a torch tensor of shape [batch, sequence]
        the size of sequence may vary from call to call
        returns: a torch tensor of shape [batch, sequence, vocab] with the
        logits returned from the model
        """
        with torch.no_grad():
        
            
        
            # out = self.model(inps)[0]
            # return out  # [:, :, :self.tokenizer.vocab_size]

            print(inps)

        
            out = self.model(inps)
            return out.logits  # [:, :, :self.tokenizer.vocab_size]
        
        



    def _model_generate(self, context, max_length, eos_token_id):
        return self.model.generate(
            context,
            max_length=max_length,
            eos_token_id=eos_token_id,
            do_sample=False
        )
```

The errors is as below:

```sh

  File "/home/krishna/2024/FederatedScope/federatedscope/llm/offsite_tuning/server.py", line 282, in eval
    self.llm_evaluate()
  File "/home/krishna/2024/FederatedScope/federatedscope/llm/offsite_tuning/server.py", line 304, in llm_evaluate
    results = evaluator.simple_evaluate(
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/utils.py", line 161, in _wrapper
    return fn(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/evaluator.py", line 86, in simple_evaluate
    results = evaluate(
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/utils.py", line 161, in _wrapper
    return fn(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/evaluator.py", line 247, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/base.py", line 185, in loglikelihood
    return self._loglikelihood_tokens(new_reqs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/base.py", line 295, in _loglikelihood_tokens
    self._model_call(batched_inps), dim=-1
  File "/home/krishna/2024/FederatedScope/federatedscope/llm/trainer/trainer.py", line 141, in _model_call
    out = self.model(inps)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/peft/peft_model.py", line 1083, in forward
    return self.base_model(
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/peft/tuners/tuners_utils.py", line 161, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1183, in forward
    outputs = self.model(
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1070, in forward
    layer_outputs = decoder_layer(
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 798, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 408, in forward
    key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
  File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/cache_utils.py", line 130, in update
    return self.key_cache[layer_idx], self.value_cache[layer_idx]
IndexError: list index out of range
```

Any idea to resolve this issue? Thank you in advance.






Hi @KKNakkav2 , it looks like this is a problem internal to your (HF) model's code.

 I'm not 100% sure how you have created this pruned model earlier in your code, but this error looks like the model's config hasn't updated, and as such it still thinks there are 32 (or at least more than 11) layers in the model.

I'd print `layer_idx` and see what comes out of it, and also try running a `.generate()` and forward pass of your pruned model before wrapping it in that Adapter class.

Thank you @haileyschoelkopf  for the prompt response.

I was able to solve this by setting `model.config.use_cache` to `False`

```sh

# load HF model
model = AutoModelForCausalLM.from_pretrained(
        model_name,
        cache_dir = cache_dir,
        **kwargs)

# adding below line fixed this
model.config.use_cache = False
```

I'm not sure about complete reasons for this error. But somehow, cached model information was the issue (error was coming from `transformers/cache_utils.py` script).

Thanks





Have you fixed it yet?

> 您好，
> 
> 我对 Llama-7B 的模型进行了修剪和进一步微调。
> 
> 原始模型包含 32 层，但我修剪后的模型仅包含 11 层。
> 
> 当我运行 时，我正在使用问题。`simple_evaluate`
> 
> tokenizer = AutoTokenizer.from_pretrained(
>                  model_name, 
>                 cache_dir = self._cfg.llm.cache_dir)
>  lm_eval_model = LMEvalAdaptor(pruned_llama7b_model, tokenizer)
> results = evaluator.simple_evaluate(
>             model=lm_eval_model,
>             tasks=[dataset_name],
>             batch_size=128,
>             no_cache=True,
>         )
> # taken from https://github.com/mit-han-lab/offsite-tuning/blob/main/offsite_tuning/eval_harness.py
> class LMEvalAdaptor(BaseLM):
> 
>     def __init__(self, model, tokenizer, batch_size=1):
>         super().__init__()
> 
>         assert isinstance(batch_size, int)
> 
>         self.model = model
>         self.model.eval()
> 
>         self.tokenizer = tokenizer
> 
>         self.vocab_size = self.tokenizer.vocab_size
> 
>         self._batch_size = batch_size
>         self._max_length = -1
> 
>     @property
>     def eot_token_id(self):
>         return self.tokenizer.eos_token_id
> 
>     # @property
>     # def max_length(self):
>     #     if hasattr(self.model.config, 'n_ctx'):
>     #         return self.model.config.n_ctx
>     #     elif hasattr(self.model.config, 'max_position_embeddings'):
>     #         return self.model.config.max_position_embeddings
>     #     else:
>     #         return 2048
>         
> 
>     @property
>     def max_length(self):
>         if self._max_length != -1:
>             return self._max_length
>         if hasattr(self.model.config, 'n_ctx'):
>             return self.model.config.n_ctx
>         elif hasattr(self.model.config, 'max_position_embeddings'):
>             return self.model.config.max_position_embeddings
>         elif hasattr(self.model.config, 'n_positions'):
>             return self.model.config.n_positions
>         # Jerry: need to check these defaults
>         # elif 'bloom' in self.model_name:
>         #     return 2048
>         # elif 'llama' in self.model_name:
>         #     return 2048  # TODO: did not check this
>         # elif 'mpt' in self.model_name:
>         #     return 2048
>         # elif 'falcon' in self.model_name:
>         #     return 2048
>         else:
>             print(self.model.config)
>             raise NotImplementedError
> 
>     @property
>     def max_gen_toks(self):
>         return 256
> 
>     @property
>     def batch_size(self):
>         return self._batch_size
> 
>     @property
>     def device(self):
>         return "cuda"
> 
>     def tok_encode(self, string: str):
>         return self.tokenizer.encode(string, add_special_tokens=False)
> 
>     def tok_decode(self, tokens):
>         return self.tokenizer.decode(tokens)
> 
>     def _model_call(self, inps):
>         """
>         inps: a torch tensor of shape [batch, sequence]
>         the size of sequence may vary from call to call
>         returns: a torch tensor of shape [batch, sequence, vocab] with the
>         logits returned from the model
>         """
>         with torch.no_grad():
>         
>             
>         
>             # out = self.model(inps)[0]
>             # return out  # [:, :, :self.tokenizer.vocab_size]
> 
>             print(inps)
> 
>         
>             out = self.model(inps)
>             return out.logits  # [:, :, :self.tokenizer.vocab_size]
>         
>         
> 
> 
> 
>     def _model_generate(self, context, max_length, eos_token_id):
>         return self.model.generate(
>             context,
>             max_length=max_length,
>             eos_token_id=eos_token_id,
>             do_sample=False
>         )
> 错误如下：
> 
>   File "/home/krishna/2024/FederatedScope/federatedscope/llm/offsite_tuning/server.py", line 282, in eval
>     self.llm_evaluate()
>   File "/home/krishna/2024/FederatedScope/federatedscope/llm/offsite_tuning/server.py", line 304, in llm_evaluate
>     results = evaluator.simple_evaluate(
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/utils.py", line 161, in _wrapper
>     return fn(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/evaluator.py", line 86, in simple_evaluate
>     results = evaluate(
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/utils.py", line 161, in _wrapper
>     return fn(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/evaluator.py", line 247, in evaluate
>     resps = getattr(lm, reqtype)([req.args for req in reqs])
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/base.py", line 185, in loglikelihood
>     return self._loglikelihood_tokens(new_reqs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/lm_eval/base.py", line 295, in _loglikelihood_tokens
>     self._model_call(batched_inps), dim=-1
>   File "/home/krishna/2024/FederatedScope/federatedscope/llm/trainer/trainer.py", line 141, in _model_call
>     out = self.model(inps)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
>     result = forward_call(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/peft/peft_model.py", line 1083, in forward
>     return self.base_model(
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
>     result = forward_call(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/peft/tuners/tuners_utils.py", line 161, in forward
>     return self.model.forward(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
>     output = module._old_forward(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1183, in forward
>     outputs = self.model(
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
>     result = forward_call(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
>     output = module._old_forward(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1070, in forward
>     layer_outputs = decoder_layer(
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
>     result = forward_call(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
>     output = module._old_forward(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 798, in forward
>     hidden_states, self_attn_weights, present_key_value = self.self_attn(
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
>     result = forward_call(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
>     output = module._old_forward(*args, **kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 408, in forward
>     key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
>   File "/home/krishna/miniconda3/envs/fs-llm/lib/python3.9/site-packages/transformers/cache_utils.py", line 130, in update
>     return self.key_cache[layer_idx], self.value_cache[layer_idx]
> IndexError: list index out of range
> 有什么解决这个问题的想法吗？先谢谢你。

Have you fixed it yet?