Repository: EleutherAI/lm-evaluation-harness
Issue: evaluate model from local machine
Root cause: Documentation Deficiency

hi,

i am wondering if it is possible to evaluate a llm saved locally with this library. the llm is derived from other llms that are found on huggingface & i am saving it with 'save_pretrained'. the tokenizer is also saved locally with 'save_pretrained'. is this possible to do without uploading the model + tokenizer to hugging-face hub?

in the [readme advanced usage section](https://github.com/EleutherAI/lm-evaluation-harness/tree/master#advanced-usage) it states "you can pass a local path via pretrained=". in the evaluator.simple_evaluate() function (on the master branch) i passed in 

model_name="hf-causal", model_args=f"pretrained={local/path/to/model},tokenizer={local/path/to/tokenizer}". 

when i do so, i get the error 'huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': {local/path/to/model} Use `repo_type` argument if needed.'

i already read https://github.com/EleutherAI/lm-evaluation-harness/issues/374 but it didn't solve my issue because the model they use is in hugginface hub.

thanks!

Independent of our codebase, can you load the model using the HF transformers library's AutoModel functionality? If so, what input do you pass to the constructor?

@umarbeknasimov did you get the solution? I am facing the same issue.

> @umarbeknasimov did you get the solution? I am facing the same issue.

Independent of our codebase, can you load the model using the HF transformers library's AutoModel functionality? If so, what input do you pass to the constructor?

Closing because I am unable to replicate this. Note that `{local/path/to/model}` is a placeholder, and so, if you want to load a model stored at `/mnt/c/me/llama-2-7`, you should run with `--model_args pretrained=/mnt/c/me/llama-2-7`. 

From `Repo id must be in the form 'repo_name' or 'namespace/repo_name': {local/path/to/model}` it seems like you may be not replacing this placeholder, similar to #995 . If this is not the case, feel free to reopen this issue. 

sorry for the late reply. it was an error with the directory names but now im struggling to load the model because the tokenizer is missing.

@haileyschoelkopf i know that {local/path/to/model} is a placeholder & i am replacing it with actual path to the saved model.

@StellaAthena i am trying to use the HF AutoModel API to load the model. the exact model is mistral-7b. first, i load the hugginface model with `AutoModelForCausalLM.from_pretrained(model_name)`. then i apply some transformations and the save the new model weights with `save_pretrained(dir)` which saves some files in the dir file. but then i try to load the transformed model weights back with the same `AutoModelForCausalLM.from_pretrained(dir)`, i get the error 

`OSError: Can't load tokenizer for 'Mistral-7B-Avg-v0.1-model'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'Mistral-7B-Avg-v0.1-model' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.`

do you have context on how to add the tokenizer info in the directory? i tried moving the tokenizer files saved from calling 'save_pretrained(dir)` on the AutoTokenizer object but it led to the same error. thank you!