Repository: EleutherAI/lm-evaluation-harness
Issue: How to fix the token length of the model input?
Root cause: Documentation Deficiency

When testing the dataset like MMLU , the input token length varies for each inference. How can I fix the token length of the model input? 

You can pass in `max_length` to constrain max sequence length, and it will left truncate any inputs greater than that

i have the same problem
`CUDA_VISIBLE_DEVICES=0 lm_eval --model hf \
    --model_args pretrained=openai-community/gpt2,max_length=1024,dtype="bfloat16"\
    --tasks leaderboard_musr\
    --device cuda:0 \
    --num_fewshot 0\
    --batch_size 8`


and here is the error id does not disapeared. i am using v0.4.3 same forked version as lm-leaderboard

   [evaluator.py:433] Running loglikelihood requests
Token indices sequence length is longer than the specified maximum sequence length for this model (1139 > 1024). Running this sequence through the model will result in indexing errors


> You can pass in `max_length` to constrain max sequence length, and it will left truncate any inputs greater than that

@baberabb well I mean how  to have a fixed input length for each inference, taking 2048 as an example, where the input length is always 2048. When the actual input content is less than 2048, padding is added to reach this length, rather than simply limiting the maximum input length to no more than 2048.


> i have the same problem `CUDA_VISIBLE_DEVICES=0 lm_eval --model hf \ --model_args pretrained=openai-community/gpt2,max_length=1024,dtype="bfloat16"\ --tasks leaderboard_musr\ --device cuda:0 \ --num_fewshot 0\ --batch_size 8`
> 
> and here is the error id does not disapeared. i am using v0.4.3 same forked version as lm-leaderboard
> 
> [evaluator.py:433] Running loglikelihood requests Token indices sequence length is longer than the specified maximum sequence length for this model (1139 > 1024). Running this sequence through the model will result in indexing errors

have you tried set max_length=2048?

> @baberabb well I mean how to have a fixed input length for each inference, taking 2048 as an example, where the input length is always 2048. When the actual input content is less than 2048, padding is added to reach this length, rather than simply limiting the maximum input length to no more than 2048.

yeah, the inputs are padded to the largest sequence length in a batch ([logliklehood](https://github.com/EleutherAI/lm-evaluation-harness/blob/389347ee98ee4c82949083ba9a5f5ef1bb021a6c/lm_eval/models/huggingface.py#L1099) , [generations](https://github.com/EleutherAI/lm-evaluation-harness/blob/389347ee98ee4c82949083ba9a5f5ef1bb021a6c/lm_eval/models/huggingface.py#L1276)), so not fixed. I think the simplest way to handle this would be to modify the [_model_call](https://github.com/EleutherAI/lm-evaluation-harness/blob/389347ee98ee4c82949083ba9a5f5ef1bb021a6c/lm_eval/models/huggingface.py#L808) (add right padding, and trim it from the outputs) and [model_generate](https://github.com/EleutherAI/lm-evaluation-harness/blob/389347ee98ee4c82949083ba9a5f5ef1bb021a6c/lm_eval/models/huggingface.py#L834) (add left padding) methods.

@baberabb Jumping in for a side question. I notice that the seqlen in the model decreases as the evaluation goes along. For example, with batch 64, the first iteration input is (64, 720), second iteration (64, 718) , until the very end it becomes (64, 30). Something like this. Is there any query sorting mechanism (based on seqlen) implemented in lm-eval or huggingface dataset that is used? If you happen to know about this, could you provide a pointer to where this sorting is implemented in the codebase? 

Thanks in advance. 

> @baberabb Jumping in for a side question. I notice that the seqlen in the model decreases as the evaluation goes along. For example, with batch 64, the first iteration input is (64, 720), second iteration (64, 718) , until the very end it becomes (64, 30). Something like this. Is there any query sorting mechanism (based on seqlen) implemented in lm-eval or huggingface dataset that is used? If you happen to know about this, could you provide a pointer to where this sorting is implemented in the codebase?
> 
> Thanks in advance.

Hi! yeah, the sequences are [sorted](https://github.com/EleutherAI/lm-evaluation-harness/blob/d4ae963510ab2f7c0eb38c8f2914a383a06c4e1b/lm_eval/models/huggingface.py#L990) by token length (using [this](https://github.com/EleutherAI/lm-evaluation-harness/blob/d4ae963510ab2f7c0eb38c8f2914a383a06c4e1b/lm_eval/models/utils.py#L352) utility function).