Repository: EleutherAI/lm-evaluation-harness
Issue: RuntimeError: CUDA error: device-side assert triggered
Root cause: Validation Gap

Hi,

I got an error "RuntimeError: CUDA error: device-side assert triggered" when I was running multiple GPUs, 

my command is as follows:

python -m lm_eval --model hf \
    --tasks xxxx \
    --model_args pretrained=/mainfs/scratch/jm4n21/Meta-Llama-3-70B-Instruct/,tokenizer=/mainfs/scratch/jm4n21/Meta-Llama-3-70B-Instruct/,parallelize=True,load_in_4bit=True \
    --num_fewshot 0 \
    --write_out \
    --log_samples \
    --apply_chat_template \
    --output_path xxxxx
 

My error is as follows:

bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [31,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1720538643151/work/aten/src/ATen/native/cuda/Indexing.cu:1231: indexSelectSmallIndex: block: [33,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
    cli_evaluate()
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/evaluator.py", line 296, in simple_evaluate
    results = evaluate(
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/evaluator.py", line 468, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/models/huggingface.py", line 1276, in generate_until
    cont = self._model_generate(
  File "/mainfs/scratch/jm4n21/lm-evaluation-harness/lm_eval/models/huggingface.py", line 848, in _model_generate
    return self.model.generate(
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/transformers/generation/utils.py", line 2024, in generate
    result = self._sample(
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/transformers/generation/utils.py", line 2982, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1189, in forward
    outputs = self.model(
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 971, in forward
    causal_mask = self._update_causal_mask(
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1065, in _update_causal_mask
    if AttentionMaskConverter._ignore_causal_mask_sdpa(
  File "/home/jm4n21/.conda/envs/lm-eval/lib/python3.9/site-packages/transformers/modeling_attn_mask_utils.py", line 279, in _ignore_causal_mask_sdpa
    elif (is_training or not is_tracing) and torch.all(attention_mask == 1):
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


It might link to https://github.com/huggingface/trl/issues/1821#event-13448727256, but I was wondering if any solution for this.

Many thanks ahead

I used a single GPU, but I encountered exactly the same error.

yes,i have the same error use the command:

m_eval --model hf     --model_args pretrained=dc3.5-7b-32k     --tasks mmlu     --device cuda:4     --batch_size 8  --num_fewshot 5  --output_path ./eval_out --use_cache ./eval_cache_mmlu

I got the same error. Before, I encountered OOM error, so I reduced the batch size by half.

When I ran my code again, I got:
RuntimeError: GET was unable to find an engine to execute this computation
../aten/src/ATen/native/cuda/Indexing.cu:1284: indexSelectLargeIndex: block: [226,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
... (and so on)

So, I reduced the batch size again (now it is batch_size=16). When I ran my code, that's when I got:
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

../aten/src/ATen/native/cuda/Indexing.cu:1284: indexSelectLargeIndex: block: [226,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
... (and so on)

It's weird because I'm using the smallest GPT model (12 layers) with some minor modifications in the activation functions.

Maybe it is related to the task? I have multiple tasks, and the evaluation stops at the same percentage.

> I got the same error. Before, I encountered OOM error, so I reduced the batch size by half.
> 
> When I ran my code again, I got: RuntimeError: GET was unable to find an engine to execute this computation ../aten/src/ATen/native/cuda/Indexing.cu:1284: indexSelectLargeIndex: block: [226,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ... (and so on)
> 
> So, I reduced the batch size again (now it is batch_size=16). When I ran my code, that's when I got: RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
> 
> ../aten/src/ATen/native/cuda/Indexing.cu:1284: indexSelectLargeIndex: block: [226,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed. ... (and so on)
> 
> It's weird because I'm using the smallest GPT model (12 layers) with some minor modifications in the activation functions.
> 
> Maybe it is related to the task? I have multiple tasks, and the evaluation stops at the same percentage.

maybe not? cause I'm running my own task : (

These errors occur randomly. Sometimes they happen, and at other times, the same settings work without issue, only to fail again later. I suspect there are deeper conflicts that I am unaware of. If anyone has any suggestions, please let me know : )

> These errors occur randomly. Sometimes they happen, and at other times, the same settings work without issue, only to fail again later. I suspect there are deeper conflicts that I am unaware of. If anyone has any suggestions, please let me know : )

I inspected the tasks provided in the lm-eval. The ones with long input sequences raised an error.

I fixed mine by setting the input tokens equal to the block size. So, my code truncates or pads the input. Mine is a custom model/code, so I have different _generate_until_, _loglikelihood_, and _loglikelihood_rolling_.

Since your case involves finetuning a pretrained model to a custom task, you might want to double-check the task's input length.

Hope it helps!