Repository: open-compass/opencompass
Issue: [Bug] LiveCodeBench评测报错
Root cause: Resource Mishandling

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A40',
 'MMEngine': '0.10.4',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.66',
 'OpenCV': '4.10.0',
 'PyTorch': '2.5.1+cu124',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.4\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.4, '
                              'CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
                              'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
                              'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0]',
 'TorchVision': '0.20.1+cu124',
 'lmdeploy': '0.7.1',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.4.1+2176224',
 'sys.platform': 'linux',
 'transformers': '4.49.0'}

### 重现问题 - 代码/配置示例

python3 /nas/xz/opencompass/run.py --hf-path /nas/xz/model_para/Qwen2.5-7B  --hf-type base --datasets livecodebench_gen --hf-num-gpus 8

### 重现问题 - 命令或脚本

评估过程报错，lcb_code_execution这一项只有1.46分

### 重现问题 - 错误信息

cat /nas/xz/LLaMA-Factory/saves/oc/Qwen2.5-7B-livecodebench_gen.log

INFO 03-10 20:14:29 [__init__.py:256] Automatically detected platform cuda.
03/10 20:14:31 - OpenCompass - INFO - Loading livecodebench_gen: /nas/xz/opencompass/opencompass/configs/./datasets/livecodebench/livecodebench_gen.py
03/10 20:14:31 - OpenCompass - INFO - Loading example: /nas/xz/opencompass/opencompass/configs/./summarizers/example.py
03/10 20:14:31 - OpenCompass - INFO - Current exp folder: outputs/default/20250310_201431
03/10 20:14:31 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
03/10 20:14:31 - OpenCompass - INFO - ./data/code_generation_lite does not exist!Start Download data automatically!If you have downloaded the data before,You can specific `COMPASS_DATA_CACHE` to avoid downloading~
Downloading http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/code_generation_lite.zip to /root/.cache/opencompass/data/code_generation_lite.zip
3.2/3.2 GB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00
Extracting /root/.cache/opencompass/data/code_generation_lite.zip to /root/.cache/opencompass/data
lcb_code_generation test 400
lcb_code_generation train 400
03/10 20:20:18 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/execution-v2
lcb_code_execution test 479
lcb_code_execution train 479
03/10 20:20:18 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/test_generation
lcb_test_output test 442
lcb_test_output train 442
03/10 20:20:18 - OpenCompass - INFO - Partitioned into 1 tasks.
03/10 20:20:21 - OpenCompass - WARNING - Debug mode, log will be saved to tmp/3399630_debug.log
03/10 23:05:49 - OpenCompass - INFO - Partitioned into 3 tasks.
03/10 23:05:50 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/code_generation_lite
lcb_code_generation test 400
lcb_code_generation train 400
03/10 23:07:00 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/code_generation_lite
03/10 23:07:04 - OpenCompass - INFO - LCBCodeGeneration: Evaluating 400...
 10%|▉         | 38/400 [00:08<01:12,  5.03it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'makeTheIntegerZero'"), <traceback object at 0x7f4b5be969c0>)
 12%|█▏        | 49/400 [00:12<02:42,  2.16it/s]alarm went off
 13%|█▎        | 53/400 [00:14<02:07,  2.71it/s]alarm went off
 18%|█▊        | 70/400 [00:21<02:01,  2.72it/s]alarm went off
 22%|██▎       | 90/400 [00:29<03:01,  1.71it/s]alarm went off
 24%|██▎       | 94/400 [00:35<05:00,  1.02it/s]alarm went off
 25%|██▌       | 101/400 [00:39<02:31,  1.98it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'minLengthAfterRemovals'"), <traceback object at 0x7f4b5be969c0>)
 33%|███▎      | 132/400 [00:49<01:23,  3.21it/s]alarm went off
 38%|███▊      | 154/400 [00:58<01:34,  2.59it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'minimumOperationsToMakeEqual'"), <traceback object at 0x7f499473e8c0>)
 40%|███▉      | 158/400 [00:59<01:30,  2.67it/s]Fatal Python error: Segmentation fault

Current thread 0x00007f4b5c6bf740 (most recent call first):
  File "<string>", line 39 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  File "<string>", line 44 in count_numbers
  ...

Extension modules: mkl._mklinit, mkl._py_mkl_service, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, yaml._yaml, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, _brotli, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.tslib, pandas._libs.lib, pandas._libs.hashing, pyarrow.lib, pandas._libs.ops, pyarrow._compute, pandas._libs.arrays, pandas._libs.index, pandas._libs.join, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.internals, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.tslibs.strptime, pandas._libs.groupby, pandas._libs.testing, pandas._libs.parsers, pandas._libs.json, regex._regex, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, xxhash._xxhash, pyarrow._json, markupsafe._speedups, PIL._imaging, gmpy2.gmpy2, PIL._imagingft, msgspec._core, zmq.backend.cython._zmq, msgpack._cmsgpack, google._upb._message, setproctitle, uvloop.loop, ray._raylet, sklearn.feature_extraction._hashing_fast, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, sklearn.datasets._svmlight_format_fast, websockets.speedups, tree_sitter._binding, tree_sitter_languages.core, rapidfuzz._feature_detector_cpp, rapidfuzz.distance._initialize_cpp, rapidfuzz.distance.metrics_cpp_avx2, rapidfuzz.fuzz_cpp_avx2, rapidfuzz.process_cpp_impl, rapidfuzz.utils_cpp, Levenshtein.levenshtein_cpp, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5o, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5l, h5py._selector (total: 256)
 40%|████      | 161/400 [01:00<01:19,  3.00it/s]alarm went off
 44%|████▍     | 177/400 [01:07<01:20,  2.77it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'maximumLength'"), <traceback object at 0x7f4b5be969c0>)
 46%|████▌     | 182/400 [01:09<01:17,  2.83it/s]alarm went off
 50%|█████     | 202/400 [01:14<00:54,  3.61it/s]alarm went off
100%|██████████| 400/400 [01:58<00:00,  3.37it/s]
03/10 23:09:03 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_code_generation]: {'extracted_predictions': {0: ['# YOUR CODE HERE'], 1: ['# YOUR CODE HERE'], 2: ['# YOUR CODE HERE'], 3: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n    # Read the length of the string and the number of characters to be deleted\n    n, k = map(int, input().split())\n    \n    # Read the string\n    s = input()\n    \n    # Count the frequency of each character in the string\n    freq = [0] * 26\n    for char in s:\n        freq[ord(char) - ord(\'a\')] += 1\n    \n    # Count the number of characters with odd frequency\n    odd_count = sum(1 for count in freq if count % 2 != 0)\n    \n    # Check if it is possible to remove exactly k characters to form a palindrome\n    if odd_count <= k + 1:\n        print("YES")\n    else:\n        print("NO")'], 4: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n    # Read the size of the array and the number k\n    n, k = map(int, input().split())\n    \n    # Read the array of integers\n    a = list(map(int, input().split()))\n    \n    # Initialize the count of operations\n    operations = 0\n    \n    # Iterate through the array\n    for i in range(n):\n        # If the current element is not divisible by k, increment the count of operations\n        if a[i] % k != 0:\n            operations += 1\n    \n    # Print the minimum number of operations needed\n    print(operations)'], 5: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n    # Read the integer n\n    n = int(input())\n    \n    # Check if n is divisible by 3\n    if n % 3 == 0:\n        # If n is divisible by 3, Vanya wins\n        print("First")\n    else:\n        # If n is not divisible by 3, Vova wins\n        print("Second")'], 6: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n    # Read the number of boxes\n    n = int(input())\n    \n    # Read the weights of the boxes\n    a = list(map(int, input().split()))\n    \n    # Calculate the total weight of all boxes\n    total_weight = sum(a)\n    \n    # Calculate the maximum absolute difference between the total weights of any two trucks\n    max_diff = 0\n    for k in range(1, n + 1):\n        # Calculate the weight of each truck\n        truck_weight = total_weight // k\n        \n        # Calculate the maximum and minimum weight of any two trucks\n        max_weight = truck_weight * k\n        min_weight = truck_weight * (k - 1)\n        \n        # Calculate the absolute difference between the maximum and minimum weight\n        diff = abs(max_weight - min_weight)\n        \n        # Update the maximum absolute difference\n        max_diff = max(max_diff, diff)\n    \n    # Print the maximum absolute difference\n    print(max_diff)'], 7: ['# YOUR CODE HERE'], 8: ['# YOUR CODE HERE'], 9: ['class Solution:\n    def countSeniors(self, details: List[str]) -> int:\n        count = 0\n        for detail in details:\n            age = int(detail[11:13])\n            if age > 60:\n                count += 1\n        return count'], 10: ['class Solution:\n    def matrixSum(self, nums: List[List[int]]) -> int:\n        for i in range(len(nums)):\n            nums[i].sort(reverse=True)\n        score = 0\n        for j in range(len(nums[0])):\n            max_num = 0\n            for i in range(len(nums)):\n                max_num = max(max_num, nums[i][j])\n            score += max_num\n        return score'], 11: ['class Solution:\n    def maximumOr(self, nums: List[int], k: int) -> int:\n        n = len(nums)\n        left = [0] * n\n        right = [0] * n\n        left[0] = nums[0]\n        right[n - 1] = nums[n - 1]\n        for i in range(1, n):\n            left[i] = left[i - 1] | nums[i]\n        for i in range(n - 2, -1, -1):\n            right[i] = right[i + 1] | nums[i]\n        ans = 0\n        for i in range(n):\n            ans = max(ans, left[i - 1] | (nums[i] << k) | right[i + 1])\n        return ans'], 12: ['class Solution:\n    def maxStrength(self, nums: List[int]) -> int:\n        if len(nums) == 1:\n            return nums[0]\n        nums.sort()\n        if nums[-1] == 0:\n            return 0\n        if nums[0] == 0:\n            return nums[-1]\n        if nums[0] < 0 and nums[1] < 0:\n            return nums[-1] * nums[-2] * nums[0]\n        return nums[-1] * nums[-2]'], 13: ['class Solution:\n    def minExtraChar(self, s: str, dictionary: List[str]) -> int:\n        dp = [0] * (len(s) + 1)\n        for i in range(len(s), -1, -1):\n            dp[i] = dp[i + 1] + 1\n            for word in dictionary:\n                if s[i:i + len(word)] == word:\n                    dp[i] = min(dp[i], dp[i + len(word)])\n        return dp[0]'], 14: ['class Solution:\n    def buyChoco(self, prices: List[int], money: int) -> int:\n        prices.sort()\n        if prices[0] + prices[1] <= money:\n            return money - (prices[0] + prices[1])\n        else:\n            return money'], 15: ['class Solution:\n    def count(self, num1: str, num2: str, min_sum: int, max_sum: int) -> int:\n        MOD = 10**9 + 7\n        n1, n2 = len(num1), len(num2)\n        dp = [[[0] * (max_sum + 1) for _ in range(2)] for _ in range(n1 + 1)]\n        dp[0][0][0] = 1\n\n        for i in range(n1):\n            for j in range(2):\n                for k in range(max_sum + 1):\n                    for d in range(10):\n                        if j == 0 and d > int(num1[i]):\n                            break\n                        if j == 1 and d > int(num2[i]):\n                            break\n                        dp[i + 1][j][k + d] += dp[i][j][k]\n                        dp[i + 1][j][k + d] %= MOD\n\n        ans = 0\n        for i in range(n1, n2 + 1):\n            for j in range(2):\n                for k in range(min_sum, max_sum + 1):\n                    ans += dp[i][j][k]\n                    ans %= MOD\n\n        return ans'], 
...
 398: ['# YOUR CODE HERE'], 399: ['# YOUR CODE HERE']}, 

'eval_results': defaultdict(<class 'list'>, {0: [[-2]], 1: [[-2]], 2: [[-2]], 3: [[True, True, True, True]], 4: [[False]], 5: [[False]], 6: [[False]], 7: [[-2]], 8: [[-2]], 9: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 10: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 11: [[-1]], 12: [[False]], 13: [[-1]], 14: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 15: [[-1]], 16: [[True, -1]], 17: [[-1]], 18: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 19: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 20: [[False]], 21: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 22: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 23: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 24: [[False]], 25: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 26: [[False]], 27: [[False]], 28: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 29: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 30: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 31: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 32: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 33: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 34: [[True, True, True, False]], 35: [[True, True, True, True, True, True, True, True, True, True, True, True, True, -1]], 36: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 37: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 38: [[False]], 39: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 40: [[-2]], 41: [[-1]], 42: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 43: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 44: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 45: [[False]], 46: [[False]], 47: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 48: [[-1]], 49: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 50: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 51: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 52: [[True, False]], 53: [[False]], 54: [[False]], 55: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 56: [[False]], 57: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 58: [[True, True, True, True, True, True, True, -1]], 59: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 60: [[True, True, True, False]], 61: [[True, False]], 62: [[False]], 63: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 64: [[False]], 65: [[-2]], 66: [[True, True, True, True, True, True, True, True, True, True, True, True, False]], 67: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 68: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 69: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 70: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 71: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 72: [[False]], 73: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 74: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 75: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 76: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 77: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 78: [[True, True, True, True, False]], 79: [[False]], 80: [[False]], 81: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, -1]], 82: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 83: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 84: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 85: [[False]], 86: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 87: [[False]], 88: [[True, True, True, True, True, True, True, True, True, True, True, True, -1]], 89: [[-1]], 90: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 91: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 92: [[False]], 93: [[-1]], 94: [[True, False]], 95: [[False]], 96: [[-1]], 97: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 98: [[False]], 99: [[False]], 100: [[False]], 101: [[False]], 102: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 103: [[False]], 104: [[-2]], 105: [[False]], 106: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 107: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 108: [[True, True, True, True, True, False]], 109: [[False]], 110: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 111: [[False]], 112: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 113: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 114: [[False]], 115: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 116: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 117: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 118: [[False]], 119: [[False]], 120: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 121: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 122: [[True, True, True, True, True, False]], 123: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 124: [[True, True, True, True, True, True, True, True, True, True, True, -1]], 125: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 126: [[False]], 127: [[False]], 128: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 129: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 130: [[False]], 131: [[True, True, True, True, True, True, True, False]], 132: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 133: [[False]], 134: [[False]], 135: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 136: [[False]], 137: [[False]], 138: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 139: [[False]], 140: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 141: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 142: [[False]], 143: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 144: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 145: [[True, True, False]], 146: [[True, False]], 147: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 148: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 149: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 150: [[-1]], 151: [[True, False]], 152: [[False]], 153: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 154: [[False]], 155: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 156: [[-1]], 157: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 158: [[-2]], 159: [[True, False]], 160: [[-1]], 161: [[-2]], 162: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 163: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 164: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 165: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 166: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 167: [[True, False]], 168: [[-1]], 169: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 170: [[True, True, False]], 171: [[True, True, True, True, True, True, True, True, True, True, True, True, -1]], 172: [[False]], 173: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 174: [[False]], 175: [[False]], 176: [[False]], 177: [[True, True, True, True, True, True, True, False]], 178: [[False]], 179: [[-2]], 180: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 181: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 182: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 183: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 184: [[False]], 185: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 186: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 187: [[True, True, True, True, False]], 188: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 189: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 190: [[True, False]], 191: [[-2]], 192: [[-2]], 193: [[-2]], 194: [[-2]], 195: [[-2]], 196: [[-2]], 197: [[-2]], 198: [[-2]], 199: [[-2]], 200: [[-2]], 201: [[-2]], 202: [[False]], 203: [[-2]], 204: [[-2]], 205: [[False]], 206: [[-2]], 207: [[False]], 208: [[False]], 209: [[-2]], 210: [[-2]], 211: [[-2]], 212: [[-2]], 213: [[-2]], 214: [[-2]], 215: [[-2]], 216: [[-2]], 217: [[-2]], 218: [[-2]], 219: [[-2]], 220: [[-2]], 221: [[-2]], 222: [[True, True, True, True, True, True, True, True, True, True]], 223: [[-2]], 224: [[-2]], 225: [[False]], 226: [[-2]], 227: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 228: [[-1]], 229: [[False]], 230: [[-2]], 231: [[-2]], 232: [[-2]], 233: [[True, True, False]], 234: [[False]], 235: [[-2]], 236: [[-2]], 237: [[-2]], 238: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 239: [[-2]], 240: [[-2]], 241: [[-2]], 242: [[-2]], 243: [[-2]], 244: [[-2]], 245: [[-2]], 246: [[-2]], 247: [[-2]], 248: [[-2]], 249: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 250: [[-2]], 251: [[-2]], 252: [[-2]], 253: [[-2]], 254: [[-2]], 255: [[False]], 256: [[-2]], 257: [[False]], 258: [[-2]], 259: [[-2]], 260: [[-2]], 261: [[-2]], 262: [[-2]], 263: [[-2]], 264: [[-2]], 265: [[-2]], 266: [[True, True, True, True, True, True, True, -1]], 267: [[-2]], 268: [[-2]], 269: [[-2]], 270: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 271: [[-2]], 272: [[-2]], 273: [[-2]], 274: [[-2]], 275: [[-2]], 276: [[-2]], 277: [[-2]], 278: [[-2]], 279: [[-2]], 280: [[-2]], 281: [[-2]], 282: [[-2]], 283: [[-2]], 284: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 285: [[-2]], 286: [[-2]], 287: [[-2]], 288: [[-2]], 289: [[True, True, True, True, True, True, True, True, True, True, True]], 290: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 291: [[-1]], 292: [[-2]], 293: [[-2]], 294: [[-2]], 295: [[-2]], 296: [[-2]], 297: [[-2]], 298: [[-2]], 299: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 300: [[False]], 301: [[-2]], 302: [[-2]], 303: [[-2]], 304: [[-2]], 305: [[True, True, True, True, True, True, True, True, True, True, True]], 306: [[False]], 307: [[-2]], 308: [[-2]], 309: [[-2]], 310: [[-2]], 311: [[True, False]], 312: [[-2]], 313: [[False]], 314: [[-2]], 315: [[-2]], 316: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 317: [[-2]], 318: [[-2]], 319: [[-2]], 320: [[-2]], 321: [[True, True, True, True, True, True, True, True, True, True, True, True, True]], 322: [[-2]], 323: [[False]], 324: [[-2]], 325: [[-2]], 326: [[True, True, True, True, True, True, True, True, True]], 327: [[False]], 328: [[False]], 329: [[-2]], 330: [[-2]], 331: [[-2]], 332: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 333: [[True, True, False]], 334: [[-2]], 335: [[-2]], 336: [[-2]], 337: [[-2]], 338: [[True, False]], 339: [[-2]], 340: [[-2]], 341: [[-2]], 342: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 343: [[False]], 344: [[False]], 345: [[-2]], 346: [[-2]], 347: [[-2]], 348: [[-2]], 349: [[-2]], 350: [[-2]], 351: [[True, True, True, True, True, True, True, True, True, True, True, True]], 352: [[-2]], 353: [[-2]], 354: [[-2]], 355: [[-2]], 356: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 357: [[-2]], 358: [[-2]], 359: [[-2]], 360: [[-2]], 361: [[-2]], 362: [[-2]], 363: [[-2]], 364: [[True, True, True, True, True, True, True, True, True, True, True, True, True]], 365: [[-2]], 366: [[-2]], 367: [[-2]], 368: [[-2]], 369: [[True, True, True, False]], 370: [[True, True, True, True, True, True, True, True, True, True, True]], 371: [[-2]], 372: [[-2]], 373: [[-2]], 374: [[-2]], 375: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 376: [[-2]], 377: [[False]], 378: [[-2]], 379: [[True, True, True, True, True, True, True, True, True, True, True, True]], 380: [[-2]], 381: [[-2]], 382: [[-2]], 383: [[-2]], 384: [[-2]], 385: [[-2]], 386: [[-2]], 387: [[-2]], 388: [[-2]], 389: [[-2]], 390: [[True, False]], 391: [[False]], 392: [[True, True, False]], 393: [[-2]], 394: [[-2]], 395: [[False]], 396: [[-2]], 397: [[-2]], 398: [[-2]], 399: [[-2]]}), 'pass@1': 27.75, 'detail': {'pass@1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 100.0, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.0, 8: 0.0, 9: 100.0, 10: 100.0, 11: 0.0, 12: 0.0, 13: 0.0, 14: 100.0, 15: 0.0, 16: 0.0, 17: 0.0, 18: 100.0, 19: 100.0, 20: 0.0, 21: 100.0, 22: 100.0, 23: 100.0, 24: 0.0, 25: 100.0, 26: 0.0, 27: 0.0, 28: 100.0, 29: 100.0, 30: 100.0, 31: 100.0, 32: 100.0, 33: 100.0, 34: 0.0, 35: 0.0, 36: 100.0, 37: 100.0, 38: 0.0, 39: 100.0, 40: 0.0, 41: 0.0, 42: 100.0, 43: 100.0, 44: 100.0, 45: 0.0, 46: 0.0, 47: 100.0, 48: 0.0, 49: 100.0, 50: 100.0, 51: 100.0, 52: 0.0, 53: 0.0, 54: 0.0, 55: 100.0, 56: 0.0, 57: 100.0, 58: 0.0, 59: 100.0, 60: 0.0, 61: 0.0, 62: 0.0, 63: 100.0, 64: 0.0, 65: 0.0, 66: 0.0, 67: 100.0, 68: 100.0, 69: 100.0, 70: 100.0, 71: 100.0, 72: 0.0, 73: 100.0, 74: 100.0, 75: 100.0, 76: 100.0, 77: 100.0, 78: 0.0, 79: 0.0, 80: 0.0, 81: 0.0, 82: 100.0, 83: 100.0, 84: 100.0, 85: 0.0, 86: 100.0, 87: 0.0, 88: 0.0, 89: 0.0, 90: 100.0, 91: 100.0, 92: 0.0, 93: 0.0, 94: 0.0, 95: 0.0, 96: 0.0, 97: 100.0, 98: 0.0, 99: 0.0, 100: 0.0, 101: 0.0, 102: 100.0, 103: 0.0, 104: 0.0, 105: 0.0, 106: 100.0, 107: 100.0, 108: 0.0, 109: 0.0, 110: 100.0, 111: 0.0, 112: 100.0, 113: 100.0, 114: 0.0, 115: 100.0, 116: 100.0, 117: 100.0, 118: 0.0, 119: 0.0, 120: 100.0, 121: 100.0, 122: 0.0, 123: 100.0, 124: 0.0, 125: 100.0, 126: 0.0, 127: 0.0, 128: 100.0, 129: 100.0, 130: 0.0, 131: 0.0, 132: 100.0, 133: 0.0, 134: 0.0, 135: 100.0, 136: 0.0, 137: 0.0, 138: 100.0, 139: 0.0, 140: 100.0, 141: 100.0, 142: 0.0, 143: 100.0, 144: 100.0, 145: 0.0, 146: 0.0, 147: 100.0, 148: 100.0, 149: 100.0, 150: 0.0, 151: 0.0, 152: 0.0, 153: 100.0, 154: 0.0, 155: 100.0, 156: 0.0, 157: 100.0, 158: 0.0, 159: 0.0, 160: 0.0, 161: 0.0, 162: 100.0, 163: 100.0, 164: 100.0, 165: 100.0, 166: 100.0, 167: 0.0, 168: 0.0, 169: 100.0, 170: 0.0, 171: 0.0, 172: 0.0, 173: 100.0, 174: 0.0, 175: 0.0, 176: 0.0, 177: 0.0, 178: 0.0, 179: 0.0, 180: 100.0, 181: 100.0, 182: 100.0, 183: 100.0, 184: 0.0, 185: 100.0, 186: 100.0, 187: 0.0, 188: 100.0, 189: 100.0, 190: 0.0, 191: 0.0, 192: 0.0, 193: 0.0, 194: 0.0, 195: 0.0, 196: 0.0, 197: 0.0, 198: 0.0, 199: 0.0, 200: 0.0, 201: 0.0, 202: 0.0, 203: 0.0, 204: 0.0, 205: 0.0, 206: 0.0, 207: 0.0, 208: 0.0, 209: 0.0, 210: 0.0, 211: 0.0, 212: 0.0, 213: 0.0, 214: 0.0, 215: 0.0, 216: 0.0, 217: 0.0, 218: 0.0, 219: 0.0, 220: 0.0, 221: 0.0, 222: 100.0, 223: 0.0, 224: 0.0, 225: 0.0, 226: 0.0, 227: 100.0, 228: 0.0, 229: 0.0, 230: 0.0, 231: 0.0, 232: 0.0, 233: 0.0, 234: 0.0, 235: 0.0, 236: 0.0, 237: 0.0, 238: 100.0, 239: 0.0, 240: 0.0, 241: 0.0, 242: 0.0, 243: 0.0, 244: 0.0, 245: 0.0, 246: 0.0, 247: 0.0, 248: 0.0, 249: 100.0, 250: 0.0, 251: 0.0, 252: 0.0, 253: 0.0, 254: 0.0, 255: 0.0, 256: 0.0, 257: 0.0, 258: 0.0, 259: 0.0, 260: 0.0, 261: 0.0, 262: 0.0, 263: 0.0, 264: 0.0, 265: 0.0, 266: 0.0, 267: 0.0, 268: 0.0, 269: 0.0, 270: 100.0, 271: 0.0, 272: 0.0, 273: 0.0, 274: 0.0, 275: 0.0, 276: 0.0, 277: 0.0, 278: 0.0, 279: 0.0, 280: 0.0, 281: 0.0, 282: 0.0, 283: 0.0, 284: 100.0, 285: 0.0, 286: 0.0, 287: 0.0, 288: 0.0, 289: 100.0, 290: 100.0, 291: 0.0, 292: 0.0, 293: 0.0, 294: 0.0, 295: 0.0, 296: 0.0, 297: 0.0, 298: 0.0, 299: 100.0, 300: 0.0, 301: 0.0, 302: 0.0, 303: 0.0, 304: 0.0, 305: 100.0, 306: 0.0, 307: 0.0, 308: 0.0, 309: 0.0, 310: 0.0, 311: 0.0, 312: 0.0, 313: 0.0, 314: 0.0, 315: 0.0, 316: 100.0, 317: 0.0, 318: 0.0, 319: 0.0, 320: 0.0, 321: 100.0, 322: 0.0, 323: 0.0, 324: 0.0, 325: 0.0, 326: 100.0, 327: 0.0, 328: 0.0, 329: 0.0, 330: 0.0, 331: 0.0, 332: 100.0, 333: 0.0, 334: 0.0, 335: 0.0, 336: 0.0, 337: 0.0, 338: 0.0, 339: 0.0, 340: 0.0, 341: 0.0, 342: 100.0, 343: 0.0, 344: 0.0, 345: 0.0, 346: 0.0, 347: 0.0, 348: 0.0, 349: 0.0, 350: 0.0, 351: 100.0, 352: 0.0, 353: 0.0, 354: 0.0, 355: 0.0, 356: 100.0, 357: 0.0, 358: 0.0, 359: 0.0, 360: 0.0, 361: 0.0, 362: 0.0, 363: 0.0, 364: 100.0, 365: 0.0, 366: 0.0, 367: 0.0, 368: 0.0, 369: 0.0, 370: 100.0, 371: 0.0, 372: 0.0, 373: 0.0, 374: 0.0, 375: 100.0, 376: 0.0, 377: 0.0, 378: 0.0, 379: 100.0, 380: 0.0, 381: 0.0, 382: 0.0, 383: 0.0, 384: 0.0, 385: 0.0, 386: 0.0, 387: 0.0, 388: 0.0, 389: 0.0, 390: 0.0, 391: 0.0, 392: 0.0, 393: 0.0, 394: 0.0, 395: 0.0, 396: 0.0, 397: 0.0, 398: 0.0, 399: 0.0}}}
03/10 23:09:05 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/execution-v2
lcb_code_execution test 479
lcb_code_execution train 479
03/10 23:09:16 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_code_execution]: {'pass@1': 1.4613778705636742}
03/10 23:09:16 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/test_generation
lcb_test_output test 442
lcb_test_output train 442
100%|██████████| 442/442 [00:00<00:00, 38219.65it/s]
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 0)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
invalid syntax (<string>, line 1)
03/10 23:09:16 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_test_output]: {'pass@1': 40.04524886877828, 'detail': {'pass@1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 100.0, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.0, 8: 100.0, 9: 100.0, 10: 0.0, 11: 0.0, 12: 0.0, 13: 0.0, 14: 0.0, 15: 0.0, 16: 100.0, 17: 100.0, 18: 0.0, 19: 0.0, 20: 0.0, 21: 100.0, 22: 0.0, 23: 100.0, 24: 0.0, 25: 100.0, 26: 100.0, 27: 0.0, 28: 100.0, 29: 0.0, 30: 0.0, 31: 100.0, 32: 0.0, 33: 0.0, 34: 0.0, 35: 0.0, 36: 0.0, 37: 0.0, 38: 0.0, 39: 0.0, 40: 100.0, 41: 0.0, 42: 0.0, 43: 100.0, 44: 100.0, 45: 0.0, 46: 0.0, 47: 0.0, 48: 0.0, 49: 0.0, 50: 100.0, 51: 100.0, 52: 100.0, 53: 100.0, 54: 0.0, 55: 100.0, 56: 0.0, 57: 100.0, 58: 0.0, 59: 0.0, 60: 0.0, 61: 0.0, 62: 100.0, 63: 100.0, 64: 0.0, 65: 0.0, 66: 0.0, 67: 0.0, 68: 0.0, 69: 100.0, 70: 0.0, 71: 0.0, 72: 0.0, 73: 0.0, 74: 0.0, 75: 0.0, 76: 100.0, 77: 0.0, 78: 100.0, 79: 100.0, 80: 100.0, 81: 100.0, 82: 100.0, 83: 0.0, 84: 0.0, 85: 0.0, 86: 0.0, 87: 100.0, 88: 0.0, 89: 0.0, 90: 100.0, 91: 0.0, 92: 100.0, 93: 100.0, 94: 0.0, 95: 100.0, 96: 0.0, 97: 100.0, 98: 0.0, 99: 100.0, 100: 0.0, 101: 100.0, 102: 0.0, 103: 100.0, 104: 100.0, 105: 100.0, 106: 100.0, 107: 100.0, 108: 0.0, 109: 0.0, 110: 0.0, 111: 100.0, 112: 100.0, 113: 100.0, 114: 0.0, 115: 0.0, 116: 0.0, 117: 100.0, 118: 0.0, 119: 0.0, 120: 0.0, 121: 0.0, 122: 0.0, 123: 0.0, 124: 0.0, 125: 100.0, 126: 0.0, 127: 0.0, 128: 0.0, 129: 0.0, 130: 0.0, 131: 100.0, 132: 0.0, 133: 0.0, 134: 100.0, 135: 100.0, 136: 100.0, 137: 0.0, 138: 0.0, 139: 100.0, 140: 100.0, 141: 0.0, 142: 0.0, 143: 0.0, 144: 100.0, 145: 100.0, 146: 0.0, 147: 0.0, 148: 100.0, 149: 0.0, 150: 100.0, 151: 100.0, 152: 0.0, 153: 0.0, 154: 0.0, 155: 0.0, 156: 100.0, 157: 100.0, 158: 100.0, 159: 100.0, 160: 100.0, 161: 100.0, 162: 100.0, 163: 100.0, 164: 0.0, 165: 0.0, 166: 100.0, 167: 100.0, 168: 100.0, 169: 0.0, 170: 0.0, 171: 0.0, 172: 0.0, 173: 100.0, 174: 100.0, 175: 100.0, 176: 100.0, 177: 0.0, 178: 100.0, 179: 0.0, 180: 100.0, 181: 0.0, 182: 0.0, 183: 100.0, 184: 0.0, 185: 100.0, 186: 100.0, 187: 100.0, 188: 0.0, 189: 100.0, 190: 100.0, 191: 0.0, 192: 0.0, 193: 0.0, 194: 0.0, 195: 0.0, 196: 0.0, 197: 100.0, 198: 0.0, 199: 0.0, 200: 0.0, 201: 0.0, 202: 100.0, 203: 0.0, 204: 0.0, 205: 100.0, 206: 0.0, 207: 0.0, 208: 100.0, 209: 100.0, 210: 100.0, 211: 100.0, 212: 100.0, 213: 0.0, 214: 100.0, 215: 0.0, 216: 0.0, 217: 100.0, 218: 100.0, 219: 0.0, 220: 0.0, 221: 100.0, 222: 0.0, 223: 100.0, 224: 0.0, 225: 0.0, 226: 0.0, 227: 100.0, 228: 100.0, 229: 0.0, 230: 0.0, 231: 0.0, 232: 0.0, 233: 100.0, 234: 0.0, 235: 100.0, 236: 0.0, 237: 100.0, 238: 0.0, 239: 0.0, 240: 0.0, 241: 0.0, 242: 0.0, 243: 100.0, 244: 100.0, 245: 0.0, 246: 0.0, 247: 100.0, 248: 100.0, 249: 0.0, 250: 0.0, 251: 0.0, 252: 100.0, 253: 0.0, 254: 0.0, 255: 0.0, 256: 0.0, 257: 0.0, 258: 100.0, 259: 100.0, 260: 100.0, 261: 0.0, 262: 100.0, 263: 0.0, 264: 0.0, 265: 100.0, 266: 0.0, 267: 100.0, 268: 0.0, 269: 0.0, 270: 0.0, 271: 100.0, 272: 0.0, 273: 0.0, 274: 0.0, 275: 0.0, 276: 0.0, 277: 0.0, 278: 100.0, 279: 100.0, 280: 0.0, 281: 100.0, 282: 100.0, 283: 0.0, 284: 0.0, 285: 0.0, 286: 0.0, 287: 100.0, 288: 0.0, 289: 0.0, 290: 100.0, 291: 0.0, 292: 100.0, 293: 100.0, 294: 0.0, 295: 0.0, 296: 0.0, 297: 100.0, 298: 0.0, 299: 100.0, 300: 0.0, 301: 0.0, 302: 0.0, 303: 0.0, 304: 0.0, 305: 0.0, 306: 0.0, 307: 100.0, 308: 0.0, 309: 100.0, 310: 100.0, 311: 100.0, 312: 0.0, 313: 100.0, 314: 0.0, 315: 100.0, 316: 100.0, 317: 0.0, 318: 100.0, 319: 100.0, 320: 100.0, 321: 0.0, 322: 100.0, 323: 0.0, 324: 0.0, 325: 100.0, 326: 100.0, 327: 100.0, 328: 100.0, 329: 100.0, 330: 0.0, 331: 0.0, 332: 0.0, 333: 100.0, 334: 100.0, 335: 100.0, 336: 0.0, 337: 100.0, 338: 100.0, 339: 100.0, 340: 0.0, 341: 100.0, 342: 100.0, 343: 0.0, 344: 0.0, 345: 0.0, 346: 100.0, 347: 100.0, 348: 100.0, 349: 100.0, 350: 0.0, 351: 0.0, 352: 100.0, 353: 0.0, 354: 0.0, 355: 0.0, 356: 0.0, 357: 0.0, 358: 0.0, 359: 0.0, 360: 100.0, 361: 100.0, 362: 0.0, 363: 0.0, 364: 100.0, 365: 0.0, 366: 0.0, 367: 100.0, 368: 100.0, 369: 0.0, 370: 0.0, 371: 0.0, 372: 0.0, 373: 100.0, 374: 0.0, 375: 0.0, 376: 0.0, 377: 0.0, 378: 100.0, 379: 100.0, 380: 0.0, 381: 0.0, 382: 100.0, 383: 100.0, 384: 100.0, 385: 100.0, 386: 0.0, 387: 0.0, 388: 100.0, 389: 0.0, 390: 100.0, 391: 0.0, 392: 0.0, 393: 100.0, 394: 0.0, 395: 0.0, 396: 100.0, 397: 0.0, 398: 0.0, 399: 0.0, 400: 0.0, 401: 0.0, 402: 0.0, 403: 100.0, 404: 0.0, 405: 0.0, 406: 100.0, 407: 100.0, 408: 0.0, 409: 0.0, 410: 0.0, 411: 0.0, 412: 100.0, 413: 0.0, 414: 0.0, 415: 0.0, 416: 0.0, 417: 100.0, 418: 0.0, 419: 0.0, 420: 0.0, 421: 0.0, 422: 100.0, 423: 0.0, 424: 100.0, 425: 0.0, 426: 0.0, 427: 0.0, 428: 0.0, 429: 0.0, 430: 100.0, 431: 0.0, 432: 0.0, 433: 0.0, 434: 100.0, 435: 0.0, 436: 100.0, 437: 0.0, 438: 0.0, 439: 100.0, 440: 0.0, 441: 0.0}}}
dataset              version    metric    mode      Qwen2.5-7B_hf
-------------------  ---------  --------  ------  ---------------
lcb_code_generation  f0ed6c     pass@1    gen               27.75
lcb_code_execution   24f99f     pass@1    gen                1.46
lcb_test_output      1fe37c     pass@1    gen               40.05
03/10 23:09:16 - OpenCompass - INFO - write summary to /nas/xz/LLaMA-Factory/saves/oc/outputs/default/20250310_201431/summary/summary_20250310_201431.txt
03/10 23:09:16 - OpenCompass - INFO - write csv to /nas/xz/LLaMA-Factory/saves/oc/outputs/default/20250310_201431/summary/summary_20250310_201431.csv


The markdown format results is as below:

| dataset | version | metric | mode | Qwen2.5-7B_hf |
|----- | ----- | ----- | ----- | -----|
| lcb_code_generation | f0ed6c | pass@1 | gen | 27.75 |
| lcb_code_execution | 24f99f | pass@1 | gen | 1.46 |
| lcb_test_output | 1fe37c | pass@1 | gen | 40.05 |

03/10 23:09:16 - OpenCompass - INFO - write markdown summary to /nas/xz/LLaMA-Factory/saves/oc/outputs/default/20250310_201431/summary/summary_20250310_201431.md

### 其他信息

cat tmp/3399630_debug.log

03/10 20:20:24 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_code_generation,Qwen2.5-7B_hf/lcb_code_execution,Qwen2.5-7B_hf/lcb_test_output]
INFO 03-10 20:20:27 [__init__.py:256] Automatically detected platform cuda.
Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.27s/it]
03/10 20:20:37 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/code_generation_lite
lcb_code_generation test 400
lcb_code_generation train 400
03/10 20:21:49 - OpenCompass - INFO - Start inferencing [Qwen2.5-7B_hf/lcb_code_generation]
[2025-03-10 20:21:50,415] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-10 20:21:50,416] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|██████████| 50/50 [15:04<00:00, 18.10s/it]
03/10 20:36:55 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/execution-v2
lcb_code_execution test 479
lcb_code_execution train 479
03/10 20:36:55 - OpenCompass - INFO - Start inferencing [Qwen2.5-7B_hf/lcb_code_execution]
[2025-03-10 20:36:55,583] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-10 20:36:55,583] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|██████████| 60/60 [1:32:48<00:00, 92.81s/it] 
03/10 22:09:44 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/test_generation
lcb_test_output test 442
lcb_test_output train 442
03/10 22:09:44 - OpenCompass - INFO - Start inferencing [Qwen2.5-7B_hf/lcb_test_output]
[2025-03-10 22:09:44,358] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-10 22:09:44,358] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|██████████| 56/56 [56:02<00:00, 60.04s/it]  
03/10 23:05:46 - OpenCompass - INFO - time elapsed: 9922.02s