Repository: open-compass/opencompass
Issue: [Bug] livecodebench evaluator may need to check the memory footprint 
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```python
{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'GPU 0,1': 'NVIDIA GeForce GTX TITAN X',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
 'OpenCV': '4.11.0',
 'PyTorch': '2.6.0+cu124',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2024.2-Product Build 20240605 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 12.4\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, '
                              'COMMIT_SHA=2236df1770800ffea5697b11b0bb0d910b2e59e1, '
                              'CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
                              'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
                              'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0]',
 'TorchVision': '0.21.0+cu124',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.4.0+',
 'sys.platform': 'linux',
 'transformers': '4.48.3'}
```

### Reproduces the problem - code/configuration sample

opencompass/datasets/livecodebench/evaluator.py
```python
    def _temp_run(sample, generation, debug, result, metadata_list, timeout):

        from .testing_util import run_test
        res, metadata = run_test(sample,
                                 test=generation,
                                 debug=debug,
                                 timeout=timeout)
        result.append(res)
        metadata_list.append(metadata)
```


### Reproduces the problem - command or script

```shell
python run.py --models dsllama.py --datasets livecodebench_gen.py -m eval -r 20250423_084518 -w /home/deep/opencompass/opencompass-main/outputs/ --debug --dump-eval-details
```

### Reproduces the problem - error message

When run the code generated by the model, the system memory is full . 
 
Code generated by the model
```python
from collections import deque
import os 
import psutil

class Solution:
    def minLength(self, s: str) -> int:
        visited = set()
        queue = deque()
        queue.append(s)
        visited.add(s)
        min_len = len(s)
        
        while queue:
            current = queue.popleft()
            current_len = len(current)
            if current_len < min_len:
                min_len = current_len
            if min_len == 0:
                break
            process = psutil.Process(os.getpid())
            mem_info = process.memory_info()
            print(f"内存使用: {mem_info.rss / 1024 / 1024:.2f} MB")
            for i in range(len(current) - 1):
                substr = current[i:i+2]
                if substr in ("AB", "CD"):
                    new_str = current[:i] + current[i+2:]
                    if new_str not in visited:
                        visited.add(new_str)
                        queue.append(new_str)
        return min_len
    
input = "ABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCDABCD"
solution = Solution()
result = solution.minLength(input)
print(result)  
```

### Other information

make some temporary changes 
```python
    def _temp_run(sample, generation, debug, result, metadata_list, timeout):
        #NOTE 为了避免生成的代码占用过多内存
        import sys
        if sys.platform.startswith('linux'):
            import resource
            # 设置最大虚拟内存为 100mb
            soft, hard = 100 * 1024 * 1024,  100 * 1024 * 1024
            resource.setrlimit(resource.RLIMIT_AS, (soft, hard))
        from .testing_util import run_test
        res, metadata = run_test(sample,
                                 test=generation,
                                 debug=debug,
                                 timeout=timeout)
        result.append(res)
        metadata_list.append(metadata)
```

Thank you. Would you be interested in creating a pull request to enhance this?