Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] AttributeError: vllm_model
Root cause: Resource Mishandling

**Describe the bug**
The following error occurred when using the vllm module. It did not occur when using the openai_llm module. This error may have occurred when the `vllm_model` was deleted. I am unsure of the exact cause, so I would appreciate it if you could investigate.

**configuration**
```yaml
vectordb:
  ...
node_lines:
- node_line_name: retrieve_node_line
  ...
- node_line_name: post_retrieve_node_line
  nodes:
    - node_type: prompt_maker
      ...
    - node_type: generator
      strategy:
        metrics: # bert_score 및 g_eval 사용 역시 추천합니다. 빠른 실행을 위해 여기서는 제외하고 하겠습니다.
          - metric_name: rouge
          - metric_name: sem_score
            embedding_model: openai
#          - metric_name: bert_score
#            lang: ko
      modules:
        - module_type: openai_llm
          llm: gpt-4o-mini
          temperature: [ 0.1, 1.0 ]
          batch: 16
        - module_type: vllm
          llm: meta-llama/Llama-3.2-1B-instruct
          temperature: [ 0.1, 1.0 ]
          # tensor_parallel_size: 4 # If the gpu is two.
          max_tokens: 128
          max_model_len: 200
          gpu_memory_utilization: 0.5
```

**Full Error log**
```
Ingesting VectorDB... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 1/1 0:00:00
Evaluating...         ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━  67% 2/3 0:03:08
[11/25/24 18:06:53] ERROR    [__init__.py:60] >> Unexpected exception                                                                                      __init__.py:60
                             ╭──────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────╮               
                             │ /root/.pyenv/versions/3.10.13/lib/python3.10/runpy.py:196 in _run_module_as_main                                          │               
                             │                                                                                                                           │               
                             │   193 │   main_globals = sys.modules["__main__"].__dict__                                                                 │               
                             │   194 │   if alter_argv:                                                                                                  │               
                             │   195 │   │   sys.argv[0] = mod_spec.origin                                                                               │               
                             │ ❱ 196 │   return _run_code(code, main_globals, None,                                                                      │               
                             │   197 │   │   │   │   │    "__main__", mod_spec)                                                                          │               
                             │   198                                                                                                                     │               
                             │   199 def run_module(mod_name, init_globals=None,                                                                         │               
                             │                                                                                                                           │               
                             │ /root/.pyenv/versions/3.10.13/lib/python3.10/runpy.py:86 in _run_code                                                     │               
                             │                                                                                                                           │               
                             │    83 │   │   │   │   │      __loader__ = loader,                                                                         │               
                             │    84 │   │   │   │   │      __package__ = pkg_name,                                                                      │               
                             │    85 │   │   │   │   │      __spec__ = mod_spec)                                                                         │               
                             │ ❱  86 │   exec(code, run_globals)                                                                                         │               
                             │    87 │   return run_globals                                                                                              │               
                             │    88                                                                                                                     │               
                             │    89 def _run_module_code(code, init_globals=None,                                                                       │               
                             │                                                                                                                           │               
                             │ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher │               
                             │ /../../debugpy/__main__.py:71 in <module>                                                                                 │               
                             │                                                                                                                           │               
                             │   68 │                                                                                                                    │               
                             │   69 │   from debugpy.server import cli                                                                                   │               
                             │   70 │                                                                                                                    │               
                             │ ❱ 71 │   cli.main()                                                                                                       │               
                             │   72                                                                                                                      │               
                             │                                                                                                                           │               
                             │ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher │               
                             │ /../../debugpy/../debugpy/server/cli.py:501 in main                                                                       │               
                             │                                                                                                                           │               
                             │   498 │   │   │   │   "code": run_code,                                                                                   │               
                             │   499 │   │   │   │   "pid": attach_to_pid,                                                                               │               
                             │   500 │   │   │   }[options.target_kind]                                                                                  │               
                             │ ❱ 501 │   │   │   run()                                                                                                   │               
                             │   502 │   except SystemExit as exc:                                                                                       │               
                             │   503 │   │   log.reraise_exception(                                                                                      │               
                             │   504 │   │   │   "Debuggee exited via SystemExit: {0!r}", exc.code, level="debug"                                        │               
                             │                                                                                                                           │               
                             │ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher │               
                             │ /../../debugpy/../debugpy/server/cli.py:351 in run_file                                                                   │               
                             │                                                                                                                           │               
                             │   348 │   log.describe_environment("Pre-launch environment:")                                                             │               
                             │   349 │                                                                                                                   │               
                             │   350 │   log.info("Running file {0!r}", target)                                                                          │               
                             │ ❱ 351 │   runpy.run_path(target, run_name="__main__")                                                                     │               
                             │   352                                                                                                                     │               
                             │   353                                                                                                                     │               
                             │   354 def run_module():                                                                                                   │               
                             │                                                                                                                           │               
                             │ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundl │               
                             │ e/pydevd_runpy.py:310 in run_path                                                                                         │               
                             │                                                                                                                           │               
                             │   307 │   │   # Not a valid sys.path entry, so run the code directly                                                      │               
                             │   308 │   │   # execfile() doesn't help as we want to allow compiled files                                                │               
                             │   309 │   │   code, fname = _get_code_from_file(run_name, path_name)                                                      │               
                             │ ❱ 310 │   │   return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name,                                    │               
                             │       script_name=fname)                                                                                                  │               
                             │   311 │   else:                                                                                                           │               
                             │   312 │   │   # Finder is defined for path, so add it to                                                                  │               
                             │   313 │   │   # the start of sys.path                                                                                     │               
                             │                                                                                                                           │               
                             │ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundl │               
                             │ e/pydevd_runpy.py:127 in _run_module_code                                                                                 │               
                             │                                                                                                                           │               
                             │   124 │   fname = script_name if mod_spec is None else mod_spec.origin                                                    │               
                             │   125 │   with _TempModule(mod_name) as temp_module, _ModifiedArgv0(fname):                                               │               
                             │   126 │   │   mod_globals = temp_module.module.__dict__                                                                   │               
                             │ ❱ 127 │   │   _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name,                                    │               
                             │       script_name)                                                                                                        │               
                             │   128 │   # Copy the globals of the temporary module, as they                                                             │               
                             │   129 │   # may be cleared when the temporary module goes away                                                            │               
                             │   130 │   return mod_globals.copy()                                                                                       │               
                             │                                                                                                                           │               
                             │ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundl │               
                             │ e/pydevd_runpy.py:118 in _run_code                                                                                        │               
                             │                                                                                                                           │               
                             │   115 │   run_globals.update(                                                                                             │               
                             │   116 │   │   __name__=mod_name, __file__=fname, __cached__=cached, __doc__=None,                                         │               
                             │       __loader__=loader, __package__=pkg_name, __spec__=mod_spec                                                          │               
                             │   117 │   )                                                                                                               │               
                             │ ❱ 118 │   exec(code, run_globals)                                                                                         │               
                             │   119 │   return run_globals                                                                                              │               
                             │   120                                                                                                                     │               
                             │   121                                                                                                                     │               
                             │                                                                                                                           │               
                             │ /app/evaluate/main.py:138 in <module>                                                                                     │               
                             │                                                                                                                           │               
                             │   135 │   # parse()                                                                                                       │               
                             │   136 │   # chunk()                                                                                                       │               
                             │   137 │   # make_qa()                                                                                                     │               
                             │ ❱ 138 │   evaluate()                                                                                                      │               
                             │   139                                                                                                                     │               
                             │                                                                                                                           │               
                             │ /app/evaluate/main.py:127 in evaluate                                                                                     │               
                             │                                                                                                                           │               
                             │   124 │   │   os.makedirs(project_dir)                                                                                    │               
                             │   125 │                                                                                                                   │               
                             │   126 │   evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir=project_dir)                                  │               
                             │ ❱ 127 │   evaluator.start_trial(config)                                                                                   │               
                             │   128                                                                                                                     │               
                             │   129                                                                                                                     │               
                             │   130 if __name__ == "__main__":                                                                                          │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/evaluator.py:138 in │               
                             │ start_trial                                                                                                               │               
                             │                                                                                                                           │               
                             │   135 │   │   │   validator = Validator(                                                                                  │               
                             │   136 │   │   │   │   qa_data_path=self.qa_data_path, corpus_data_path=self.corpus_data_path                              │               
                             │   137 │   │   │   )                                                                                                       │               
                             │ ❱ 138 │   │   │   validator.validate(yaml_path)                                                                           │               
                             │   139 │   │                                                                                                               │               
                             │   140 │   │   os.environ["PROJECT_DIR"] = self.project_dir                                                                │               
                             │   141                                                                                                                     │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/validator.py:92 in  │               
                             │ validate                                                                                                                  │               
                             │                                                                                                                           │               
                             │   89 │   │   │   │   corpus_data_path=corpus_path.name,                                                                   │               
                             │   90 │   │   │   │   project_dir=temp_project_dir,                                                                        │               
                             │   91 │   │   │   )                                                                                                        │               
                             │ ❱ 92 │   │   │   evaluator.start_trial(yaml_path, skip_validation=True)                                                   │               
                             │   93 │   │   │   qa_path.close()                                                                                          │               
                             │   94 │   │   │   corpus_path.close()                                                                                      │               
                             │   95 │   │   │   os.unlink(qa_path.name)                                                                                  │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/evaluator.py:206 in │               
                             │ start_trial                                                                                                               │               
                             │                                                                                                                           │               
                             │   203 │   │   │   │   if i == 0:                                                                                          │               
                             │   204 │   │   │   │   │   previous_result = self.qa_data                                                                  │               
                             │   205 │   │   │   │   logger.info(f"Running node line {node_line_name}...")                                               │               
                             │ ❱ 206 │   │   │   │   previous_result = run_node_line(                                                                    │               
                             │   207 │   │   │   │   │   node_line, node_line_dir, previous_result, progress, task_eval                                  │               
                             │   208 │   │   │   │   )                                                                                                   │               
                             │   209                                                                                                                     │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/node_line.py:52 in  │               
                             │ run_node_line                                                                                                             │               
                             │                                                                                                                           │               
                             │   49 │                                                                                                                    │               
                             │   50 │   summary_lst = []                                                                                                 │               
                             │   51 │   for node in nodes:                                                                                               │               
                             │ ❱ 52 │   │   previous_result = node.run(previous_result, node_line_dir)                                                   │               
                             │   53 │   │   node_summary_df = load_summary_file(                                                                         │               
                             │   54 │   │   │   os.path.join(node_line_dir, node.node_type, "summary.csv")                                               │               
                             │   55 │   │   )                                                                                                            │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/schema/node.py:57   │               
                             │ in run                                                                                                                    │               
                             │                                                                                                                           │               
                             │    54 │   def run(self, previous_result: pd.DataFrame, node_line_dir: str) -> pd.DataFrame:                               │               
                             │    55 │   │   logger.info(f"Running node {self.node_type}...")                                                            │               
                             │    56 │   │   input_modules, input_params = self.get_param_combinations()                                                 │               
                             │ ❱  57 │   │   return self.run_node(                                                                                       │               
                             │    58 │   │   │   modules=input_modules,                                                                                  │               
                             │    59 │   │   │   module_params=input_params,                                                                             │               
                             │    60 │   │   │   previous_result=previous_result,                                                                        │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/run │               
                             │ .py:47 in run_generator_node                                                                                              │               
                             │                                                                                                                           │               
                             │    44 │   if "generation_gt" not in qa_data.columns:                                                                      │               
                             │    45 │   │   raise ValueError("You must have 'generation_gt' column in qa.parquet.")                                     │               
                             │    46 │                                                                                                                   │               
                             │ ❱  47 │   results, execution_times = zip(                                                                                 │               
                             │    48 │   │   *map(                                                                                                       │               
                             │    49 │   │   │   lambda x: measure_speed(                                                                                │               
                             │    50 │   │   │   │   x[0].run_evaluator,                                                                                 │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/run │               
                             │ .py:49 in <lambda>                                                                                                        │               
                             │                                                                                                                           │               
                             │    46 │                                                                                                                   │               
                             │    47 │   results, execution_times = zip(                                                                                 │               
                             │    48 │   │   *map(                                                                                                       │               
                             │ ❱  49 │   │   │   lambda x: measure_speed(                                                                                │               
                             │    50 │   │   │   │   x[0].run_evaluator,                                                                                 │               
                             │    51 │   │   │   │   project_dir=project_dir,                                                                            │               
                             │    52 │   │   │   │   previous_result=previous_result,                                                                    │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/strategy.py:14 in   │               
                             │ measure_speed                                                                                                             │               
                             │                                                                                                                           │               
                             │    11 │   Method for measuring execution speed of the function.                                                           │               
                             │    12 │   """                                                                                                             │               
                             │    13 │   start_time = time.time()                                                                                        │               
                             │ ❱  14 │   result = func(*args, **kwargs)                                                                                  │               
                             │    15 │   end_time = time.time()                                                                                          │               
                             │    16 │   return result, end_time - start_time                                                                            │               
                             │    17                                                                                                                     │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/schema/base.py:25   │               
                             │ in run_evaluator                                                                                                          │               
                             │                                                                                                                           │               
                             │   22 │   │   *args,                                                                                                       │               
                             │   23 │   │   **kwargs,                                                                                                    │               
                             │   24 │   ):                                                                                                               │               
                             │ ❱ 25 │   │   instance = cls(project_dir, *args, **kwargs)                                                                 │               
                             │   26 │   │   result = instance.pure(previous_result, *args, **kwargs)                                                     │               
                             │   27 │   │   del instance                                                                                                 │               
                             │   28 │   │   return result                                                                                                │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/vll │               
                             │ m.py:29 in __init__                                                                                                       │               
                             │                                                                                                                           │               
                             │    26 │   │   sampling_params_init_params = pop_params(                                                                   │               
                             │    27 │   │   │   SamplingParams.from_optional, input_kwargs                                                              │               
                             │    28 │   │   )                                                                                                           │               
                             │ ❱  29 │   │   self.vllm_model = LLM(model, **input_kwargs)                                                                │               
                             │    30 │   │                                                                                                               │               
                             │    31 │   │   # delete not sampling param keys in the kwargs                                                              │               
                             │    32 │   │   kwargs_keys = list(kwargs.keys())                                                                           │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/utils.py:1028 in inner │               
                             │                                                                                                                           │               
                             │   1025 │   │   │   │   │   │   stacklevel=3,  # The inner function takes up one level                                     │               
                             │   1026 │   │   │   │   │   )                                                                                              │               
                             │   1027 │   │   │                                                                                                          │               
                             │ ❱ 1028 │   │   │   return fn(*args, **kwargs)                                                                             │               
                             │   1029 │   │                                                                                                              │               
                             │   1030 │   │   return inner  # type: ignore                                                                               │               
                             │   1031                                                                                                                    │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/entrypoints/llm.py:210 │               
                             │ in __init__                                                                                                               │               
                             │                                                                                                                           │               
                             │   207 │   │   self.engine_class = self.get_engine_class()                                                                 │               
                             │   208 │   │                                                                                                               │               
                             │   209 │   │   # TODO(rob): enable mp by default (issue with fork vs spawn)                                                │               
                             │ ❱ 210 │   │   self.llm_engine = self.engine_class.from_engine_args(                                                       │               
                             │   211 │   │   │   engine_args, usage_context=UsageContext.LLM_CLASS)                                                      │               
                             │   212 │   │                                                                                                               │               
                             │   213 │   │   self.request_counter = Counter()                                                                            │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py:5 │               
                             │ 85 in from_engine_args                                                                                                    │               
                             │                                                                                                                           │               
                             │    582 │   │   engine_config = engine_args.create_engine_config()                                                         │               
                             │    583 │   │   executor_class = cls._get_executor_cls(engine_config)                                                      │               
                             │    584 │   │   # Create the LLM engine.                                                                                   │               
                             │ ❱  585 │   │   engine = cls(                                                                                              │               
                             │    586 │   │   │   vllm_config=engine_config,                                                                             │               
                             │    587 │   │   │   executor_class=executor_class,                                                                         │               
                             │    588 │   │   │   log_stats=not engine_args.disable_log_stats,                                                           │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py:3 │               
                             │ 47 in __init__                                                                                                            │               
                             │                                                                                                                           │               
                             │    344 │   │   self.input_processor = input_registry.create_input_processor(                                              │               
                             │    345 │   │   │   model_config)                                                                                          │               
                             │    346 │   │                                                                                                              │               
                             │ ❱  347 │   │   self.model_executor = executor_class(vllm_config=vllm_config, )                                            │               
                             │    348 │   │                                                                                                              │               
                             │    349 │   │   if self.model_config.task != "embedding":                                                                  │               
                             │    350 │   │   │   self._initialize_kv_caches()                                                                           │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/executor/executor_base │               
                             │ .py:36 in __init__                                                                                                        │               
                             │                                                                                                                           │               
                             │    33 │   │   self.speculative_config = vllm_config.speculative_config                                                    │               
                             │    34 │   │   self.prompt_adapter_config = vllm_config.prompt_adapter_config                                              │               
                             │    35 │   │   self.observability_config = vllm_config.observability_config                                                │               
                             │ ❱  36 │   │   self._init_executor()                                                                                       │               
                             │    37 │                                                                                                                   │               
                             │    38 │   @abstractmethod                                                                                                 │               
                             │    39 │   def _init_executor(self) -> None:                                                                               │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/executor/gpu_executor. │               
                             │ py:39 in _init_executor                                                                                                   │               
                             │                                                                                                                           │               
                             │    36 │   │   │   "GPUExecutor only supports single GPU.")                                                                │               
                             │    37 │   │                                                                                                               │               
                             │    38 │   │   self.driver_worker = self._create_worker()                                                                  │               
                             │ ❱  39 │   │   self.driver_worker.init_device()                                                                            │               
                             │    40 │   │   self.driver_worker.load_model()                                                                             │               
                             │    41 │                                                                                                                   │               
                             │    42 │   def _get_worker_kwargs(                                                                                         │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/worker/worker.py:145   │               
                             │ in init_device                                                                                                            │               
                             │                                                                                                                           │               
                             │   142 │   │   │   raise RuntimeError(                                                                                     │               
                             │   143 │   │   │   │   f"Not support device type: {self.device_config.device}")                                            │               
                             │   144 │   │   # Initialize the distributed environment.                                                                   │               
                             │ ❱ 145 │   │   init_worker_distributed_environment(self.parallel_config, self.rank,                                        │               
                             │   146 │   │   │   │   │   │   │   │   │   │   │   self.distributed_init_method,                                           │               
                             │   147 │   │   │   │   │   │   │   │   │   │   │   self.local_rank)                                                        │               
                             │   148 │   │   # Set random seed.                                                                                          │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/worker/worker.py:462   │               
                             │ in init_worker_distributed_environment                                                                                    │               
                             │                                                                                                                           │               
                             │   459 │   init_distributed_environment(parallel_config.world_size, rank,                                                  │               
                             │   460 │   │   │   │   │   │   │   │    distributed_init_method, local_rank)                                               │               
                             │   461 │                                                                                                                   │               
                             │ ❱ 462 │   ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,                                         │               
                             │   463 │   │   │   │   │   │   │   │   │     parallel_config.pipeline_parallel_size)                                       │               
                             │   464                                                                                                                     │               
                             │   465                                                                                                                     │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/distributed/parallel_s │               
                             │ tate.py:1102 in ensure_model_parallel_initialized                                                                         │               
                             │                                                                                                                           │               
                             │   1099 │   or ensure tensor-parallel and pipeline-parallel sizes are equal to expected                                    │               
                             │   1100 │   values if the model parallel groups are initialized.                                                           │               
                             │   1101 │   """                                                                                                            │               
                             │ ❱ 1102 │   backend = backend or torch.distributed.get_backend(                                                            │               
                             │   1103 │   │   get_world_group().device_group)                                                                            │               
                             │   1104 │   if not model_parallel_is_initialized():                                                                        │               
                             │   1105 │   │   initialize_model_parallel(tensor_model_parallel_size,                                                      │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/torch/distributed/distribut │               
                             │ ed_c10d.py:1215 in get_backend                                                                                            │               
                             │                                                                                                                           │               
                             │   1212 │   if _rank_not_in_group(pg):                                                                                     │               
                             │   1213 │   │   raise ValueError("Invalid process group specified")                                                        │               
                             │   1214 │   pg_store = _world.pg_map[pg] if pg in _world.pg_map else None                                                  │               
                             │ ❱ 1215 │   return Backend(not_none(pg_store)[0])                                                                          │               
                             │   1216                                                                                                                    │               
                             │   1217                                                                                                                    │               
                             │   1218 def _get_process_group_uid(pg: ProcessGroup) -> int:                                                               │               
                             │                                                                                                                           │               
                             │ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/torch/utils/_typing_utils.p │               
                             │ y:13 in not_none                                                                                                          │               
                             │                                                                                                                           │               
                             │   10                                                                                                                      │               
                             │   11 def not_none(obj: Optional[T]) -> T:                                                                                 │               
                             │   12 │   if obj is None:                                                                                                  │               
                             │ ❱ 13 │   │   raise TypeError("Invariant encountered: value was None when it should not be")                               │               
                             │   14 │   return obj                                                                                                       │               
                             │   15                                                                                                                      │               
                             ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯               
                             TypeError: Invariant encountered: value was None when it should not be                                                                      
Exception ignored in: <function Vllm.__del__ at 0x7f35f96b6dd0>
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/vllm.py", line 53, in __del__
AttributeError: vllm_model
[rank0]:[W1125 18:06:59.980650822 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
```

**Desktop (please complete the following information):**
 - OS: Ubuntu 20.04
 - Python version: 3.10.13


https://github.com/pytorch/pytorch/issues/132003
It can be a pytorch issue!

> [pytorch/pytorch#132003](https://github.com/pytorch/pytorch/issues/132003) It can be a pytorch issue!

Thank you for providing the link as well. I will take a look at it. 😃 

This issue appears to occur when the PyTorch environment is not completely removed. As a result, the issue arises when the vllm module is called twice. To avoid this, it is recommended to set `skip_validation=True` and call the vllm module only once.

```python
evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir=project_dir)
evaluator.start_trial(config, skip_validation=True)
```
and
```yaml
...
  - module_type: vllm
    llm: meta-llama/Llama-3.2-1B-instruct
    temperature: 0.1
    max_tokens: 128
    max_model_len: 200
    gpu_memory_utilization: 0.3
```

#1013 

@e7217 Hi! I understand the issue:)
If the validation run and pytorch seems not properly not clear its cache.
I added many codes to clear up the cache of pytorch, but it is hard to cover all hardware setups 🤯

First, I think we need to add documentation that skip_validation is recommended to the vllm settings.

@vkehfdl1 
I will make sure to incorporate this into the document. Thank you!