zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: SIGTERM Error during videomme and mlvu evaluation
Root cause: Resource Mishandling

Thank you for sharing such a great codebase.

I ran the following command:

accelerate launch --num_processes=4 --main_process_port=12345 -m lmms_eval \
  --model qwen2_vl \
  --model_args pretrained=my_model,device_map=cuda \
  --tasks videomme \
  --batch_size 1 \
  --log_samples \
  --log_samples_suffix qwen2_vl_dedug_vinoground \
  --output_path ./logs_videomme
However, I encountered the following error:

W0504 08:39:36.777000 2950364 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2950432 closing signal SIGTERM

I am using 4 A6000 GPUs, and there was sufficient GPU memory available at the time of execution.
Interestingly, this issue only occurred on videomme and mlvu tasks.
When I ran other tasks such as mvbench, the evaluation completed successfully without any issues.

Could you help identify the possible cause?



Did face similar issue, solved by increasing RAM to 500G/using only 2 GPUs

Thanks for sharing your experience. I actually tried increasing the RAM to 500G, but unfortunately, the server shut down completely during execution. Still, I appreciate the suggestion!

Has this issue been resolved? I'm still encountering this error on mvbench.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [New Model] Are there any plans to add Gemma 3 to the harness?
Root cause: Unimplemented Feature Gap

Is it planned to include Gemma 3 support at any point? It is a very decent image-text processor and I am not sure any of the current implementations fit the peculiarities of the model architecture. If there aren't such plans I would glad to hear nay recommendations on how to approach the evaluation of Gemma 3 on image(+text) to text benchmarks such as MMMU, DocVQA, etc.

I want to approach the evaluation of Gemma 3 on multimodal benchmarks too. Did you find any ways to evaluate Gemma 3?

Same here! Also curious about how to make it work.
I used the command
```powershell
lmms-eval --model huggingface --model_args pretrained=google/gemma-3-4b-it --tasks docvqa --log_samples --output_path ./result --verbosity=DEBUG
``` 
but it keeps telling me that
```
AttributeError: 'Gemma3Model' object has no attribute 'generate'
```
it might be I wasn't using the right argument since I'm new to this tool. 
Does anyone have some ideas??
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Model will be loaded on different devices when using multiple gpus.
Root cause: Configuration Error

It appears that models will be loaded on different gpus when setting `num_processes` to more than one, which will cause error:
`RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!`

Here's my command to launch:
```
accelerate launch --num_processes=2 -m lmms_eval --model llava   --model_args pretrained="xxx,conv_template=xxx"   --tasks gqa,vqav2,scienceqa,textvqa --batch_size 1 --log_samples --log_samples_suffix xxx --output_path ./logs/
```

I found a temporary fix by installing previous version:
`pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git@bf4c78b7e405e2ca29bf76f579371382fec3dd02`
and in this version multi-gpu inference works fine.

May I ask in which line of inference did this error occur? 

Sorry for the delay.

Here is one error message:
```
[lmms_eval/models/llava.py:386] ERROR Error Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution) in generating.
```


You might also want to try setting device_map=auto in your model_args when you do multi-processing

`--model_args pretrained=xxx,conv_template=xxx,device_map=auto`

Setting device_map to auto didn't do the trick. Here's my command:
```
srun -p xxx --gres=gpu:4 accelerate launch --num_processes=4 --main_process_port 19500 -m lmms_eval --model llava   --model_args pretrained="xxx,conv_template=xxx,device_map=auto"   --task textvqa_val,vizwiz_vqa_val,mmbench_en --batch_size 1 --log_samples --log_samples_suffix llava_hermes2_llama3_merged_data_v1.1_anyres_tune_vit --output_path ./logs/ #
```

I noticed one difference between evaluation using `v0.1.2` and `bf4c78b7e405e2ca29bf76f579371382fec3dd02` was this logger information:
`v0.1.2`:`[lmms_eval/models/llava.py:124] INFO Using single device: cuda`
`bf4c78b7e405e2ca29bf76f579371382fec3dd02`: `lmms_eval/models/llava.py:104] INFO Using 4 devices with data parallelism`

Line 104 appears to be [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/bf4c78b7e405e2ca29bf76f579371382fec3dd02/lmms_eval/models/llava.py#L104).

Sorry, my bad

Should set `device_map=""` when using multiprocess. Set `device_map=auto` only when you use num_processes=1

Thanks. Now it works!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Why does my process hang if I am using multiple GPUs
Root cause: Configuration Error

I have a recurring issue when I use any model for inference, where it hangs on multiple occasions. On a few occasions, it completes running the inference. This is the warning I get.

```
[rank1]:[W511 10:03:36.323250251 ProcessGroupNCCL.cpp:4715] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
[rank0]:[W511 10:03:36.453588556 ProcessGroupNCCL.cpp:4715] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
```

Could you advise a quick fix?



I think you are not having 2 GPUs when you running the job?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: question about score evaluation
Root cause: Documentation Deficiency


Hi, thank you for your great work on this project.

I noticed that in many task evaluations, only an exact match with the ground truth is considered correct. For example, answers like "The answer is yes" or "yes." are treated as incorrect because they do not exactly match "yes".

Is this the intended behavior?
Thanks in advance!

Hello! Yes, in order to stay as consistent as possible with the original evaluation results of the benchmarks, we conduct evaluations using the matching algorithms provided when each benchmark was released.

However, for these benchmarks, there should be a reasonable matching design, such as incorporating appropriate prompts. For example:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/c0c0a97775c2a398f220edfe4f3146a5a50d1b38/lmms_eval/tasks/worldqa/worldqa_mc.yaml#L13-L16
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Implmentation of Video Frame Sampling of Qwen2.5VL
Root cause: Architectural Constraint

I notice that [this part](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/a5cc3450be83eb3089cb10c72561efef40cdac04/lmms_eval/models/qwen2_5_vl.py#L276) make a further sampling after qwen2.5 vl adaptive frame sampling (in `process_vision_info()`).

It seems that this additional frame sampling will change the sampling fps, resulting different time stamps, which may influence video ground tasks.

Im curious about why this operation should be executed. Anyone could help me out?

Yes. And the default value of the max_num_frames parameter for Qwen2.5VL should be changed to 768 instead of 32.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Is it normal that the Qwen-2.5-VL result in the MME Benchmark test is significantly lower than InternVL3?
Root cause: Configuration Error

![Image](https://github.com/user-attachments/assets/69377867-8b5d-4ac8-9822-86e4f4fcb896)
Hello, as shown in the picture, for the mme benchmark, I used the official qwen-2.5-vl.sh and my own internvl3.sh to test, but found that the result of qwen-2.5-vl is much lower than that of internvl3. Is this result normal?

I have same question！！！

Please check if there are format errors of Qwen 2.5 VL's output.

I would also try to run a check to validate Qwen-2.5's results!

2025-05-24 05:06:14.018 | INFO     | lmms_eval.loggers.evaluation_tracker:save_results_aggregated:239 - Output path not provided, skipping saving results aggregated
qwen2_5_vl (pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,use_flash_attention_2=True,interleave_visuals=False), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|Filter|n-shot|       Metric       |   |  Value  |   |Stderr|
|-----|-------|------|-----:|--------------------|---|--------:|---|------|
|mme  |Yaml   |none  |     0|mme_cognition_score |↑  | 645.7143|±  |   N/A|
|mme  |Yaml   |none  |     0|mme_perception_score|↑  |1678.0348|±  |   N/A|

My script is:

```shell
accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,use_flash_attention_2=True,interleave_visuals=False \
    --tasks mme \
    --batch_size 1
```

Maybe the mistake is from `interleave_visuals=True`.

It's used to control the visual token position for multi-image (interleaved image-text) tasks, currently only for mmmu_val and mmmu_pro (and potentially for other interleaved image-text tasks), please do not use it unless you are sure about the operation details.

Hi, @PrLeung . Please allow me to ask how do you evaluate InternVL3? I want to do some evaluations on InternVL3 as well.

how can i run internvl3.py, i didn'tn found the script of the second.

> how can i run internvl3.py, i didn'tn found the script of the second.

Does `internvl3` support vllm inference as backend, if so you could use vllm to run any supported models.

> > how can i run internvl3.py, i didn'tn found the script of the second.
> 
> Does `internvl3` support vllm inference as backend, if so you could use vllm to run any supported models.

yes, support. and i run it use internvl2 script just now, it works.emm , i wanted delete the comment just now and see your reply.
Another, vllm didn't support gemma-3n, so what shuoud i do, thank you.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Support multiple inference instance run on a single GPU
Root cause: Unimplemented Feature Gap

Does lmms-eval support to run multiple process on a single GPU?
For 7/8 B models, it usually the case where models occupy less than 40G RAM (for qwen2/2.5) and 30G for others. And the UTL is also low, considering the gap between samples.
Is there a way to run more process than the num of physical gpu?

I think the most possible way to do so is vllm backend currently where you can send multiple requests at the same time
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: lmms_eval Task not find
Root cause: Unimplemented Feature Gap

I am trying to evaluate the Qwen2.5-VL model on the VDC benchmark using the lmms-eval framework, but I encountered an error during execution.
Here is the command I used:
accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval
--model qwen2_5_vl
--model_args=pretrained=/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,use_flash_attention_2=True,interleave_visuals=True
--tasks vdc_short
--batch_size 1
--log_samples
--log_samples_suffix vdc_shot
--output_path my_PATH
and I use lmms_eval-eval --tasks {list_groups,list_subtasks,list_tags,list} can not find VDC-related task. Could you please let me know where I might have gone wrong? Thank you very much for your help!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Development Roadmap (LMMs-Eval 0.4)
GitHub labels: documentation

discussion

## [LMMs-Eval] Update plan


# Main Features

- [x]  Adding agentic evaluation environments, tool-call feature.
- [x]  Replace `doc_to_visual` and `doc_to_text` with `doc_to_messages` @kcz358 
- [x]  Unified Interface for LLM/LMM as a judge @Luodian 
- [ ]  Multi node evaluation
- [ ]  Cached Requests
- [ ]  Auto TP for models @Trung Quang
- [x]  Integrate with nanoVLM.
    - [x]  [[huggingface/nanoVLM: The simplest, fastest repository for training/finetuning small-sized VLMs.](https://github.com/huggingface/nanoVLM?tab=readme-ov-file)](https://github.com/huggingface/nanoVLM?tab=readme-ov-file)
- [ ]  Insights Feature
    - [ ]  Can perform analysis to model’s correct/in-correct patterns.
- [ ]  New Benchmarks
    - [x]  [Video-Eval-Pro](https://arxiv.org/abs/2505.14640)(https://arxiv.org/abs/2505.14640) @Luodian  (PR from the community)
    - [ ]  [TVBench](https://openreview.net/forum?id=DrNN5qx66Z)(https://openreview.net/forum?id=DrNN5qx66Z)
    - [x]  V*
    - [x]  VLM are Blind
    - [ ]  ZeroBench
    - [ ]  VisuLogic
    - [ ]  MMVP
    - [x]  HallusionBench
    - [ ]  SimpleVQA
    - [ ]  CharXiv (RQ)
    - [ ]  CharXiv (DQ)
    - [x]  VisualWebBench
    - [ ]  LVIS-MG
    - [ ]  FSC-147
    - [ ]  DA-2K
    - [ ]  NYU-Depth V2
    - [ ]  All-Angles Bench
    - [ ]  MotionBench
    - [x]  TOMATO
    - [ ]  Dream-1K
    - [ ]  TemporalBench
    - [ ]  OVBench
    - [ ]  OVOBench
    - [ ]  StreamBench
    - [x]  MMVU
    - [ ]  Charades-STA
    - [ ]  TACoS
    - [ ]  ScreenSpot-V2
    - [ ]  ScreenSpot-Pro
    - [ ]  OSWorld
    - [ ]  Windows Agent Arena
    - [ ]  WebVoyager
    - [ ]  Online-Mind2Web
    - [ ]  Android World
    - [ ]  Poki games (from Seed VL 1.5)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: When using video dataset tasks, 'cache_dir' isn't set to use local cache
Root cause: Configuration Error

Hi, thank you for your awesome project.
I recently wanted to modify the yaml file to use my local dataset since I already have them on my machine.
So I set the 'cache_dir' in the 'dataset_kwargs' and noticed that the lmms-eval is still trying to fetch dataset from huggingface instead of using the local one.

![Image](https://github.com/user-attachments/assets/135ec935-c845-4034-b57b-4f7d11be5fc9)

Hi! I'm experiencing the same problem. How do you solve it?

> Hi! I'm experiencing the same problem. How do you solve it?

Hi, if you have assigned 'cahce_dir' in your YAML, simply modify the 'task.py' to pass in 'cahce_dir' to the huggingface download function.
This has got fixed, simply refer to this commit: https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/690

Thanks for your answer! I have another question: Like the video mlvu dataset, the files downloaded by huggingface are multiple zip files, which need to be decompressed into mp4 before they can be read. After I decompressed them, the zip files in the 'blobs/' directory are useless, but I can't delete them, which consumes a lot of disk space (because the 'snapshot_download' function always checks the contents under blobs). Is there any way to delete the useless zip files?

> Thanks for your answer! I have another question: Like the video mlvu dataset, the files downloaded by huggingface are multiple zip files, which need to be decompressed into mp4 before they can be read. After I decompressed them, the zip files in the 'blobs/' directory are useless, but I can't delete them, which consumes a lot of disk space (because the 'snapshot_download' function always checks the contents under blobs). Is there any way to delete the useless zip files?

Same question, do you solve it? @suliangxu
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: ValueError on distributed_executor_backend 'accelerate' when using 1 process with qwen2.5-vl
Root cause: Validation Gap

### Description

When running Qwen2.5-VL with `distributed_executor_backend='accelerate'` and `num_processes=1`, the following error is raised:
```
ValueError: Invalid distributed_executor_backend: accelerate. Choose either 'accelerate' or 'torchrun'.
```

After some investigation, the root cause appears to be that in the model initialization (in my case, qwen2_5_vl), `self.accelerator` is only assigned if `accelerator.num_processes > 1` [in here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/755e4e49c43f83bfb0d12097b2cad27d699711e4/lmms_eval/models/qwen2_5_vl.py#L111). This leads to `hasattr(lm, "accelerator") == False` [in here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/755e4e49c43f83bfb0d12097b2cad27d699711e4/lmms_eval/evaluator.py#L663), and consequently raises the error despite accelerate being a valid backend and the script being launched with accelerate launch.


### Suggested Fix

Although this can be fixed in the model class, the evaluator code shouldn't depend on a potentially missing attribute.

Instead, it may be better to directly use `Accelerator().wait_for_everyone()` to ensure consistent behavior whether running in single-process or multi-process mode independent of the model implementation:
```Python
from accelerate import Accelerator

...

if distributed_executor_backend == "accelerate":
    Accelerator().wait_for_everyone()

...
```

It seems to work for both `num_processes == 1` and `num_processes > 1` cases.

Yeah, I think a great way is to create an accelerator object in the main pipeline when backend is `accelerate`, do you mind create a PR for that?

Great! thanks @miikatoi
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: refcoco dataset with qwen2vl
Root cause: Configuration Error

I ran the refcoco dataset. Currently, only 64 samples are specified, and it doesn't feel right.


<img width="1170" alt="Image" src="https://github.com/user-attachments/assets/d1be2df3-170e-430f-ac65-f01d07e64628" />

lines in log

<img width="862" alt="Image" src="https://github.com/user-attachments/assets/6b079a8d-245e-4ca6-a2a8-bc2c09816ef9" />

<img width="837" alt="Image" src="https://github.com/user-attachments/assets/fe7dbde0-cab4-4378-bd69-bd1b9c9cc9c0" />

how did you fix it
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Performance gap between python and vLLM backends using the same Qwen2-VL-2B-Instruct model on ChartQA task
Root cause: Interface Contract Mismatch

Hi, team,

I’ve been running lmms_eval on the ChartQA benchmark using the Qwen/Qwen2-VL-2B-Instruct model under two different settings, and I’m seeing significant discrepancies in performance, despite using the same model.


1. Standard Python version
```
python3 -m lmms_eval \
    --model=qwen2_vl \
    --model_args=pretrained=Qwen/Qwen2-VL-2B-Instruct,device_map=cuda \
    --tasks=chartqa \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=qwen2_vl \
    --output_path="./logs1"
```

Results: 
| Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|-------|-------|------|-----:|-----------------------|---|-----:|---|-----:|
|chartqa|Yaml   |none  |     0|relaxed_augmented_split|↑  |0.8912|±  |0.0088|
|chartqa|Yaml   |none  |     0|relaxed_human_split    |↑  |0.5552|±  |0.0141|
|chartqa|Yaml   |none  |     0|relaxed_overall        |↑  |0.7232|±  |0.0090|

2. vLLM version
```
python3 -m lmms_eval \
    --model vllm \
    --model_args model_version=Qwen/Qwen2-VL-2B-Instruct,tensor_parallel_size=4 \
    --tasks chartqa \
    --batch_size 300 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs2
```

Results:
| Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|-------|-------|------|-----:|-----------------------|---|-----:|---|-----:|
|chartqa|Yaml   |none  |     0|relaxed_augmented_split|↑  |0.6752|±  |0.0133|
|chartqa|Yaml   |none  |     0|relaxed_human_split    |↑  |0.3208|±  |0.0132|
|chartqa|Yaml   |none  |     0|relaxed_overall        |↑  |0.4980|±  |0.0100|

## Issue:
Even though both configurations use Qwen/Qwen2-VL-2B-Instruct, I got lower performance results when I used the vLLM version.

Is there any known limitation or additional configuration needed when using the vllm backend?


I am unsured about the reason about this. Are there similar issue about the result difference in vllm about this?

@mikittt Hi, did you solve it? I had the same problem.

@LiamLian0727 
Sorry, but I am not yet sure of the cause.

I got better performance on another benchmark for Qwen2.5-VL-32B-Instruct when using vLLM `(python3 -m lmms_eval \
    --model vllm \)`

> I am unsured about the reason about this. Are there similar issue about the result difference in vllm about this?我不确定造成这种情况的原因。vllm 中是否存在类似的结果差异问题？

我怀疑vllm有相关部分是否存在一些bug，我测试大多数benchmark，都是vllm会低一点，尤其是videomme和mvbench，用vllm得分会低3-5%
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to evaluate generation tasks in MLVU?
Root cause: Unimplemented Feature Gap

Thanks for your wonderful work on MLLM evaluation. I have a question about MLVU evaluation.

MLVU has two types of testing data, Multiple-Choice testing and Generation testing https://github.com/JUNJIE99/MLVU/blob/main/evaluation/README.md. 

`sub_scene` and `summary` tasks are included in generation testing. How to evaluate generation tasks of MLVU in lmms-eval? It seems that in the current MLVU yaml config (`mlvu_dev.yaml`), it only contains the Multiple-Choice testing.


I think in lmms-eval. The test setting is generation

In the MLVU utility file https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/mlvu/utils.py#L107, it seems that `sub_scene` and `summary` tasks are not supported.

@shuyansy Hi, can you take a look about this issue? Thanks
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: While this lmms-eval support V*Bench?

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

- [ ] A descriptive title: [xxx] XXXX
- [ ] A detailed description

Thank you for your contributions!


https://arxiv.org/abs/2312.14135
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Resolving Evaluation Issues with Custom OPENAI_API_BASE in Model Assessment
Root cause: Configuration Error

I want to evaluate a model compatible with OpenAI API on the mathvista_test dataset, so I set custom `OPENAI_API_BASE` and `OPENAI_API_KEY`. During execution, I found no issues in the model inference process. However, during the evaluation metrics, lmms_eval automatically uses GPT for evaluation, which results in an error when calling GPT for assessment. The reason is that I have already set a custom `OPENAI_API_BASE`, which does not contain GPT-related models. How can I resolve this issue? Thanks in advance.

Solved. `OPENAI_API_URL` is the link used for evaluation, while `OPENAI_API_BASE` is the link for model inference, so they can be distinguished. Additionally, you can achieve distinction by directly modifying the URL in the evaluation code of the corresponding dataset, such as changing the value of `API_URL` in lmms_eval/tasks/mathvista/mathvista_evals.py.

Hi, when testing an OpenAI-compatible API, how to perform parallel testing? Setting the `batch-size` parameter seems to have no effect. The `accelerate launch --num_processes` command is limited by the number of GPUs on the machine. Thx.

> Hi, when testing an OpenAI-compatible API, how to perform parallel testing? Setting the `batch-size` parameter seems to have no effect. The `accelerate launch --num_processes` command is limited by the number of GPUs on the machine. Thx.

If you mean parallel API calling, you can try to set the `--api-nproc` parameter.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Bug] MM prompt too long during vllm generation
Root cause: Environment Incompatibility

Hi, lmms-eval team. Thanks for your work.

I encountered a "prompt too long" issue when generating with vllm_qwen2_vl.sh. I don't revise anything. Could you help with that?


```python
ValueError: The decoder prompt (length 337896) is longer than the maximum model length of 128000. Make sure that `max_model_len` is no smaller than the number of text tokens plus multimodal tokens. For image inputs, the
 number of image tokens depends on the number of images, and possibly their aspect ratios as well.
````

I tested my side and it works. Do you have more ideas? 

Can you try to set `transformers==4.52.4`? I think vllm didnt work correctly with latest version transformer

![Image](https://github.com/user-attachments/assets/304cd044-b505-41d1-bbc5-abe194897a97)