zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: Error related to incomplete model downloads in cache
Root cause: Validation Gap

When a model download is halted before it completes, and then a new command is used referring to the same model (e.g. the default `comet score -s src.de -h hyp.en -r ref.en` , the script will try to retrieve the cached (incomplete) download and will result in an error:
`Exception: [meta_tags.csv|hparams.yaml is missing from the checkpoint folder.`

It is resolved is the cache is cleared

Full error trace:

```

 Traceback (most recent call last):
  File "/home/chryssa/anaconda3/bin/comet", line 11, in <module>
    load_entry_point('unbabel-comet==0.0.7', 'console_scripts', 'comet')()
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/unbabel_comet-0.0.7-py3.7.egg/comet/cli.py", line 121, in score
    model = load_checkpoint(model) if os.path.exists(model) else download_model(model)
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/unbabel_comet-0.0.7-py3.7.egg/comet/models/__init__.py", line 98, in download_model
    return load_checkpoint(checkpoint_path)
  File "/home/chryssa/anaconda3/lib/python3.7/site-packages/unbabel_comet-0.0.7-py3.7.egg/comet/models/__init__.py", line 136, in load_checkpoint
    "[meta_tags.csv|hparams.yaml is missing from the checkpoint folder."
```
Repository: Unbabel/COMET
Issue: pip update
Root cause: Unimplemented Feature Gap

I would like to use [this commit](https://github.com/Unbabel/COMET/commit/d4a7a6443dd866b9eb55646eea671e676c93c8de) in my work, but it is not distributed through pip yet. Can we get a version bump and new version on pip?

Thanks!

Hi Bram, yes Ill do it tomorrow. thanks!

Hi Bram, 

Sorry for taking a bit more time than expected. Can you check if version 1.1.3 is working well on your end?

I had to also bump the python version. Python 3.7 is not compatible with new pandas versions and old pandas versions do not work on new Apple M1 computers.

From version 1.1.3 forward we will only support python >=3.8.



if everything is working feel free to close the issue.

Thanks for contributing!


Hey Ricardo. I just upgraded (on Ubuntu, for the moment, just to check) and it seems to work great. Thanks!
Repository: Unbabel/COMET
Issue: Batch size in words
Root cause: Unimplemented Feature Gap

## 🚀 Feature
being able to specify the batch size in terms of tokens could allow larger batch sizes.

### Motivation

When training on data with various sentence lengths, increasing a batch size can break the training (especially when there is a language that is tokenized to too many tokens). Specifying batch size in terms of tokens, could fix this issue and allow better usage of the GPU
Repository: Unbabel/COMET
Issue: Bootstrap resampling
Root cause: Documentation Deficiency

## 🚀 Feature
COMET could support statistical significance tests such as paired bootstrap resampling for comparing across systems.

### Motivation
Statistical significance tests, e.g. paired bootstrap resampling are encouraged as standard practice in research. Currently they are easily available for calculated metrics (BLEU, chrF etc.) via [sacrebleu](https://github.com/mjpost/sacrebleu). However, their approach does not support custom metrics.

You can use `comet-compare` command for that.

It returns the comparison of 1 or more systems using statistical significance.

Leaving here the [example from README](https://github.com/Unbabel/COMET#:~:text=When%20comparing%20multiple%20MT%20systems%20we%20encourage%20you%20to%20run%20the%20comet%2Dcompare%20command%20to%20get%20statistical%20significance%20with%20Paired%20T%2DTest%20and%20bootstrap%20resampling%20(Koehn%2C%20et%20al%202004).)

`comet-compare -s src.de -t hyp1.en hyp2.en hyp3.en -r ref.en`
Repository: Unbabel/COMET
Issue: Unable to find hparam file for a custom model
Root cause: Validation Gap

## 🐛 Bug

When I am passing a custom model for generating the score, the command fails with the following error.
```
Exception: hparams.yaml file is missing!
```
I tried copying a `hparams.yaml` file to the same directory as the model but the score command fails with the same error.

I've then looked at the code to understand where it is checking for the path and tried to have a similar directory structure that is expected.
https://github.com/Unbabel/COMET/blob/0d58039da7861bebd920c890d857e93ec3f8ea12/comet/models/__init__.py#L61

This fixed the issue for me.

### To Reproduce 

Run comet-score command with a custom model, but no `hparams` file in the parent directory.

### Expected behaviour
Generate the comet-score with custom models, with(or may be without specifying) `hparams.yaml`

### Possible Solutions
- [Not sure if possible] Pass default hparams if `hparams.yaml` file is not found
- Update the docs to specify that there should be a `hparams.yaml` file in the parent directory of the custom model
- Ability to specify `hparams.yaml` file as a param to command line

which version are you using? and what is the pytorch lightning version you have installed?
Repository: Unbabel/COMET
Issue: Can't install v1.2.0
Root cause: Environment Incompatibility

## 🐛 Bug
I think v1.2.0 was released, then rolled back as a pre-release. I was able to install it a day or two ago, but this is no longer possible.

### To Reproduce 

```
pip install --upgrade pip  # ensures that pip is current 
pip install unbabel-comet==1.2.0
```

### Expected behaviour

unbabel-comet v1.2.0 is installed

### Screenshots

![image](https://user-images.githubusercontent.com/6475564/205463636-cdc81175-32b7-40d9-acf8-6d3fe6b0292e.png)


### Environment
OS: `Linux 4e9e05fda7ba 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 GNU/Linux`
Packaging: `pip 22.3.1 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)`
Version 

### Additional context

Only a day or two ago, I was able to install with poetry & pip.


There's a workaround, of course:

```
pip install -e git+https://github.com/Unbabel/COMET.git@v1.2.0#egg=unbabel-comet
```
but it would be nice if the latest version were in the registry. 

Hi @dblandan I released version 1.2.0 but just for a few hours and then decided to roll-back. We want to release new `default` models but we want to avoid confusion and people reporting scores that are not comparable with previous versions. To avoid that I need to finish documentation on new models first.


Sorry by the confusion! Also, entmax was breaking the install and I asked Ben Peters if they could bump entmax, I see that it was fixed already!
Repository: Unbabel/COMET
Issue: Update to entmax 1.1 to fix poetry error
Root cause: External Dependency Breakage

## 🚀 Feature

There is a known issue with poetry installing entmax 1.0.0. The COMET documentation recommends using pip to install torch, then install entmax after torch is installed. It is preferable to use poetry without additional pip commands required.

### Motivation

entmax v1.1 was released yesterday to address this issue.

### Alternatives

As noted above, the alternative is the current workaround of pip install + poetry.

### Additional context

https://github.com/deep-spin/entmax/pull/25
Repository: Vchitect/VBench
Issue: conda environment issue
Root cause: Environment Incompatibility

ERROR: Could not find a version that satisfies the requirement detectron2==0.6 (from versions: none)
ERROR: No matching distribution found for detectron2==0.6
can't pip install detectron2==0.6

Hi, you can try `pip install detectron2@git+https://github.com/facebookresearch/detectron2.git`

TY, but it still can't work

We've updated the installation instruction: https://github.com/Vchitect/VBench#hammer-installation Please let us know if you still encountered problems during installation.

TY for your great work!

It appears that the issue has been resolved, and the current issue is about to be closed. If you still have any questions, please feel free to reopen this issue at any time. 😊
Repository: Vchitect/VBench
Issue: FileNotFoundError: [Errno 2] No such file or directory: './VBench_full_info.json' 
Root cause: Documentation Deficiency

When using vbench installed with pip install, the error popped up.

I'm sorry you encountered this issue, you should download [VBench_full_info.json](https://raw.githubusercontent.com/Vchitect/VBench/master/VBench_full_info.json) and place it in the running directory.

[VBench_full_info.json](https://raw.githubusercontent.com/Vchitect/VBench/master/VBench_full_info.json)  is 404: Not Found

Hi, thanks for letting us know. The link is updated to: https://github.com/Vchitect/VBench/blob/master/vbench/VBench_full_info.json
The READMD is also updated.
Repository: Vchitect/VBench
Issue: so how to compute the long-prompt's human action?
Root cause: Documentation Deficiency


Videos longer than 10.0 seconds are segmented into 10.0-second clips. Since most videos are currently under 10.0 seconds, they are typically evaluated in the same way as standard VBench videos.



Closing the issue for now. Feel free to reopen it if you feel it's not resolved.
Repository: Vchitect/VBench
Issue: error: can't implement motion_smoothness successfully
Root cause: Unimplemented Feature Gap

2025-01-26 01:41:44,642 - torch.distributed.distributed_c10d - INFO - Added key: store_based_barrier_key:1 to store for rank: 0
2025-01-26 01:41:44,643 - torch.distributed.distributed_c10d - INFO - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
args: Namespace(output_path='./evaluation_results/', full_json_dir='/vlm/feilongtang/mnt/sda/feilongtang/Video_Generation/FIFO-Diffusion_public/vbench/VBench_full_info.json', videos_path='/vlm/feilongtang/mnt/sda/feilongtang/Video_Generation/FIFO-Diffusion_public/results/videocraft_v2_fifo/random_noise/A beautiful sunset over the ocean, 4K, ultra HD./200frames/fifo.gif', dimension=['subject_consistency', 'background_consistency', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', 'imaging_quality'], load_ckpt_from_local=None, read_frame=None, mode='custom_input', prompt='None', prompt_file=None, category=None, imaging_quality_preprocessing_mode='longer')
start evaluation
Evaluation meta data saved to ./evaluation_results/results_2025-01-26-01:41:44_full_info.json
cur_full_info_path: ./evaluation_results/results_2025-01-26-01:41:44_full_info.json
Using cache found in /root/.cache/torch/hub/facebookresearch_dino_main
2025-01-26 01:41:49,617 - vbench.subject_consistency - INFO - Initialize DINO success
100%|████████████████████████████████████████████████████████████████████████| 1/1 [00:47<00:00, 47.75s/it]
cur_full_info_path: ./evaluation_results/results_2025-01-26-01:41:44_full_info.json
100%|████████████████████████████████████████████████████████████████████████| 1/1 [00:33<00:00, 33.59s/it]
cur_full_info_path: ./evaluation_results/results_2025-01-26-01:41:44_full_info.json
Loading [networks.AMT-S.Model] from [/root/.cache/vbench/amt_model/amt-s.pth]...
  0%|                                                                                | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/vlm/feilongtang/mnt/sda/feilongtang/Video_Generation/FIFO-Diffusion_public/evaluate.py", line 153, in <module>
    main()
  File "/vlm/feilongtang/mnt/sda/feilongtang/Video_Generation/FIFO-Diffusion_public/evaluate.py", line 139, in main
    my_VBench.evaluate(
  File "/root/miniconda3/envs/fifo/lib/python3.9/site-packages/vbench/__init__.py", line 153, in evaluate
    results = evaluate_func(cur_full_info_path, self.device, submodules_list, **kwargs)
  File "/root/miniconda3/envs/fifo/lib/python3.9/site-packages/vbench/motion_smoothness.py", line 188, in compute_motion_smoothness
    all_results, video_results = motion_smoothness(motion, video_list)
  File "/root/miniconda3/envs/fifo/lib/python3.9/site-packages/vbench/motion_smoothness.py", line 175, in motion_smoothness
    score_per_video = motion.motion_score(video_path)
  File "/root/miniconda3/envs/fifo/lib/python3.9/site-packages/vbench/motion_smoothness.py", line 119, in motion_score
    raise NotImplementedError
NotImplementedError

and my tree structure just like this:

![Image](https://github.com/user-attachments/assets/54e6e0d1-69c9-46b8-9343-63ef83893aa0)
Repository: Vchitect/VBench
Issue: Does vbench ask the videos all with same resolution and duration?
Root cause: Documentation Deficiency

Does vbench ask the videos all with same resolution and duration?
Does the leaderboard result all for same resolution and duration?
The difficulty of different duration and resolution video generation is various.

You can find details about the models here: https://github.com/Vchitect/VBench/tree/master/sampled_videos#what-are-the-details-of-the-video-generation-models

Each model is allowed to generate videos with its own duration and resolution. We recommend using the optimal settings for each individual model.

Closing the issue now. Feel free to reopen it if you feel it's not resolved.
Repository: Vchitect/VBench
Issue: Bug in evaluating "color"
Root cause: External Dependency Breakage

Thanks for your work. When I evaluated "color" dimension, the 
https://datarelease.blob.core.windows.net/grit/models/grit_b_densecap_objectdet.pth URL has the following error:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>PublicAccessNotPermitted</Code>
<Message>Public access is not permitted on this storage account. RequestId:ed36f76d-e01e-006a-3e9b-78824b000000 Time:2025-02-06T13:33:15.8489994Z</Message>
</Error>


How can I solve this problem?


You can try using https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/grit_b_densecap_objectdet.pth first
Repository: Vchitect/VBench
Issue: cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known”
Root cause: Environment Incompatibility


Because the network exit does not support IPv6 ，Is there an alternative solution?
carol@XPITDN0502269 VBench_Test % source myenv/bin/activate
(myenv) caorl@XPITDN0502269 VBench_Test % vbench evaluate --dimension "imaging_quality" --videos_path "/Users/xmly/VBench_Test" --mode=custom_input
[2025-02-09 15:27:54,009] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.
[W socket.cpp:697] [c10d] The IPv6 network addresses of (XPITDN0502269.local, 56412) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).



From the logs, it looks like the error is related to your local network configuration. 

According to ChatGPT:
![Image](https://github.com/user-attachments/assets/d2511653-ab99-48f7-9ab0-749057e8c555)

Closing the issue for now. Feel free to reopen it if you feel it's not resolved.
Repository: Vchitect/VBench
Issue: Public access is not permitted
Root cause: External Dependency Breakage

Hi!

Thank you for the work. I encountered a problem while running the evaluation. When running (vbench/utils.py", line 344) the terminal will show the following error (partial error message):
https://datarelease.blob.core.windows.net/grit/models/grit_b_densecap_objectdet.pth
ERROR 409: Public access is not permitted on this storage account..

When I try to access the given link directly, the showing page is as follows:
![Image](https://github.com/user-attachments/assets/2c8d24e8-98b3-411d-8a75-ad1b4ce8fac3)

Would you please help me with this issue? Thank you in advance!

same question


same here, could people that have downloaded this model put a copy on hugging face ? Thanks!  Currently I am not able to evaluate any new models with vbench  :-(

@tianweiy @yhd-ai @xxn1850051  

Sorry, since the original third-party link is inaccessible, we have uploaded a backup of the model to huggingface, which you can use to continue reviewing on VBench by replacing the following link. [GRiT checkpoint](https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/grit_b_densecap_objectdet.pth)

https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/grit_b_densecap_objectdet.pth

> [@tianweiy](https://github.com/tianweiy) [@yhd-ai](https://github.com/yhd-ai) [@xxn1850051](https://github.com/xxn1850051)
> 
> Sorry, since the original third-party link is inaccessible, we have uploaded a backup of the model to huggingface, which you can use to continue reviewing on VBench by replacing the following link. [GRiT checkpoint](https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/grit_b_densecap_objectdet.pth)
> 
> https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/grit_b_densecap_objectdet.pth

Thank you for your help!
Repository: Vchitect/VBench
Issue: Cannot find and process videos under provided directory
Root cause: Documentation Deficiency

I run the following
`vbench evaluate --videos_path ./demo/videos --dimension temporal_flickering`

and get the following

```
args: Namespace(func=<function evaluate at 0x7f3b34fd6950>, output_path='./evaluation_results/', full_json_dir='./VBench_full_info.json', videos_path='./demo/videos', dimension='temporal_flickering', load_ckpt_from_local=None, read_frame=None)
start evaluation
Evaluation meta data saved to ./evaluation_results/temporal_flickering_full_info.json
0it [00:00, ?it/s]
```

It seems it cannot find the videos under the directory

Hi, thanks for your interest in our work!
If the generated JSON does not contain any results, VBench may not be able to find the corresponding video possibly due to an incorrect sampling video name.  The solution can be referenced in [this issue](https://github.com/Vchitect/VBench/issues/4#issuecomment-1916100751).

When testing the `temporal_flickering` dimension, there should be videos with corresponding formatted names in your `./demo/videos` folder. For example, with the first prompt `In a still frame, a stop sign`, there should be [`In a still frame, a stop sign-0.mp4`,..., `In a still frame, a stop sign-4.mp4`].

Forget this issue, it's the same with my latest issue, same question and the answer is write new `Vbench_full.json` for your own data.
Repository: Vchitect/VBench
Issue: Temporal flickering
Root cause: Documentation Deficiency

Is it necessary to use static filtering in evaluating temporal flickering？ I got an empty folder “temporal_filtered_cilps”. Does it mean that I failed to evaluate this metric?



Yes, the temporal_flickering dimension requires static filtering before evaluation. This step is aimed at selecting video segments that are close to static for the evaluation. Typically, more video samples need to be collected for this dimension to meet the filtering requirements.
Repository: Vchitect/VBench
Issue: Error when I run evaluation
Root cause: Documentation Deficiency

My command:
`vbench evaluate --dimension overall_consistency --videos_path ./output.mp4 --mode=custom_input --prompt "A video of a cat."`

Error report:
`usage: evaluate.py [-h] [--output_path OUTPUT_PATH] [--full_json_dir FULL_JSON_DIR] --videos_path VIDEOS_PATH --dimension DIMENSION [DIMENSION ...]    
                   [--load_ckpt_from_local LOAD_CKPT_FROM_LOCAL] [--read_frame READ_FRAME] [--mode {custom_input,vbench_standard,vbench_category}]     
                   [--prompt PROMPT] [--prompt_file PROMPT_FILE] [--category CATEGORY]                                                                 
                   [--imaging_quality_preprocessing_mode IMAGING_QUALITY_PREPROCESSING_MODE]                                                           
evaluate.py: error: unrecognized arguments: video of a cat.`

Was getting the same error, we don't need the ``--prompt`` and the string followed by it.

So something like this:
``vbench evaluate --dimension overall_consistency --videos_path ./output.mp4 --mode=custom_input``
Repository: Vchitect/VBench
Issue: i2v_background evaluate failed
Root cause: External Dependency Breakage

  File "/home/anaconda3/envs/qxm_cogv_080/lib/python3.10/site-packages/vbench2_beta_i2v/__init__.py", line 36, in evaluate
    results = evaluate_func(cur_full_info_path, self.device, submodules_list, **kwargs)
  File "/home/anaconda3/envs/qxm_cogv_080/lib/python3.10/site-packages/vbench2_beta_i2v/i2v_background.py", line 70, in compute_i2v_background
    dream_model, preprocess = dreamsim(pretrained=True)
  File "/home/anaconda3/envs/qxm_cogv_080/lib/python3.10/site-packages/dreamsim/model.py", line 293, in dreamsim
    ours_model = PeftModel.from_pretrained(ours_model.base_model.model, load_dir).to(device)
  File "/home/anaconda3/envs/qxm_cogv_080/lib/python3.10/site-packages/peft/peft_model.py", line 323, in from_pretrained
    config = PEFT_TYPE_TO_CONFIG_MAPPING[
  File "/home/anaconda3/envs/qxm_cogv_080/lib/python3.10/site-packages/peft/config.py", line 139, in from_pretrained
    config = config_cls(**kwargs)
TypeError: LoraConfig.__init__() got an unexpected keyword argument 'layer_replication'
[ERROR] 2025-03-07-18:04:49 (PID:1120966, Device:0, RankID:-1) ERR99999 UNKNOWN application exception


dreamsim==0.2.1
peft==0.7.1

You can try upgrading peft to  0.14.0, along with transformers==4.48.3 and torch==1.13.1
Repository: Vchitect/VBench
Issue: vbench2_beta_i2v/data/crop.zip unavailable on Google Drive
Root cause: Environment Incompatibility

Hi!

When I run `sh vbench2_beta_i2v/download_data.sh` and it tries to `gdown --id 1JANXpTxg90M3Exi5WGnVNagb1nqyTJ4o --output vbench2_beta_i2v/data/crop.zip` I get this error:

```
Failed to retrieve file url:

        Cannot retrieve the public link of the file. You may need to change
        the permission to 'Anyone with the link', or have had many accesses.
        Check FAQ in https://github.com/wkentaro/gdown?tab=readme-ov-file#faq.

You may still be able to access the file from the browser:

        https://drive.google.com/uc?id=1JANXpTxg90M3Exi5WGnVNagb1nqyTJ4o

but Gdown can't. Please check connections and permissions.
```

The other two files `vbench2_beta_i2v/data/origin.zip` and `vbench2_beta_i2v/data/i2v-bench-info.json` download just fine.

Could you look into this file sharing settings and fix them?

P.S. Yesterday I was able to download all three files.

Thanks,
Piotr

NVM, I was on a wrong branch. Your `origin/master` works fine 😄 

Thank you for raising this issue. We have recently updated the crop.zip file to include images with a 3:2 ratio. Additionally, we have updated the download_data.sh script. You can check the details here: [link](https://github.com/Vchitect/VBench/blob/master/vbench2_beta_i2v/download_data.sh)