zhiminy/EvalEng huggingface dataset:
Repository: microsoft/promptbench
Issue: About Runtime
Root cause: Documentation Deficiency

Hi！Thanks for your awesome work！When I run the commend `python main.py --model llama-13b --dataset mnli --attack bertattack --shot 0 --generate_len 20` with one NVIDIA 3090 24G, it takes already 20 hours and I don't know when it would finish. The logs is like:
```
2023-07-26 08:31:41,068 - INFO - gt: 2
2023-07-26 08:31:41,068 - INFO - Pred: -1
2023-07-26 08:31:41,068 - INFO - sentence: Assess the connection between the following sentences and count it as 'entailment', 'neutral', or 'contradiction':Premise: He hadn't seen even pictures of such things since the few silent movies run in some of the little art theaters. Hypothesis: He had recently seen pictures depicting those things. Answer: 
```
but the results file only contain a file named mnli which is also empty. Would you please give me a general time consuming estimation with your machines? Thanks a lot!

Hi,

Thank you for your attention!

Attacks such as BertAttack and TextFooler can be extremely time-consuming. I'd recommend starting with the StressTest attack, which may take less than 30 minutes, to verify correctness. For reference, running Vicuna-13B may take over 5 days, possibly even a week, on 2 V100s (16GB).

Thanks for your reply! I use the StressTest Attack as you recommend. However, it takes already one day rather than half an hour, and seems to continue generating. My machine is a NVIDIA 3090 24G, I use the llama-13b (download from hugging face), and I run the command `python main.py --model llama-13b --dataset mnli --attack stresstest  --shot 0 --generate_len 20`  without modifying the source code. I wonder where I can improve to rapidly generate a complete result？Thanks again.

Hi,

The running time issue may be due to two reasons:

1. GPU Memory: llama-13B requires at least 26GB GPU memory, but 3090 only has 24GB, causing CPU loading and slowing down inference. Can you load llama-7b instead and have a try?
2. generate_len: For MNLI, a generate_len of around 5 works well. You can try generate_len=3 first.

Hope this helps!

Thanks for your help! It works and I successfully get the result. 

By the way, I wonder if OPT model is also supported in your source code? Thanks!

@Harry-mic I think you can use any model within this codebase. This is the point of open-source, right? But our experience is that OPT family is often bad at language understanding. You can have a try.

Cause I notice you implement the opt-66b in the optional model set, but it was commented out. That's why I put forward this issue.

This benchmark supports all kinds of open-source and private LLMs. You only need to call their apis in the code. We comment out OPT-66b since it is too large for this study. We then use BLOOM, which has an API in huggingface that we can call. Note again, adversarial attack is expensive and model size matters.
Repository: microsoft/promptbench
Issue: Llama2 adversarial prompts
Root cause: Unimplemented Feature Gap

The prompts for Llama 2 have not been provided in `prompts/adv_prompts`, so running `load_adv_prompt` doesn't work when using Llama 2. Could these be added, please? Thanks!

Hi, thank you for your interest in prompt attacks! We cannot provide Llama2 adversarial prompts as we have only conducted adversarial attacks on Llama1 models. However, you could try using those Llama1 adversarial prompts with Llama2 models, as [our paper](https://arxiv.org/abs/2306.04528) demonstrated their transferability.

Hi, thanks for the reply! Where can the Llama1 adversarial prompts be found? Also, why were adversarial attacks not run on Llama2?

Hi, apologize for the confusion in my previous messages. We actually conducted the adversarial attacks on Llama2, not Llama1. Could you please send an email to kaijiezhu11@gmail.com? This way, I can share the Llama2 adversarial prompts with you directly.
Repository: microsoft/promptbench
Issue: Question-answer template attack

Hey, during the attacks conducted for the paper, was the question answer part of the prompt aka `\nQuestion: {content}\nAnswer:` attacked? Thank you!

Hi, thanks for your interest in prompt attacks! Regarding your question, the behavior you observed depends on the unmodifiable words you've set. Please check [this example notebook](https://github.com/microsoft/promptbench/blob/main/examples/prompt_attack.ipynb) on how to specify unmodifiable words. You could simply add ['Question', 'Answer'] into unmodifiable words.

In the output, you will see entries similar to the following:

```
--------------------------------------------------
Modifiable words:  ['As', 'a', 'sentiment', 'classifier', 'determine', 'whether', 'the', 'following', 'text', 'is', 'or', 'Please', 'classify', 'Question', 'Answer']
--------------------------------------------------
```

This will help you ensure that sections like `\nQuestion: {content}\nAnswer:` will not be attacked during the prompt attack process.

Thanks!
Repository: microsoft/promptbench
Issue: Attack goal function details
Root cause: Documentation Deficiency

Hey, what's the reason behind using the difference in scores for the attack goal function instead of the PDR? 

https://github.com/microsoft/promptbench/blob/3daaf2d3734ec8c08302f73b93e640973a1edd41/promptbench/prompt_attack/goal_function.py#L276-L279

Also, why has `maximizable=True` not been passed to the goal function?

https://github.com/microsoft/promptbench/blob/3daaf2d3734ec8c08302f73b93e640973a1edd41/promptbench/prompt_attack/goal_function.py#L242-L248

1. Using PDR is also beneficial, I think it has a similar impact as the score differences.
2. In the adversarial prompts goal function, the `maximizable` argument is redundant.

Thank you for clarifying!
Repository: microsoft/promptbench
Issue: DyVal can't be used via pypi install
Root cause: Environment Incompatibility

After i installed via pypi and execute examples, i found that
`
No module named 'promptbench.dyval.DAG'
`
I checked pkgs in my environment and I confirmed that this folder is missing in this path. can you solve it?

Hi, thanks for bringing this bug to us! This will be fixed in a few days. Currently you could try directly git clone this repo to use dyval instead.

Stale issue message
Repository: microsoft/promptbench
Issue: About Semantic Attacks Against Vicuna
Root cause: Unimplemented Feature Gap

Hi!

Thanks for your nice work!

One small question is that when I visited your [demo site](https://huggingface.co/spaces/March07/PromptBench), and I choose "Vicuna" + "MNLI" + “Semantic” + “zero-shot task”, the site return nothing. Therefore, I am confused and  could you please replenish these adversarial prompts? Many thanks!

Best wishes

<img width="813" alt="pb-issue" src="https://github.com/microsoft/promptbench/assets/90805377/0342a4f0-a67b-4bb2-8fb9-fc80d4db4a80">


Hi,

Thank you for bringing this error to our attention! We will update the attack results shortly.

Hi!

Thank you very much!

I also noticed that when I selected model as Vicuna, and selected dataset squad_v2，UN Multi, and math, there are also not adversarial prompts.

Yes, Vicuna performs much worse on these datasets, we choose not to evaluate Vicuna on these datasets.
Repository: microsoft/promptbench
Issue: Can promptbench be used to attack samples instead of prompts?
Root cause: Documentation Deficiency

Hi~ 
Thanks to your great contribution to the community.
I would like to try attacking the samples of datasets using promptbench instead of attacking the prompt. Is this feasible? Do you have any suggestions? Thank you so much!

Hi, if you would like to attack samples, you could refer to the [textattack](https://github.com/QData/TextAttack) repo, which is the base of prompt attacks.

Thank you very much for your reply. I am referring to TextAttack. I found TextAttack has a parameter like --pct-words-to-swap to limit the range of perturbations to the text. May I ask, in promptbench, what is the default magnitude of perturbation? Are there any constraints?

Yes, please refer to [this file](https://github.com/microsoft/promptbench/blob/main/promptbench/prompt_attack/__init__.py) where we defines the constraints of different attacks. 

Also, we have the [LabelConstraint](https://github.com/microsoft/promptbench/blob/main/promptbench/prompt_attack/label_constraint.py) where we prevent the attack from modifying certain words (such as the label 'positive' for sentiment analysis).

Thank you very much for solving my problem~
Repository: microsoft/promptbench
Issue: Want to know what code is used to implement the attention visualization in the figure below?
GitHub labels: no-issue-activity

![微信图片_20240913235257](https://github.com/user-attachments/assets/0e2aff1c-ea7b-49aa-91b7-b7ab4ce516c4)


Hi, you could refer to #78, the code is in [this file](https://github.com/microsoft/promptbench/blob/main/promptbench/utils/visualize.py).

Thanks, I already know how to run it, thank you very much

Stale issue message

When I run the code.  It meets a problem. ValueError: Expected input batch_size (22) to match target batch_size (1).
Repository: microsoft/promptbench
Issue: How to evaluate the prompt score, not at the time of attack, but only?
Root cause: Documentation Deficiency

Because the accuracy of my experimental results does not match the paper. Because I want to directly evaluate the prompt after being attacked by text to see if the score is wrong?

Hi, thanks for your interest! You could find the results on this [website](https://huggingface.co/spaces/March07/PromptBench).

Stale issue message
Repository: microsoft/promptbench
Issue: best 3 prompts for each categories

Hi,

Thanks for this great work! Could I ask where I can find the best 3 clean prompts mentioned in the paper for each categories and dataset? I can only find all prompts, but I wish to reproduce the results with only 3 (or just 1 that performs best)..




Hi, it may be a little bit complicate, we have posted the results [here](https://huggingface.co/spaces/March07/PromptBench), you may retrieve the relevant results to reproduce them.

Hi, thanks!
Repository: microsoft/promptbench
Issue: prompt attack
Root cause: Documentation Deficiency

eg
"Determine if the given pair of sentences displays entailment, neutral, or contradiction. Respond with 'entailment', 'neutral', or 'contradiction'\nQuestion: {content}\nAnswer:"
The prompt words Answer and Question cannot be modified. If they are attacked, the accuracy will drop a lot. I did not see this in the prompt words in your conclusion.

Yes, in our setup, terms like ‘entailment’ and ‘Question’ are fixed and unmodifiable because we believe changing them is unnecessary and would cause a significant shift in meaning compared to the original clean prompts.

In this [example](https://github.com/microsoft/promptbench/blob/main/examples/prompt_attack.ipynb), you can define your own unmodifiable words.

Stale issue message
Repository: microsoft/promptbench
Issue: MMLU subset
Root cause: Documentation Deficiency

There are 800+ samples in the MMLU subset (data/mmlu.json). How does it work?

How does it aligns with 564 samples as introduced in paper?

Hi,

We stored the [MMLU validation dataset](https://huggingface.co/datasets/lukaemon/mmlu/viewer/philosophy/validation)  in JSON format. When loading the MMLU dataset, we select **10 samples from the validation set of each task**. However, not all tasks have 10 examples available in the validation set. Due to this, the total number of samples that are measured amounts to 564.

Thanks for the reply.

I mean there are around 870 samples in https://github.com/microsoft/promptbench/blob/main/data/MMLU.json.

Any hints on how it is constructed?

The JSON file has approximately 870 samples from 57 tasks' validation sets according to this [MMLU validation dataset](https://huggingface.co/datasets/lukaemon/mmlu/viewer/philosophy/validation).

Stale issue message
Repository: microsoft/promptbench
Issue: May I ask how you used a large generative model to allow him to accurately answer classification tasks for experiments?
GitHub labels: no-issue-activity

I want to test other models according to your method, but I find it difficult to classify generative models. Please tell me how you solved it?

Stale issue message

You can ask the generative models to output in some certain formats, like <<<True>>>, then you can use regex to parse the answer.

Stale issue message
Repository: microsoft/promptbench
Issue: Load a model fine-tuned with Peft
Root cause: Validation Gap

When I try to load a fine-tuned model with Peft, the follow error issued. What is the correct way to load this kind of model? Is there anyway?
    model = pb.LLMModel(model='maritaca-ai/sabia-7b', max_new_tokens=10, temperature=0.0001, device='cuda', model_dir='C:\\Users\\allan\\Documents\\dissertacao\\fine-tuning\\sabia-assin2')    
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    
  File "C:\Users\allan\Documents\promptbench\promptbench\models\__init__.py", line 84, in __init__
    self.model = self._create_model(max_new_tokens, temperature, device, dtype, model_dir, system_prompt, api_key, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\allan\Documents\promptbench\promptbench\models\__init__.py", line 96, in _create_model
    return model_class(self.model_name, max_new_tokens, temperature, device, dtype, system_prompt, model_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\allan\Documents\promptbench\promptbench\models\models.py", line 301, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(model_dir, device_map=device, torch_dtype=dtype)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\allan\miniconda3\Lib\site-packages\transformers\models\auto\tokenization_auto.py", line 825, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\allan\miniconda3\Lib\site-packages\transformers\tokenization_utils_base.py", line 2048, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\allan\miniconda3\Lib\site-packages\transformers\tokenization_utils_base.py", line 2287, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\allan\miniconda3\Lib\site-packages\transformers\models\llama\tokenization_llama_fast.py", line 133, in __init__
    super().__init__(
  File "C:\Users\allan\miniconda3\Lib\site-packages\transformers\tokenization_utils_fast.py", line 111, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum ModelWrapper at line 277138 column 3

Stale issue message
Repository: microsoft/promptbench
Issue: the PromptAttack is the same to PromptModify?

Hi, thanks for your excellent works! 
Whether I can understand that _promptattack_ is the same to _promptmodify_, I could know which prompt is better before and after modification.

Hi, thanks for your interest in our work!

Yes, prompt attacks and prompt optimization are essentially similar except for their goals. For attack, we want to optimize the prompt to achieve the worst performance while making the adversarial prompts look similar to the original ones, while for prompt optimization methods, they want to find the prompts that increase the model performance.

> Hi, thanks for your interest in our work!您好，感谢您对我们的工作感兴趣！
> 
> Yes, prompt attacks and prompt optimization are essentially similar except for their goals. For attack, we want to optimize the prompt to achieve the worst performance while making the adversarial prompts look similar to the original ones, while for prompt optimization methods, they want to find the prompts that increase the model performance.是的，除了目标不同，即时攻击和即时优化本质上是相似的。对于 attack，我们希望优化 Prompt 以实现最差的性能，同时使对抗性 Prompts 看起来与原始 Prompts 相似，而对于 Prompt 优化方法，他们希望找到提高模型性能的 Prompt。

Thank you very much for your patient reply！
Repository: mir-evaluation/mir_eval
Issue: chord.mirex produces erroneous results on chords with less than three semitones.
Root cause: Validation Gap

The mirex comparison function incorrectly considers chord spellings that result in fewer than three active semitones, e.g. `C:maj(*3)` or `C:5`; this is the same issue that caused #98, but the fix will need to be different.

The seemingly logical thing to do would be the following, when comparing a reference `R` and an estimation `E`:
- if sum(R.semitones) < 3 and R != 'N', -> -1
- else -> mirex(R, E)
Repository: mir-evaluation/mir_eval
Issue: numpydoc format docstrings
GitHub labels: enhancement

**tldr: changing the docstring format will make integration easier.**
## Background

I've implemented a [first cut](https://github.com/marl/jams/commit/4413d2dfe3192d875e04d16dbf1fb2b2ce0686b1) at JAMS<->mir_eval integration, as described in [this comment thread](https://github.com/marl/jams/issues/16) (paging @justinsalamon).

The basic idea here is that jams provides wrappers for the top-level mir_eval `evaluate()` functions, which do some basic sanity checking and schema validation on the input (JAMS) objects, and then convert them into mir_eval-friendly inputs.  This is implemented through a handful of decorator functions which get applied to each submodule's `evaluate`.  This is almost totally seamless, except for the docstrings.
## why should I have to change the docstring format, you jerk?

A few reasons this would be a good idea:
1. [numpydoc](https://github.com/numpy/numpydoc) format is more human-readable than raw sphinx
2. the compiled sphinx output looks the same (or better)
3. fits in with the numpy/scipy ecosystem
4. it would let JAMS's eval decorators use numpydoc's parsing functions to rip apart the docstring, modify the `Parameters` section, and put it back together.  Any changes to mir_eval's comments will propagate upstream automatically, and everything looks nice to the user.


Go for it!


... damn.

lil help?


https://github.com/dadadel/pyment ?


Nope, pyment doesn't correctly parse the current docstrings.

... but it gets pretty close, so i'll fix it with some regexps.


074eeb3f423d10ff9591aa395c59b5c40908607e


:+1:
Repository: mir-evaluation/mir_eval
Issue: Use Travis CI

Since you already have tests it is generally a good idea to have [Travis CI](https://travis-ci.org/craffel/mir_eval) to actually run them for every commit and pull request.

A simple `.travis.yml` could look like (not tested)

```
language: python
python:
    - "2.7"
    - "3.3"
    - "3.4"
virtualenv:
    system_site_packages: true
install:
    - sudo apt-get update -qq
    - if [[ $TRAVIS_PYTHON_VERSION == *"2."* ]]; then sudo apt-get install python-numpy python-scipy; fi
    - if [[ $TRAVIS_PYTHON_VERSION == *"3."* ]]; then sudo apt-get install python3-numpy python3-scipy; fi
    - pip install -e .
    - pip install nosetests sphinx
script:
    - nosetests
    - python setup.py build_sphinx
    - python setup.py egg_info -b.dev sdist --formats gztar
```


Yes, this is something we've been meaning to add!  Thanks for this.  We currently aren't officially supporting Python 3 (#74).  Otherwise, do you want to create a PR for this?


I'd love to. I suggest you added this repo to Travis first, so my PR will immediately be tested.


OK, I added this repo to travis.


I've implemented it all in PR #113. Sorry for the other messy PRs before :-)
Repository: mir-evaluation/mir_eval
Issue: io.load_delimited fails for unicode strings
Root cause: Validation Gap

It's easy to work around this problem, but I'll mention it regardless. When a file path that is passed to io.load_delimited is a unicode string, a value error is raised.

```
$ nonexistent_file = u"fake_filepath.csv"
$ mir_eval.io.load_delimited(nonexistent_file, ',')
ValueError: filename must be a string or file handle
```


I didn't even know filesystems supported unicode filenames.  The fix would be to change https://github.com/craffel/mir_eval/blob/develop/mir_eval/input_output.py#L54 to 

```
    if type(filename) == str or type(filename) == unicode:
```


> if type(filename) == str or type(filename) == unicode:

Two things:

1) use `isinstance`, not type equivalence.  The code above won't handle subclasses of str or unicode, but you probably should support that.

2) `six` provides a sane version of this through `six.stringtypes`.  see [here](https://pythonhosted.org/six/)


Ok, if/when we add `six` as a dependency (cc https://github.com/craffel/mir_eval/pull/113#discussion_r26895157 ) we should do `isinstance(six.stringtypes)`.


Even if you're really dead-set against adding dependencies, six is intentionally self-contained as a single file that you can drop into your codebase.  Scipy does this (see: scipy/_lib/six.py).


I'm totally fine with adding `six` as a dependency.


Full support to the idea of adding six as a dependency.

On Fri, Apr 24, 2015 at 2:22 PM, Colin Raffel notifications@github.com
wrote:

> I'm totally fine with adding six as a dependency.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/117#issuecomment-96024502.


Addressed in #135
Repository: mir-evaluation/mir_eval
Issue: Sonification
GitHub labels: enhancement

Not exactly a metric, but it would be nice to have something which, e.g., synthesized beeps at a list of times and added them back to the original audio to perform "evaluation by ears", because not all of us use sonic visualizer.  Similarly, we could synthesize chord labels, chromagrams, etc.
- [x] Time indices (beats, onsets, segments)
- [x] Piano roll/time-frequency representation inversion
- [x] Chord labels
- [x] Chroma sonification


Added methods for synthesizing times (e.g. onsets, beats) and time-frequency matrices (e.g. piano rolls, chroma).  It would still be useful to be able to synthesize chord labels.


To create a chord bitmask, use mir_eval.chord.encode and rotate the semitone vector by the root.


Hey @ejhumphrey check out chord label synthesis!


Pretty sweet! Making my life easier too as I sanity check my own systems....

On Thu, Jul 24, 2014 at 6:32 PM, craffel notifications@github.com wrote:

> Closed #12 https://github.com/craffel/mir_eval/issues/12.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/12#event-145375364.