zhiminy/EvalEng huggingface dataset:
Repository: jitsi/jiwer
Issue: Apparent WER bug?
Root cause: Documentation Deficiency

Love the package, thanks for all the hard work!

I came across what I think is a WER calculation bug:

```
import jiwer

jiwer.process_words('the cat went to the store', 'the car went to store front').wer
# 0.5

jiwer.process_words('the cat went to the store', 'the car went to green store').wer
# 0.3333333333333333
```

Given that WER = S + D + I / N, shouldn't the second example also have a WER of 0.5? i.e. one substitution (cat -> car), one deletion (the -> None), and one insertion (None -> green).

In the second example, the *minimal* edit distance includes two substitution:

```
calculation = jiwer.process_words('the cat went to the store', 'the car went to green store')
print(jiwer.visualize_alignment(calculation))
REF: the cat went to   the store
HYP: the car went to green store
           S             S      
```

Your solution

```
REF: the cat went to the ***** store
HYP: the car went to *** green store
           S           d     i           
```
is a valid way to change the hypothesis into the reference, but requires more operations than necessary.



Thank you! Makes complete sense.
Repository: jitsi/jiwer
Issue: jiwer gives an error when passed a very long list of strings
Root cause: Validation Gap

## Issue

When passing a very long list of strings (>350k strings) as the reference and hypothesis, jiwer gives the following error:

`chr() arg not in range(0x110000)`

What's been tried:
- Calculating wer on individual list elements - this works successfully with no error
- Splitting the large lists into smaller chunks - this works successfully with no error
- Passing the entire list to another library such as fastwer - this works successfully with no error

The error only seems to happen when the entire long list is passed into jiwer. 

## Additional Context

It seems like the vocabulary in the `_word2char` function isn't built properly. After adding words from the first N sentences in the list, words from rest of the sentences do not seem to be a part of the vocabulary. This results in the `chr() arg not found` error when [these lines](https://github.com/jitsi/jiwer/blob/7dfd00850aa9b0d46f14c2a0c82804cf0070270d/jiwer/process.py#L400C5-L405C6) are executed.

Jiwer version - `v3.0.3`

Thanks for reporting! Would you be able to share the length of the vocabulary object when generated from your input?

Yep, it's 4484. 

I cannot reproduce it with the following toy data :(

```python3
import random
import string

from typing import List

import jiwer


def random_word(low=2, high=10, rng=random.Random()) -> string:
    word = ""

    for i in range(rng.randint(low, high + 1)):
        word += rng.choice(string.ascii_lowercase)

    return word


def generate_sentence(vocabulary: List[str], low=1, high=12, rng=random.Random()):
    sentence = []

    for i in range(rng.randint(low, high + 1)):
        sentence.append(rng.choice(vocabulary))

    return " ".join(sentence)


NUM_SENTENCE = 500_000
NUM_WORDS = 5000

print('generating vocab...')
vocabulary = list(set([random_word() for _ in range(NUM_WORDS)]))
print(len(vocabulary))

print("generating reference...")
ref = [generate_sentence(vocabulary) for _ in range(NUM_SENTENCE)]
print("generating hypotheses...")
hyp = [generate_sentence(vocabulary) for _ in range(NUM_SENTENCE)]
print("calculating wer...")
print(jiwer.wer(ref, hyp))
```
Can you share the word which fails to be included in the vocabulary?


The words which are not included are normal english words - words from entire sentences aren't included like "Australia", "he", "run", etc. 

Some sentences in my list also include numbers like "1", "10", so on and can also include non-english characters at time too. Could this be a potential cause of the issue?

I think the size is not an issue, I think it's a specific sentence-pairing which fails. When you tested chunks of the dataset, did those chunks still span the entire range of reference/hypothesis pairs? 

Also, do you use a custom transform, or do you use the default?

@nikvaessen when I test chunks, the chunks do span the entire range of the pairs. I have also tried finding wer by looping over one pair at a time, that also works. 

I'm using the default transform. 

Resolved by release 3.1.0.
Repository: jitsi/jiwer
Issue: jiwer.wer(outputs_true, outputs_pred, standardize=True)
Root cause: External Dependency Breakage

Traceback (most recent call last):
File "main.py", line 222, in
main(0, args)
File "main.py", line 121, in main
model.fit(dataset_train,
File "EfficientConformer-master/models/model.py", line 344, in fit
raise e
File "EfficientConformer-master/models/model.py", line 303, in fit
wer, truths, preds, val_loss = self.evaluate(dataset, val_steps, verbose_val, eval_loss=True)
File "EfficientConformer-master/models/model.py", line 425, in evaluate
batch_wer = jiwer.wer(outputs_true, outputs_pred, standardize=True)
TypeError: wer() got an unexpected keyword argument 'standardize'

The keyword `standardize` was removed since version 2.5.0. Eithet set `jiwer<=2.4`, or change the call:

```
import jiwer

jiwer.wer(
  outputs_true, 
  outputs_pred,
  reference_transform=jiwer.wer_standardize, 
  hypothesis_transform=jiwer.wer_standardize
)
```
Repository: jitsi/jiwer
Issue: Is it possible just to get the number of errors?
Root cause: Documentation Deficiency

Is it possible just to get the number of errors?

I know I could probably just get the wer and multiply by the number of words to get the number errors, but I was hoping that was unnecessary.

Edit: The reason I am asking is this... I want to rollup all of my sentence WER into an overall document WER.

If you use [`jiwer.process_words`](https://jitsi.github.io/jiwer/reference/process/#process.process_words) you get a [`WordOutput`](https://jitsi.github.io/jiwer/reference/process/#process.WordOutput) object. The number of errors would be the sum of substitutions, insertions, and deletions available in this object.

If you want an overall WER of a document, you can use [`wer_contiguous`](https://jitsi.github.io/jiwer/reference/transformations/#transformations.wer_contiguous) transform instead. This allows the number of references and hypothesis sentences to differ.

For example:
```
import jiwer
jiwer.process_words(
    reference,
    hypothesis,
    reference_transform=jiwer.wer_contiguous,
    hypothesis_transform=jiwer.wer_contiguous,
)
```

I asked chatGPT, and here is the code it generated:

```
import jiwer

# Assuming 'original_sentences' and 'corrected_sentences' are your lists of sentences
original_sentences = [...]  # Your list of original sentences
corrected_sentences = [...]  # Your list of corrected sentences

# Calculate total WER for the document
total_wer = jiwer.wer(original_sentences, corrected_sentences)

print(f"Total WER for the document: {total_wer}")
```

Doesn't that seem easier?

EDIT: ~~That's giving me a number greater than one.~~
Actually I was using the wrong columns to calculate this.  It looks good now. Thanks!
Repository: jitsi/jiwer
Issue: [CLI] Allow CER calculation with global alignement
Root cause: Architectural Constraint

Instead of forbidding the `-g` flag usage when the `--cer` option is used,
would that make sense to use the `cer_contiguous` transformation strategy in `process_characters` instead ?

Something like the following instead of [current behavior](https://github.com/jitsi/jiwer/blob/master/jiwer/cli.py#L104-L121) :
```python
# [...]
    if compute_cer:
          out = jiwer.process_characters(
              reference_sentences,
              hypothesis_sentences,
          )
          if global_alignment:
              out = jiwer.process_characters(
                  reference_sentences,
                  hypothesis_sentences,
                  reference_transform=jiwer.cer_contiguous,
                  hypothesis_transform=jiwer.cer_contiguous,
              )
          else:
              out = jiwer.process_characters(
                  reference_sentences,
                  hypothesis_sentences,
              )
# [...]
```



Resolved  by 547a6e3d735079c624ee895821795fb8409aad57, version 3.0.4 should include the fix.
Repository: jitsi/jiwer
Issue: Add option to split alignment output based on maximum character length

This would be useful for looking at alignments of long sentences, e.g.:
```python
    out = jiwer.process_words(
        "This is a very long sentence that is much longer than the previous one or the one before that",
        "This is a very loong sentence that is not much longer than the previous one or one before that",
        )

    print(jiwer.visualize_alignment(out, max_chars=80))
```

Would output:
```txt
sentence 1
REF: This is a very  long sentence that is *** much longer than the previous one
HYP: This is a very loong sentence that is not much longer than the previous one
                        S                    I

REF: or the one before that
HYP: or *** one before that
          D
```
Repository: jitsi/jiwer
Issue: WIL and MER
Root cause: Unimplemented Feature Gap

Additionally to WER, WIL (Word Information Lost) and MER (Match Error Rate), can be good performance measures for ASR. I could not find any pypi packages that support those measures, and think they might be a good fit for **jiwer**.

See:
- [Automatic Speech Recognition
Errors Detection and Correction: A Review](https://www.researchgate.net/profile/Rahhal_Errattahi/publication/324427003_Automatic_Speech_Recognition_Errors_Detection_and_Correction_A_Review/links/5ad462590f7e9b2859365080/Automatic-Speech-Recognition-Errors-Detection-and-Correction-A-Review.pdf)
- [From WER and RIL to MER and WIL: improved evaluation
measures for connected speech recognition](https://pdfs.semanticscholar.org/8516/531ff3bd874b66b811f0bd4e21a2d6b10e54.pdf)

I'd be happy to open a pull request with those measures added, I you think they could fit this package.

Feel free to make a PR :)

@MiniXC Did you ever get around to do some coding for this? I'd be interested to use these measures too.
Repository: jitsi/jiwer
Issue: Alignment (visualization?) seems to be broken for languages that are written right-to-left 
Root cause: Validation Gap

Inputting:

import jiwer

ref = 'לשמוליק הייתה כבשה ולה לא היה כלב יופי'

hyp = 'איזה כיף לשמוליקה אתה כבשה ולהלהלה כלב'

out = jiwer.process_words(ref, hyp)

print(jiwer.visualize_alignment(out))

outputs:

REF: לשמוליק הייתה     כבשה ולה   לא     היה כלב יופי
HYP:    איזה   כיף לשמוליקה אתה כבשה ולהלהלה כלב ****
           S     S        S   S    S       S        D


Which misses insertions, substitutions and deletions...

Is there a way to handle this?
Would you be interested in a PR?

Thanks
Yonatan.
Repository: jitsi/jiwer
Issue: SyntaxError when importing jiwer
Root cause: Validation Gap

Since Python 3.12 (Ubuntu 24) when trying to import jiwer it fails with the SyntaxError:

```
 /usr/local/lib/python3.12/site-packages/my_package/my_module.py:14: in <module>
     import jiwer
 /usr/local/lib/python3.12/site-packages/jiwer/__init__.py:1: in <module>
     from .measures import *
 /usr/local/lib/python3.12/site-packages/jiwer/measures.py:49: in <module>
     from jiwer import transforms as tr
 E     File "/usr/local/lib/python3.12/site-packages/jiwer/transforms.py", line 270
 E       """
 E       ^^^
 E   SyntaxError: invalid escape sequence '\w'
```


Can you verify whether the current main branch fixed the issue? I think you can install it with pip via `pip install git+https://github.com/jitsi/jiwer.git@master`

Yes it's fixed!

Should be fixed with version 3.0.5 on pipy.
Repository: jitsi/jiwer
Issue: utterance ref cannot be empty ?
Root cause: Validation Gap

Hello,
is there a good reason why an utterance ref is required to be non-empty ?
https://github.com/jitsi/jiwer/blob/9db6e4649dfff1e91de5640e224ea51de01b0a50/jiwer/process.py#L158C1-L159C69

IMHO, i'd expect that it can be empty (sclite behavior). 
It is a valid situation, if the utterance in test set contains just silence, it's reference is empty,
and the ASR system should produce an empty string and not hallucinate any symbol.

I hacked it accordingly here:
https://github.com/KarelVesely84/jiwer/tree/allow_empty_ref

Best regards
Karel Vesely




My reasoning at the time was that evaluation datasets like `test-clean` of Librispeech do not have silent utterances, so it is better to fail fast and let the user know they made a mistake (like substituting the reference and hypothesis list).

Ok, would you be open to changing the behavior ?

Yes, do you think a `UserWarning` is more appropriate? I think with systems like Whisper, it is valid to test empty reference strings...

Yes, the `UserWarning` would be good. 
It would warn the user in the log, and it would not stop the WER calculation.

Do you know how `sclite` handles the edge-case where we only consider one utterance, with an empty reference? This leads to a division by 0. 

Not sure how sclite treats that case.

Anyway, this is unlikely to happen, as sclite is typically used with test-sets with touhsands of utterances.
To have a credible WER for an ASR system, certain amount of words/utterances is necessary in the ref.

With 1 utterance, 0 ref-word edge case, you are right, that this leads to division by zero. 

So the WER sholud be be `Inf` or `NaN`, i guess, in that case ?
(so that it is mathematically ok, according to the definition of `WER = (S + D + I) / #REF`)
Repository: jitsi/jiwer
Issue: What are the meanings of 'hypothesis' and 'reference' ？

The parameter "hypothesis" is understood by me as the result of speech recognition, and "reference" as the actual sentence.
Therefore, the minimum edit distance calculated should be the distance from "hypothesis" to "reference". However, upon testing the existing code, the edit distance is from "reference" to "hypothesis". Could this be a bug?

You are right, a reference is the gold standard, while the hypothesis is a prediction from the system. You want to measure how many deletions, insertions, or substitutions need to be made FROM the prediction TO the gold standard, in order to know how many mistakes the systems made. The other way around, if you think about it, does not make sense, why should you want to modify the gold standard?

The most important aspect in the calculation is that N in the WER formula S+D+I/N is the length of the reference, and not the hypothesis, which is accomplished by calculating the distance from the hypothesis to the reference.
Repository: microsoft/promptbench
Issue: Chatgpt
Root cause: Unimplemented Feature Gap

I want to replicate the results in the paper using the chatgpt model. But `predict_by_openai_api` method raises an error, i.e. NotImplementedError.



Hi,

For the utilization of ChatGPT, please refer to the instructions on the [OpenAI API website](https://platform.openai.com/docs/introduction) to construct the predict function in inference.py.

@shayan823 We do not offer code for ChatGPT api since calling OpenAI API is considered the common approach for today's LLM research (you can easily find it on OpenAI website). 

Stale issue message
Repository: microsoft/promptbench
Issue: Understanding the output
Root cause: Documentation Deficiency

I have made some experiments and I wondering why original score is the same as PDR metric? Here the output of my experiment:

{'original prompt': 'Classifique o texto como positivo, negativo ou neutro. \ntexto:{content} \nclasse:', '**original score**': 0.75, 'attacked prompt': 'Investigue o texto como positivo, negativo ou neutro. \ntexto:{content} \nclasse:', 'attacked score': 0.7, '**PDR**': 0.75}

Can you help me indicating the way to calculate the PDR programatically inside Promptbench?
Thanks in advance.




Stale issue message
Repository: microsoft/promptbench
Issue: Potentially a bug in `LabelConstraint`
Root cause: Validation Gap

Hello,
Firstly, thanks for your contribution and open-sourcing you code!

I got the following error trying to reproduce an attack for the SST2 dataset:
```
  File "<PATH>/promptbench/prompt_attack/label_constraint.py", line 12, in <listcomp>
    self.labels = [label.lower() for label in labels]
AttributeError: 'int' object has no attribute 'lower'
```

Which makes sense since in the config the labels consist integers:
```python
    'sst2': ['positive', 'negative', 'positive\'', 'negative\'', '0', '1', '0\'', '1\'',0, 1], 
```
But in the `LabelConstraint` constructor we have:
```python
    def __init__(self, labels=[]):
        self.labels = [label.lower() for label in labels]
```

The changes were made in [this PR](https://github.com/microsoft/promptbench/commit/3c3c8d9de619911fee59b3adc600921c6d307210).



To make it run we can simply delete the integers labels `0, 1` - tho I'm afraid the ground truth labels might be integers.
So another solution could be to apply`.lower()` only on string labels (which to me sounds like the correct approach).


What do you guys think?


Hi,

Thank you for bringing this potential error to our attention. We will address it right away. Initially, we considered using integer labels, but later decided not to use them. Removing them should suffice.

Thanks once again!

Stale issue message
Repository: microsoft/promptbench
Issue: transferibility app error
Root cause: Validation Gap

Error in transferibility app from chatgpt to Vicuna

![image](https://github.com/microsoft/promptbench/assets/8182738/93cd436c-d98c-4e9e-9eaa-06e40e5b028e)


This must be a bug. We'll fix this later.

Hi,

Thank you for bring this bug to our attention, now the bug is fixed :)

Thanks for the swift fix.
Repository: microsoft/promptbench
Issue: LLaMa 2 inference 
Root cause: Validation Gap

I tried to benchmark LLaMa 2 chat model from HuggingFace and got a `ValueError`:
`ValueError: `temperature` (=0) has to be a strictly positive float, otherwise your next token scores will be invalid.
`
It is caused by [this line](https://github.com/microsoft/promptbench/blob/23676ee00d2df64a266fc28fbd23590b56fb6a67/inference.py#L342)

Hi @jubick1337 , we are rewriting the code for PromptBench to support more models and make the code easier to use. Check the dev branch: https://github.com/microsoft/promptbench/tree/dev

Hi Jubick,

To address this issue, you can adjust the temperature setting to an extremely small value, such as temp=0.000001. This should be sufficient for the required adjustments :)

Stale issue message
Repository: microsoft/promptbench
Issue: Typo in the doc

![image](https://github.com/microsoft/promptbench/assets/8592144/f9cab745-5a00-482d-a9dc-00728b230ba7)


![image](https://github.com/microsoft/promptbench/assets/8592144/0fa4eedc-c242-4826-905a-04755f4bbc5c)


Thanks a lot for pointing out this typo! We have fixed it.
Repository: microsoft/promptbench
Issue: For the few shot setting, how many are used exactly?

It seems there is a discrepancy between the paper and the code in terms of shot configuration.

Hi zhimin, in our experiments, we randomly select three examples in the training set of a task and append them to a prompt. You can refer to [Sec 2.1](https://arxiv.org/pdf/2306.04528.pdf) for details.

> Hi zhimin, in our experiments, we randomly select three examples in the training set of a task and append them to a prompt. You can refer to [Sec 2.1](https://arxiv.org/pdf/2306.04528.pdf) for details.

Thanks for your quick reply. That is 3-shot, however, what I found from your script seems to be 10-shot: https://github.com/microsoft/promptbench/blob/36254a9ec15f929e03e1c9ad9cf6249f3b29c508/promptbench/prompts/role_oriented.py#L49 and https://github.com/microsoft/promptbench/blob/36254a9ec15f929e03e1c9ad9cf6249f3b29c508/promptbench/prompts/role_oriented.py#L90

Any idea?

The link you provide are 10 candidate prompts for each dataset, as you can see there is no 'examples' in these prompts :)

You can find 3-shot examples in [this file](https://github.com/microsoft/promptbench/blob/main/promptbench/prompts/few_shot_examples.yaml).

> The link you provide are 10 candidate prompts for each dataset, as you can see there is no 'examples' in these prompts :)
> 
> You can find 3-shot examples in [this file](https://github.com/microsoft/promptbench/blob/main/promptbench/prompts/few_shot_examples.yaml).

Yeah, I found this file as well. But I cannot find anything related to `MMLU`. Does `MMLU` evaluation use this file actually?

Thank you for highlighting this issue. 

There are two ways:
1. You can directly download the training set for MMLU from Hugging Face and select three examples for your use.
2. In the former version (before Oct. 1), the few-shots examples are stored in data/MMLU_fewshot. You can download it [here](https://github.com/microsoft/promptbench/blob/db72882693dc1a4a456ae8a24e718e099ba11fd4/data/MMLU_few_shot.json). The code to fetch examples is:
```
    with open("data/MMLU_few_shot.json", "r") as file:
        self.few_shot_data = json.load(file)  

    def get_few_shot_examples(self, task):
        content = "Here are three examples.\n"
        data = self.few_shot_data[task]
        for idx in range(min(len(data), 3)):
            content +=  ("Input: " + data[idx]["input"] + "\n" \
                        + "A : " + data[idx]["A"] + "\n" \
                        + "B : " + data[idx]["B"] + "\n" \
                        + "C : " + data[idx]["C"] + "\n" \
                        + "D : " + data[idx]["D"] + "\n\n" \
                        + "Answer : " + data[idx]["target"] + "\n" \
                        )

        return content
```
Repository: microsoft/promptbench
Issue: Some datasets are not available
Root cause: Environment Incompatibility

I'm not sure if i'm missing something, but I tried loading the MMLU, SQuADv2 - but it always gives me an error (the json file is empty)

It does seem to fail to fetch from the URL provided [here](https://github.com/microsoft/promptbench/blob/32c058b2c3dd807a274cfafc6a06d8a1b54dd362/promptbench/dataload/dataset.py#L59), since the `data` folder does not exist

```
        # check if the dataset exists, if not, download it
        self.filepath = os.path.join(self.data_dir, f"{dataset_name}.json")
        self.filepath2 = os.path.join(self.data_dir, f"{dataset_name}.jsonl")
        if not os.path.exists(self.filepath):
            if os.path.exists(self.filepath2):
                self.filepath = self.filepath2
            else:
                url = f'https://wjdcloud.blob.core.windows.net/dataset/promptbench/dataset/{dataset_name}.json'
                print(f"Downloading {dataset_name} dataset...")
                response = requests.get(url)
                with open(self.filepath, 'wb') as f:
                    f.write(response.content)
```


Thanks for highlighting this issue. We have fixed this in the latest version. please use the following command to clone the repository: `git clone git@github.com:microsoft/promptbench.git`. And then, you can use `dataset = pb.DatasetLoader.load_dataset("mmlu")` to download it from HuggingFace.

Im still facing the same issue, cloned the latest commit `1958b662cc8a866119f350779de39e5d6203b660`

Could you please give the detailed error messages and a MRE? Thanks.
<img width="1442" alt="image" src="https://github.com/microsoft/promptbench/assets/31989262/bb21164c-78eb-4979-a507-0d30ebcbcf36">


I'm sorry, i just cloned afresh and tried again - its working now, thanks for the quick help!

Delighted to hear it's resolved! Please don't hesitate to reach out for any further issues.