zhiminy/EvalEng huggingface dataset:
Repository: jitsi/jiwer
Issue: is it possible to get the exact word insertion/deletion for the WER
Root cause: Unimplemented Feature Gap

I wonder if I can get the words that need to be deleted or inserted to get the wer?
Assuming I use wer(['i','am','g'],['i','am','f']). that it will tell me that there is a problem with the 3rd word.

I think it's possible by using the shortest path found by wagner-fisher algorithm. But it's not implemented in this library.

You can get that by using `Levenshtein.editops`, that's what jiwer is using in [`_get_operation_counts`](https://github.com/jitsi/jiwer/blob/4fb7c4107438ff469375fbf256f8c12ee60b2884/jiwer/measures.py#L255). May this function could be made public?

I would choose to either include it in [jiwer.compute_measures()](https://github.com/jitsi/jiwer/blob/4fb7c4107438ff469375fbf256f8c12ee60b2884/jiwer/measures.py#L149)

or provide a new public method which hides the complexity but offers this functionality. Something like `jiwer.compute_corrections()`?



Both are good, thanks!

Some news about this please? 

This feature has been implemented by #35 and released in version 2.2.0!
Repository: jitsi/jiwer
Issue: pip installation is failing 
Root cause: Environment Incompatibility

```
Collecting jiwer
  Downloading https://files.pythonhosted.org/packages/c7/fd/88639901195f2625941efdf2a1496c540b3390149
9a986fb271af28e4436/jiwer-1.3.2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/sausharma/temp/pip-build-sMieMl/jiwer/setup.py", line 6, in <module>
        with open(path.join(this_directory, 'README.md'), encoding='utf-8') as f:
    TypeError: 'encoding' is an invalid keyword argument for this function
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /home/sausharma/temp/pip-build-sMieMl/
jiwer/
```

ooh so it required python3, dependency isn't mentioned anywhere
Closing the issue
Repository: jitsi/jiwer
Issue: ReplaceWords transform is broken
Root cause: Algorithmic Error

There are a few issues with the `ReplaceWords` transform. Most importantly, due to the way it's implemented, it actually alters any matching substring:

`ReplaceWords({'foo': 'bar'})('foobar')` returns `barbar`

This isn't necessarily a problem (it could be useful), but it is not consistent with the function name. I see two possibible fixes:
- Change internals to `re.sub(r'\b{}\b'.format(re.escape(key)), value, s)`. Note that `\b` is the word boundary mark.
- Rename the function to `ReplaceRegexes` (or keep both). The example should reflect the current behavior. Regex replaces can be nice: you can for instance use `ReplaceWords({r'\b(foo|bar)baz\b': r'\1'})` to remove 'baz' from 'foobaz' and 'barbaz' but keep it in all other cases (artificial example, I know).

The above indicates that this function (or these functions, if you go for the second option) can use a unit test. This is currently missing..

Finally, the transform is not included in `transforms.__all__`, and as a result it isn't directly available on `jiwer`, but only as part of `jiwer.transforms`.

Any thoughts on #21 ?

Thanks for raising the issue! Should be fixed with #21. 

That was fast, great work!
Repository: jitsi/jiwer
Issue: standardize parameter broken
Root cause: Algorithmic Error

MWE:

```
from jiwer import wer

e1 = wer("he's my neminis", "he is my <unk> [laughter]", standardize=True)
e2 = wer("he is my neminis", "he is my")

assert e1 == e2
```
According to the documentation, these should be equivalent, but are not.

Version 1.3 from PyPi



Hi, 

Thanks for your interest in this repo! I've created an initial fix at PR #4 . Do you have some other concerns before I merge?

@nikvaessen Thanks for the fix! No other concerns at the moment.
Repository: jitsi/jiwer
Issue: RemoveKaldiNonWords transformation not working as expected
Root cause: Documentation Deficiency

Hello, when trying to use the `RemoveKaldiNonWords` transformation, I get different results when comparing these text pairs: 

- `<unk> xx` to `xx` -> 0.5 WER (0.33 when not using `SentencesToListOfWords`)
- `<unk>xx` to `xx` -> 0.0 WER

I'd expect both to be zero when using `RemoveKaldiNonWords`. Is it an actual bug or am I not understanding the usage correctly? I've tried different order combination of the transformations in the below code, the results were always the same.

```
transformation = jiwer.Compose([
    jiwer.RemoveKaldiNonWords(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.RemoveWhiteSpace(replace_by_space=True),
    jiwer.SentencesToListOfWords(word_delimiter=' '),
])
wer = jiwer.wer(
    truth,
    hypothesis,
    truth_transform=transformation, 
    hypothesis_transform=transformation,
)
```

Your transformation should also include `jiwer.RemoveEmptyStrings`. 

```python
import jiwer

truth = "hello"
hypothesis = "hello <unk>"

buggy_custom_transform = jiwer.Compose(
    [
        jiwer.RemoveKaldiNonWords(),
        jiwer.RemoveMultipleSpaces(),
        jiwer.RemoveWhiteSpace(replace_by_space=True),
        jiwer.SentencesToListOfWords(word_delimiter=" "),
    ]
)
working_custom_transform = jiwer.Compose(
    [
        jiwer.RemoveKaldiNonWords(),
        jiwer.RemoveMultipleSpaces(),
        jiwer.RemoveWhiteSpace(replace_by_space=True),
        jiwer.RemoveEmptyStrings(),
        jiwer.SentencesToListOfWords(word_delimiter=" "),
    ]
)

buggy_error_rate = jiwer.wer(
    truth,
    hypothesis,
    truth_transform=buggy_custom_transform,
    hypothesis_transform=buggy_custom_transform,
)
correct_error_rate = jiwer.wer(
    truth,
    hypothesis,
    truth_transform=working_custom_transform,
    hypothesis_transform=working_custom_transform,
)

print(f'after transform: truth={buggy_custom_transform(truth)}, hypothesis:{buggy_custom_transform(hypothesis)}')
print(f'after transform: truth={working_custom_transform(truth)}, hypothesis:{working_custom_transform(hypothesis)}')
print("buggy wer:", buggy_error_rate)
print("correct wer:", correct_error_rate)
```

```
after transform: truth=['hello'], hypothesis:['hello', '']
after transform: truth=['hello'], hypothesis:['hello']
buggy wer: 1.0
correct wer: 0.0
```
Repository: jitsi/jiwer
Issue: Can somebody explain me what the `_preprocess()` function is doing? 

I ran the standard example and walk through it in the debugger. 

```
from jiwer import wer

ground_truth = "hello world"
hypothesis = "hello duck"

error = wer(ground_truth, hypothesis)
```

In the steps, it uses the `_preprocess()` function to convert the words/tokens into integer representations. 

`hello world` is converted to `\x00\x02` and `hello duck` is converted to `\x00\x01`. Then the Levenshtein distance is calculated on these strings rather than the original words. I am not sure how that maps to the original definition of [Word Error Rate](https://en.wikipedia.org/wiki/Word_error_rate). 

Can somebody explain to me what is happening? 

> Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.
(https://en.wikipedia.org/wiki/Levenshtein_distance)

Due to the fact that we convert each word in a sentence to an unique integer such as `\x00`, the ground truth and hypothesis sentences become  two "words" of integer tokens. We can then use the Levenshtein distance to calculate the insertions (I), deletions (D) or substitutions (S) required to change the hypothesis "integer word" into the ground truth "integer word". We can than simply plug the S, D, and I values we found in the WER formula. 

Oh, I realized this a little late, the **Levenshtein distance in WER is calculated on words** rather than characters. So that is done to convert each word into a unique ID, makes sense now. Thank you!
Repository: jitsi/jiwer
Issue: Multispeaker WER
Root cause: Unimplemented Feature Gap

Hi, thanks a bunch for this tool !

When working with speech mixtures, WER can take into account that words from each speaker might be picked up. 
There is a description of the method here: https://my.fit.edu/~vkepuska/ece5527/sctk-2.3-rc1/doc/asclite.html

Would you be willing to integrate this feature in Jiwer? 


I think there are two ways of implementing this:

1) we need a either wrap around asclite which will require shipping its binary for every platform
2) or write a custom dynamic programming solution, which would be most likely be very slow if implemented in python, or difficult if it needs to be written in C (I don't have much if any experience in writing C and integrating it into a python application). 

How would you use this feature? Are there many speech datasets which have this problem? 


Thanks for your answer. 

> How would you use this feature? Are there many speech datasets which have this problem?

All datasets that include overlapping speech have this problem. Few examples: Chime5-6, AMI, wsj0-mix, Librimix. In order to evaluate speech separation algorithm, this seems to be needed. 

I'd go with solution 1. 
I personally wouldn't ship the binaries but link to the [installation instructions](https://github.com/usnistgov/SCTK). This would be an optional feature of `jiwer`, and the user would need to make an extra step to benefit from it. WDYT?
Repository: jitsi/jiwer
Issue: Feature Request: Allow for equivalent/alternative words
Root cause: Unimplemented Feature Gap

When calculating WER, MER, and WIL it would be nice to allow for equivalent or alternative words. For example:

> Hi, This is Ann. 
> Hi, This is Anne. 
>
>The cost is 21 dollars
>The cost is $21.00 dollars
>The cost is $21
>The cost is twenty one dollars
>
>I'm going to go for a walk
>I'm gonna go for a walk

could all be equivalent

A method to implement something like this could be to allow for something like this syntax `[<alternatives>]` in the ground truth, for example:
> Hi this is [Anne|Ann] I'm [going to|gonna] go for a walk



I think that you should use the pre-processing transform API to solve these cases by normalizing all text to a single alternative.
Repository: jitsi/jiwer
Issue: WER score bigger than 1.0
Root cause: Documentation Deficiency

Literature states that WER is bounded between 0 and 1.

```python
wer(truth='fries', hypothesis='no fry')
```
gives

    > 2.0

Is this an expected behaviour?

Yes.

Given the formula `WER = (S + D + I) / N`, and your example with `N=1`, `D=1`, `S=1` would result in `WER = (1 + 1 + 0) / 1 = 2/1 = 2`.

See also https://en.wikipedia.org/wiki/Word_error_rate, which states:

_"Note that since N is the number of words in the reference, the word error rate can be larger than 1.0, and thus, the word accuracy can be smaller than 0.0. "_

Any literature which states WER is bounded between 0 and 1 is wrong (or has different assumptions perhaps). Note that the WER is bounded between 0 and infinity, as a prediction could potentially return an infinite list of words. Therefore the deletion term `D` in the formula can grow arbitrarily large.

Very clear. Thanks for your response.
Repository: jitsi/jiwer
Issue: Pypi version outdated
Root cause: Configuration Error

After the fix on PR #4 the Pypi version is outdated.

What Pypi version? Do you men the pip version? Python version?

Yes, for the installation via `pip install jiwer`.
https://pypi.org/project/jiwer/#history

You mean a new release on pypi is necesary? (ping @nikvaessen )

Yes, there is a workaround by installing using the git repo `pip install git+https://github.com/jitsi/asr-wer`, but if you use `pip install jirew` will install the version without the fix on PR #4.

I've now bumped the version on pipy to 1.3.1

Thanks, btw awesome package 😃 !!!
Repository: jitsi/jiwer
Issue: module 'jiwer.transforms' has no attribute 'ReduceToListOfListOfWords'
Root cause: Environment Incompatibility

Hi,

I did downloaded this repo as zip and facing following exception after configuring the project, when executing analyzer.py :

**AttributeError: module 'jiwer.transforms' has no attribute 'ReduceToListOfListOfWords'**

I don't know what this `analyzer.py` looks like, how you run it, and how you installed the package in your (virtual) environment. Without further information, I would advice you to install the package using either `pip` or `poetry`.

issue created by mistake, please delete this thread. thanks
Repository: jitsi/jiwer
Issue: Don't support Chinese?


Hi, 

I made this example script, with references and predictions taken from https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn

```python3
from jiwer import wer, cer

ground_truths = [
    "宋朝末年年间定居粉岭围。",
    "渐渐行动不便",
    "二十一年去世。",
    "他们自称恰哈拉。",
    "局部干涩的例子包括有口干、眼睛干燥、及阴道干燥。",
    "嘉靖三十八年，登进士第三甲第二名。",
    "这一名称一直沿用至今。",
    "同时乔凡尼还得到包税合同和许多明矾矿的经营权。",
    "为了惩罚西扎城和塞尔柱的结盟，盟军在抵达后将外城烧毁。",
    "河内盛产黄色无鱼鳞的鳍射鱼。",
]

hypothesis = [
    "宋朝末年年间定居分定为",
    "建境行动不片",
    "二十一年去世",
    "他们自称家哈",
    "菊物干寺的例子包括有口肝眼睛干照以及阴到干",
    "嘉靖三十八年登进士第三甲第二名",
    "这一名称一直沿用是心",
    "同时桥凡妮还得到包税合同和许多民繁矿的经营权",
    "为了曾罚西扎城和塞尔素的节盟盟军在抵达后将外曾烧毁",
    "合类生场环色无鱼林的骑射鱼",
]

wer_score = wer(truth=ground_truths, hypothesis=hypothesis)
print(f"{wer_score=}")

cer_score = cer(truth=ground_truths, hypothesis=hypothesis)
print(f"{cer_score=}")
```
Which outputs:
```
wer_score=1.0
cer_score=0.30201342281879195
```

What would the correct answer be? Does `word_error_rate` even make sense in zh-cn, as as far as I know, each character is a word? 


Thanks for your reply, but the wer_score=1.0, is mean totally wrong word, i do not think so, for example,  ground_truths  = 宋朝末年年间定居粉岭围,  hypothesis =  "宋朝末年年间定居分定为" only last 3 words is wrong, so the wer must be < 1.0. if you improve the result, i can test it again. thanks! 

Semantically, I assume that you need the "character" error rate instead of the word error rate, as I assume they should be equivalent for Chinese. Therefore, is the CER score of 0.3 correct?

yes, CER is correct, i just focused on WER yesterday, because CER is confuse for chinese words.

” “.join('你好') => "你 好"
即可
可以加 jieba 优化一下
Repository: jitsi/jiwer
Issue: Avoid error when a string in the truth is empty after transformation
Root cause: Documentation Deficiency

A suggestion to add:
To avoid getting an error message when transforming the truth leads to an empty string, I added the following in _preprocess:
pop the empty string item from the transformed_truth and the transformed_hypothesis.
I found this useful for my purposes. Perhaps others may too.

    # Apply transforms. The transforms should collapses input to a list of list of words
    transformed_truth = truth_transform(truth)
    transformed_hypothesis = hypothesis_transform(hypothesis)
    n=0
    for e in transformed_truth:
        if (len(e) == 0):
           #if truth is empty, remove that item from the list and from the transformed hypotheses
           transformed_truth.pop(n)
           transformed_hypothesis.pop(n)
        n += 1

    # raise an error if the ground truth is empty or the output
    # is not a list of list of strings


I must say that this is a weird edge case, as in general you don't want the ground truth to be an empty string. Furthermore, the word-error-rate is undefined for an empty ground truth string (division by zero). But you're right that an "aggressive" transformation pipeline can lead to (unintended) empty ground truth strings. I'm not sure what the solution is, other than failing fast so people are aware there is most likely a bug/unintended side-effect in their transformation pipeline.

Also note that `jiwer.RemoveEmptyStrings()` exists for this purpose.
Repository: jitsi/jiwer
Issue: Alignment options similar to `fstalign`

[fstalign](https://github.com/revdotcom/fstalign/blob/develop/docs/Advanced-Usage.md) has an `--output-sbs` flag which allows the user to use the alignment for post-processing. I was wondering if `jiwer` has such a flag?

The latest version (2.6.0) provides, and can visualize, the alignment between each ground-truth/hypothesis pair.
Repository: jitsi/jiwer
Issue: Batch vs Individual results are not same
Root cause: Documentation Deficiency

I have tried calculating WER and CER using jiwer in one case I'm running jiwer.wer and jiwer.cer function individually on single (target, prediction) pairs and take the average at the end, and in the other case, I'm passing the whole list of targets and predictions to jiwer.wer and jiwer.cer functions. In both cases, I'm getting different WER and CER with the same set of target and prediction pairs which ideally shouldn't be the case.

Any idea why is this happening?

This might be related to https://github.com/jitsi/jiwer/issues/46? Are you using the latest version?

Thanks for quick reply @nikvaessen, I am using version 2.3.0

Can you share a minimal example?

![Screenshot from 2022-08-19 15-02-46](https://user-images.githubusercontent.com/18329235/185590349-9ce0487d-23f7-4f57-ae73-b860db84f641.png)



As you can see WER and CER is different in both cases.

The problem is taking the mean across 2 word-error-rate values is not the same as computing the WER over two sentences.

If you look at the formula of WER:

![image](https://user-images.githubusercontent.com/7225987/188173780-6840d572-3df1-4470-b40c-b10896971031.png)

For your case:

```python
import jiwer

# sentence 1 - 1 substitution, 3 correct
ref0 = "a b c d"
hyp0 = "a b c e"

wer0 = jiwer.wer(ref0, hyp0) # 0.25

# sentence 2 - 1 substitution, 2 correct
ref1 = "ab bc cd"
hyp1 = "ba bc cd"
 
wer1 = jiwer.wer(ref1, hyp1) # 0.33

# manual computation answer
s = 1 + 1
d = 0
i = 0
c = 3 + 2

manual_wer = (s + d + i) / (s + d + c) # 0.2857142857142857
average_wer = (wer0 + wer1) / 2 # 0.29166666666666663
weighted_average = wer0 * (4 / 7) + wer1 * (3 / 7) # 0.2857142857142857

wer = jiwer.wer([ref0, ref1], [hyp0, hyp1]) # 0.2857142857142857
```




In practise, the WER is a weighted average, the longer the sentence, the more opportunity for mistakes, etc.
Repository: jitsi/jiwer
Issue: Doesn't work on lists
Root cause: Algorithmic Error

Hi,

Applying jiwer.wer on reference ['a a'], hypothesis ['a', 'a'] gives a WER of 1, whereas it should be 0?


fixed by https://github.com/jitsi/asr-wer/commit/2b96cc5d7c37e84954f615880c5f65de99d5e805
Repository: jitsi/jiwer
Issue: Update Levenshtein dependency to maintained version
Root cause: External Dependency Breakage

Currently, you depend on python-Levenshtein which is outdated and not maintained. This leads to issues when compiling. For instance, I cannot compile it for Python 3.7 on both my Windows and Ubuntu machines by default without installing/updating build tools. The newer, maintained Levenshtein provides [prebuilt wheels](https://pypi.org/project/Levenshtein/#files) which saves a lot of headaches.

Usage is otherwise the same. So no other changes are needed than updating the dependency.
Repository: jitsi/jiwer
Issue: Major performance regression in 2.5.0 for jiwer.transforms.RemovePunctuation
Root cause: Algorithmic Error

There is a major performance regression in jiwer.transforms.RemovePunctuation in 2.5.0
`%timeit jiwer.transforms.RemovePunctuation()` gives `228 ms ± 4.34 ms per loop` on 2.5.0
but
`%timeit jiwer.transforms.RemovePunctuation()` gives `1.35 µs ± 8.61 ns per loop` on 2.4.0
this is `168,888` **times** slower.

I had a pipeline where I was computing the WER both with and without removing punctuation. I was recreating the transform list each time. This can be solved by computing it up front and caching the transformations, but I am very surprised to find this to be the cause of my regression.

Looks like this is caused by `https://github.com/jitsi/jiwer/commit/0fc3893027bcc03114b34e33da5c5298b7da20e1#` 

Thanks for spotting the issue and making a quick PR! Release 2.5.1 should have the fix.
Repository: jitsi/jiwer
Issue: Current licenses might not be allowed

Hey, I see this project depends on levenshtein but that package is released under GPL license meaning packages using it (like this one) need to have a GPL license but the current one is not

Source: https://www.gnu.org/licenses/gpl-faq.html#IfLibraryIsGPL

Thanks for sharing. I'm planning to move away from the dependency as soon as possible.

Resolved as of latest release on pipy (2.5.2).
Repository: jitsi/jiwer
Issue: jiwer.visualize_measures doesn't work as in the docs
Root cause: Documentation Deficiency

When I try to run

```python3
import jiwer

out = jiwer.compute_measures(
    ["short one here", "quite a bit of longer sentence"],
    ["shoe order one", "quite bit of an even longest sentence here"],
)

print(jiwer.visualize_measures(out))
```

I get `AttributeError: module 'jiwer' has no attribute 'visualize_measures'`

It seems the function is not implemented yet in jiwer 3.0.0 or maybe under another name ?

A working example:
 
```python3
import jiwer

out = jiwer.process_words(
    ["short one here", "quite a bit of longer sentence"],
    ["shoe order one", "quite bit of an even longest sentence here"],
)

print(jiwer.visualize_alignment(out))
```

Where did read the example you posted? 

It was in the docs, it is now corrected, thank you.
Repository: jitsi/jiwer
Issue: Version 3.0.0 can produce wrong results
Root cause: Algorithmic Error

Hi,

when I use jiwer version 3.0.0 with the following commands:

```python
wer_transform = jiwer.Compose(
    [
        jiwer.ToLowerCase(),
        jiwer.RemoveMultipleSpaces(),
        jiwer.Strip(),
        jiwer.ReduceToListOfListOfWords(),
    ]
    )
wer = jiwer.wer(
        truth = 'This is a short Sentence with a few Words with upper and Lower cases',
        hypothesis = 'His is a short Sentence with a few Words with upper and Lower cases',
        truth_transform = wer_transform,
        hypothesis_transform = wer_transform,
        )
```
I get wer=0.0714, which is the expected result based on the WER formula.

But when I use the non-deprecated keywords 'reference' and 'reference_transform' instead of 'truth' and 'truth_transform':
```python
wer = jiwer.wer(
        reference = 'This is a short Sentence with a few Words with upper and Lower cases',
        hypothesis = 'His is a short Sentence with a few Words with upper and Lower cases',
        reference_transform = wer_transform,
        hypothesis_transform = wer_transform,
        )
```
I get wer=0.2857, which is clearly wrong.
This does not happen if the 'jiwer.ToLowerCase()' transform is removed from the transformation, thus I expect the bug to be related to that.

Cheers,
   Paul

Thanks for noticing! Fixed in release 3.0.1.
Repository: jitsi/jiwer
Issue: Regarding visualize_alignment() function.

```visualize_alignment()``` function takes parameters one output from ```process_words()``` and second ```show_measures```. If I want to show only those sentences with errors greater than zero. How can I proceed?

I know I can loop through, but adding this as an argument would be good.

Should be available from version 3.0.2 onwards.