Repository: jitsi/jiwer
Issue: wer_standardize error
Root cause: Validation Gap

I used the below code to try and process the data but got a ValueError


```
import jiwer

string_wer_data["standardized_ref"] = jiwer.wer_standardize((string_wer_data["reference"]))
string_wer_data["standardized_hyp"] = jiwer.wer_standardize((string_wer_data["hypothesis"]))
```

emulating nikvaessen's response to another issue:
```
import jiwer

jiwer.wer(
  outputs_true, 
  outputs_pred,
  reference_transform=jiwer.wer_standardize, 
  hypothesis_transform=jiwer.wer_standardize
)
```

_Originally posted by @nikvaessen in https://github.com/jitsi/jiwer/issues/85#issuecomment-1915271530_


ValueError:
```
Traceback (most recent call last):
  File "/Users/emily/Desktop/whisper-input/wer_modules.py", line 73, in <module>
    string_wer_data["standardized_ref"] = jiwer.wer_standardize((string_wer_data["reference"]))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emily/miniconda3/lib/python3.12/site-packages/jiwer/transforms.py", line 130, in __call__
    text = tr(text)
           ^^^^^^^^
  File "/Users/emily/miniconda3/lib/python3.12/site-packages/jiwer/transforms.py", line 87, in __call__
    raise ValueError(
ValueError: input 0 
```
            

Can you verify the following assertions on  `string_wer_data["reference"] `:

```python3
assert isinstance(string_wer_data["reference"], list)
assert all(isinstance(e, str) for e in string_wer_data["reference"])
```

i verified those assertions are true

thanks so much for looking into this, i was wondering if there’s any other info i can provide?

Can you provide a minimal code sample which reproduces the error?

```
import pandas as pd
import numpy as np

import jiwer
import glob
import os

hypotheses = []
references = []

# glob to get file list-- directory name, glob will give list of all files
hyps = glob.glob('/Users/emily/Desktop/whisper-output-turbo-word/*.txt')
refs = glob.glob('/Users/emily/Desktop/cd-reference-txt/*.txt')

for hyp, ref in zip(hyps, refs):
    with open(hyp) as hypfile, open(ref) as reffile: #loops through each file #hypfile = file handle
        hyp_read = hypfile.read() #get file contents w/ .read() method
        ref_read = reffile.read()
        hypotheses.append(hyp_read) #append() can add whole list as element to list
        references.append(ref_read)

whisper_wer_data = pd.DataFrame(dict(hypothesis=hypotheses, reference=references))

string_wer_data = whisper_wer_data.astype("string") #creates copy of df in str type


string_wer_data["standardized_ref"] = jiwer.wer_standardize((string_wer_data["reference"]))
string_wer_data["standardized_hyp"] = jiwer.wer_standardize((string_wer_data["hypothesis"]))
```

the files are on my local machine, is there anything else i should provide?

Preferably, it should be a _minimal_ code sample, which does not read to read any data from disk. This way, I can reproduce the error on my computer and debug the issue.

I would suggest that you try to isolate the particular string which causes the `ValueError`.  You could loop over each file, read the content, and separately call `wer_standardize` on it.

apologies, i'm not very familiar with all the terminology for troubleshooting issues. i'll try to do as you suggest