Commit
•
0624af3
1
Parent(s):
b9993c0
Update README.md
Browse files
README.md
CHANGED
|
@@ -496,7 +496,25 @@ BLEU score on [mExpresso](https://huggingface.co/facebook/seamless-expressive#me
|
|
| 496 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
| 497 |
| 1.23.0 | canary-1b | 23.84 | 35.74 | 28.29 |
|
| 498 |
|
|
|
|
| 499 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 500 |
|
| 501 |
## NVIDIA Riva: Deployment
|
| 502 |
|
|
|
|
| 496 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
| 497 |
| 1.23.0 | canary-1b | 23.84 | 35.74 | 28.29 |
|
| 498 |
|
| 499 |
+
## Model Fairness Evaluation
|
| 500 |
|
| 501 |
+
As outlined in the paper "Towards Measuring Fairness in AI: the Casual Conversations Dataset", we assessed the parakeet-tdt-1.1b model for fairness. The model was evaluated on the CausalConversations-v1 dataset, and the results are reported as follows:
|
| 502 |
+
|
| 503 |
+
### Gender Bias:
|
| 504 |
+
|
| 505 |
+
| Gender | Male | Female | N/A | Other |
|
| 506 |
+
| :--- | :--- | :--- | :--- | :--- |
|
| 507 |
+
| Num utterances | 19325 | 24532 | 926 | 33 |
|
| 508 |
+
| % WER | 14.64 | 12.92 | 17.88 | 126.92 |
|
| 509 |
+
|
| 510 |
+
### Age Bias:
|
| 511 |
+
|
| 512 |
+
| Age Group | (18-30) | (31-45) | (46-85) | (1-100) |
|
| 513 |
+
| :--- | :--- | :--- | :--- | :--- |
|
| 514 |
+
| Num utterances | 15956 | 14585 | 13349 | 43890 |
|
| 515 |
+
| % WER | 14.64 | 13.07 | 13.47 | 13.76 |
|
| 516 |
+
|
| 517 |
+
(Error rates for fairness evaluation are determined by normalizing both the reference and predicted text, similar to the methods used in the evaluations found at https://github.com/huggingface/open_asr_leaderboard.)
|
| 518 |
|
| 519 |
## NVIDIA Riva: Deployment
|
| 520 |
|