sanchit-gandhi HF staff commited on
Commit
c0e17f1
1 Parent(s): bd4a047

OpenReview -> Arxiv

Browse files
Files changed (1) hide show
  1. app.py +5 -3
app.py CHANGED
@@ -90,13 +90,13 @@ st.dataframe(table.style.format(subset=["esb-score", *TEST_SETS, *OPTIONAL_TEST_
90
 
91
  st.markdown(
92
  """
93
- ESB was proposed in *ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition* by ...
94
  \n
95
  The abstract of the paper is as follows:
96
  \n
97
  *Speech recognition applications cover a range of different audio and text distributions, with different speaking styles, background noise, transcription punctuation and character casing. However, many speech recognition systems require dataset-specific tuning (audio filtering, punctuation removal and normalisation of casing), therefore assuming a-priori knowledge of both the audio and text distributions. This tuning requirement can lead to systems failing to generalise to other datasets and domains. To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. Benchmarked systems must use the same data pre- and post-processing algorithm across datasets - assuming the audio and text data distributions are a-priori unknown. We compare a series of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark, demonstrating how a single speechsystem can be applied and evaluated on a wide range of data distributions. We find E2E systems to be effective across datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems tuned to a specific dataset. Our analysis reveals that transcription artefacts, such as punctuation and casing, pose difficulties for ASR systems and should be included in evaluation. We believe E2E benchmarking over a range of datasets promotes the research of multi-domain speech recognition systems.*
98
  \n
99
- For more information, please see the official submission on [OpenReview.net](https://openreview.net/forum?id=9OL2fIfDLK).
100
  """
101
  )
102
 
@@ -106,7 +106,9 @@ st.markdown(
106
  \n
107
  To submit to ESB, download the audio data for the nine mandatory ESB test sets from [esb/datasets](https://huggingface.co/datasets/esb/datasets). The test sets contain audio data only. Evaluate your system on the nine test sets by generating predictions for the unlabelled audio samples. For each test set, save the predictions to a .txt file in the order that the audio samples are provided, with one prediction per line. Name the .txt file according to the ESB test set names shown in the table (e.g. the predictions for LibriSpeech test-clean should be named ls-clean.txt).
108
  \n
109
- Once you have evaluated your system on all nine test sets, move the predictions into one folder and zip it. The name you assign to the zipped folder will be the name that is shown on the table (e.g. whisper-aed.zip will be displayed as whisper-aed). Upload your zipped submissions for scoring and placement on the leaderboard. Should you experience any issues, open an issue using the link [new discussion](https://huggingface.co/spaces/esb/leaderboard/discussions/new) and tag `@sanchit-gandhi`.
 
 
110
  """
111
  )
112
 
 
90
 
91
  st.markdown(
92
  """
93
+ ESB was proposed in *ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition*, by [Sanchit Gandhi](https://huggingface.co/sanchit-gandhi), [Patrick von Platen](https://huggingface.co/patrickvonplaten) and [Sasha Rush](https://huggingface.co/srush).
94
  \n
95
  The abstract of the paper is as follows:
96
  \n
97
  *Speech recognition applications cover a range of different audio and text distributions, with different speaking styles, background noise, transcription punctuation and character casing. However, many speech recognition systems require dataset-specific tuning (audio filtering, punctuation removal and normalisation of casing), therefore assuming a-priori knowledge of both the audio and text distributions. This tuning requirement can lead to systems failing to generalise to other datasets and domains. To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. Benchmarked systems must use the same data pre- and post-processing algorithm across datasets - assuming the audio and text data distributions are a-priori unknown. We compare a series of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark, demonstrating how a single speechsystem can be applied and evaluated on a wide range of data distributions. We find E2E systems to be effective across datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems tuned to a specific dataset. Our analysis reveals that transcription artefacts, such as punctuation and casing, pose difficulties for ASR systems and should be included in evaluation. We believe E2E benchmarking over a range of datasets promotes the research of multi-domain speech recognition systems.*
98
  \n
99
+ For more information, refer to the paper submission on [Arxiv](https://arxiv.org/abs/2210.13352).
100
  """
101
  )
102
 
 
106
  \n
107
  To submit to ESB, download the audio data for the nine mandatory ESB test sets from [esb/datasets](https://huggingface.co/datasets/esb/datasets). The test sets contain audio data only. Evaluate your system on the nine test sets by generating predictions for the unlabelled audio samples. For each test set, save the predictions to a .txt file in the order that the audio samples are provided, with one prediction per line. Name the .txt file according to the ESB test set names shown in the table (e.g. the predictions for LibriSpeech test-clean should be named ls-clean.txt).
108
  \n
109
+ Once you have evaluated your system on all nine test sets, move the predictions into one folder and zip it. The name you assign to the zipped folder will be the name that is shown on the table (e.g. whisper-aed.zip will be displayed as whisper-aed). Upload your zipped submissions for scoring and placement on the leaderboard.
110
+ \n
111
+ Should you experience any issues, open an issue using the link [new discussion](https://huggingface.co/spaces/esb/leaderboard/discussions/new) and tag `@sanchit-gandhi`.
112
  """
113
  )
114