Paper: SQS: Speech Quality Assessment in the Data Annotation Context (ICLR-2024)

Reviewer comment (Novelty):
The novelty of this paper is very limited. In fact, there is no novelty in terms of the machine learning perspective. This paper is more suitable to be submitted to speech-related conferences (e.g., Interspeech, Icassp, etc.). Specifically, this work simply employs a pre-trained speech quality estimator (NISQA) and finetunes on its own dataset.

Author response:
Thank you for your point of view. Despite of the fact that our work could be also suitable for speech-domain conferences, we considered this paper matching one of the ICL topic “applications in vision, audio, speech, natural language processing, robotics, neuroscience, or any other field”. We consider the novelty of this work is the definition of a metric able to provide information about how is the audio recording, i.e., related to speech intelligibility, according to the possible impact on the annotation performance, i.e., resulted label.

Reviewer comment (Evaluation):
The comparison to other models (e.g., NISQA, DNSMOS) is also unfair, because of different training data and the label scale. In DNSMOS, the MOS scale is from 1 to 5, however, in the collected dataset, the label scale is from 1 to 4. Although it may be okay for Pearson correlation, metrics such as Mean Square Error (MSE) will be significantly affected by this scale mismatch.

Author response:
We appreciate the reviewer for the observation. We agree that the scale differences between MOS and SQS values can affect MSE. In response to this, we propose using NMSE (Normalized Mean Square Error) to address these scale differences. The results with the new evaluation approach using NMSE are presented in the revised version of the paper.

Reviewer comment (Experiments):
Although each audio sample is 10 seconds, do they contain the same number of words? I believe it will also affect the transcription time.

Author response:
Indeed, the number of words can influence the transcription time of the file; however, in addition to that, there are other more significant features, specifically the audio quality as shown in Fig 5.

Reviewer comment (Experiments):
The research relies heavily on the RTVE2020 Database for experimentation. The results might be limited and may not generalize well to other databases or real-world scenarios. For example, the paper does not speak much about the data collection framework setting (whether clean references were provided, quality of those clean references, raters’ qualifications, what type of questions were asked...).

Author response:
We appreciate the feedback related to the whole article. Sigma’s proprietary tool refers to one of our audio annotation tools which was used to carry out the annotation process to obtain SQS values over the 1,020 audio samples. Raters were able to select a value of the defined SQS scale (1-4) for each audio sample. In Table 1, a score of 5 (Excellent) is not considered because in a real-world transcription context, the quality of the samples of the audio to be annotated often contains background noise or other artifacts, thereby affecting the intelligibility of the spoken content. In this point, the reference has been directly extracted from our clients which we cannot mention for confidentiality reasons. Furthermore, in this initial exploration, only two ratings per audio sample were considered due to resource constraints in evaluating 1,020 audios.