Question: How is the quality of singing voice measured?

Evidence:
raw_evidence: To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score.
The automatic score roughly followed the design in BIBREF13. The pitch tracker of librosa package BIBREF18 was employed to extract pitch information of the input and output audio. Then the output pitch was compared to the input pitch using the normalized cross correlation (NCC) which would give a score between 0 and 1. The higher the score is, the better the output pitch matches the input pitch. We conducted the evaluation on USVC (our) and PitchNet. The evaluated automatic scores on conversion and reconstruction tasks are shown in Tab. TABREF14. Our method performed better both on conversion and reconstruction. The scores of reconstruction are higher than conversion since both models were trained using a reconstruction loss. However, the score of our method on conversion is even higher than the score of USVC (Our) on reconstruction.
Mean Opinion Score (MOS) was used as a subjective metric to evaluate the quality of the converted audio. Two questions were asked: (1) what is quality of the audio? (naturalness) (2) How well does the converted version match the original? (similarity) A score of 1-5 would be given to answer the questions. The evaluation was conducted on USVC (Our) and PitchNet. Besides, the converted samples provided by BIBREF0 was also included to give a more convincing evaluation. As shown by Tab. TABREF15, the naturalness and similarity of our method are both higher than the other two ones. Our implementation of USVC performed slightly lower than the original author's because we cannot fully reproduce the results of them.
Next we qualitatively analyze the influence of input pitch in our method. We used different pitch as input to observe how the output pitch would change along with the input pitch. The input pitch was multiplied by 0.7, 1.0 and 1.3 respectively. And the output pitch was also extracted by the pitch tracker of the librosa package. Fig. FIGREF16 plots the pitch of input audio and output audio with different pitch as input while keeping the target singer the same. As shown by Fig. FIGREF16, the output pitch changes significantly along with the input pitch. The examples are also presented at our website.
raw_evidence: To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score.
The automatic score roughly followed the design in BIBREF13. The pitch tracker of librosa package BIBREF18 was employed to extract pitch information of the input and output audio. Then the output pitch was compared to the input pitch using the normalized cross correlation (NCC) which would give a score between 0 and 1. The higher the score is, the better the output pitch matches the input pitch. We conducted the evaluation on USVC (our) and PitchNet. The evaluated automatic scores on conversion and reconstruction tasks are shown in Tab. TABREF14. Our method performed better both on conversion and reconstruction. The scores of reconstruction are higher than conversion since both models were trained using a reconstruction loss. However, the score of our method on conversion is even higher than the score of USVC (Our) on reconstruction.
Mean Opinion Score (MOS) was used as a subjective metric to evaluate the quality of the converted audio. Two questions were asked: (1) what is quality of the audio? (naturalness) (2) How well does the converted version match the original? (similarity) A score of 1-5 would be given to answer the questions. The evaluation was conducted on USVC (Our) and PitchNet. Besides, the converted samples provided by BIBREF0 was also included to give a more convincing evaluation. As shown by Tab. TABREF15, the naturalness and similarity of our method are both higher than the other two ones. Our implementation of USVC performed slightly lower than the original author's because we cannot fully reproduce the results of them.

Answers:
To compare the conversions between USVC and PitchNet, we employed an automatic evaluation score and a human evaluation score.
Automatic: Normalized cross correlation (NCC)
Manual: Mean Opinion Score (MOS)