  Traditionally, Quality of Experience (QoE) for a communication system is
evaluated through a subjective test. The most common test method for speech QoE
is the Absolute Category Rating (ACR), in which participants listen to a set of
stimuli, processed by the underlying test conditions, and rate their perceived
quality for each stimulus on a specific scale. The Comparison Category Rating
(CCR) is another standard approach in which participants listen to both
reference and processed stimuli and rate their quality compared to the other
one. The CCR method is particularly suitable for systems that improve the
quality of input speech. This paper evaluates an adaptation of the CCR test
procedure for assessing speech quality in the crowdsourcing set-up. The CCR
method was introduced in the ITU-T Rec. P.800 for laboratory-based experiments.
We adapted the test for the crowdsourcing approach following the guidelines
from ITU-T Rec. P.800 and P.808. We show that the results of the CCR procedure
via crowdsourcing are highly reproducible. We also compared the CCR test
results with widely used ACR test procedures obtained in the laboratory and
crowdsourcing. Our results show that the CCR procedure in crowdsourcing is a
reliable and valid test method.
