  In recent years, machine speech-to-speech and speech-to-text translation has
gained momentum thanks to advances in artificial intelligence, especially in
the domains of speech recognition and machine translation. The quality of such
applications is commonly tested with automatic metrics, such as BLEU, primarily
with the goal of assessing improvements of releases or in the context of
evaluation campaigns. However, little is known about how such systems compare
to human performances in similar communicative tasks or how the performance of
such systems is perceived by final users.
  In this paper, we present the results of an experiment aimed at evaluating
the quality of a simultaneous speech translation engine by comparing it to the
performance of professional interpreters. To do so, we select a framework
developed for the assessment of human interpreters and use it to perform a
manual evaluation on both human and machine performances. In our sample, we
found better performance for the human interpreters in terms of
intelligibility, while the machine performs slightly better in terms of
informativeness. The limitations of the study and the possible enhancements of
the chosen framework are discussed. Despite its intrinsic limitations, the use
of this framework represents a first step towards a user-centric and
communication-oriented methodology for evaluating simultaneous speech
translation.
