  Generative spoken language modeling involves learning jointly the acoustic
and linguistic characteristics of a language from raw audio only (without text
or labels). We introduce metrics to automatically evaluate the generated output
in terms of acoustic and linguistic quality in two associated end-to-end tasks,
respectively: speech resynthesis (repeating the speech input using the system's
own voice), and speech generation (producing novel speech outputs conditional
on a spoken prompt, or unconditionally), and validate these metrics with human
judgment. We test baseline systems consisting of a discrete speech encoder
(returning discrete, low bitrate, pseudo-text units), a generative language
model (trained on pseudo-text units), and a speech decoder (generating a
waveform from pseudo-text). By comparing three state-of-the-art unsupervised
speech encoders (Contrastive Predictive Coding (CPC), wav2vec 2.0, HuBERT), and
varying the number of discrete units (50, 100, 200), we investigate how the
generative performance depends on the quality of the learned units as measured
by unsupervised metrics (zero-shot probe tasks). We will open source our
evaluation stack and baseline models.
