M ULTIPLE OUTPUT SAMPLES FOR EACH INPUT IN A SINGLE - OUTPUT G AUSSIAN PROCESS
Anonymous authors
Paper under double-blind review
A BSTRACT
The standard Gaussian Process (GP) is formulated to only consider a single output sample for each input in the training set.
Datasets for subjective tasks, such as spoken language assessment, may be annotated with output labels from multiple human raters for each input.
This paper proposes to generalise the GP to allow for multiple output samples per input in the training set.
This differs from a multioutput GP, because all output samples are from the same task here.
The output density function is formulated to be the joint likelihood of observing all output samples.
Through this, the hyper-parameters are optimised using a criterion that is similar to minimising a Kullback-Leibler divergence.
The test set predictions are inferred fairly similarly to a standard GP, with a key difference being in the optimised hyper-parameters.
This approach is evaluated on spoken language assessment tasks, using the public speechocean762 dataset and an internal Tamil language dataset.
The results show that by using the proposed method, the GP is able to compute a test set output distribution that is more similar to the collection of reference outputs annotated by multiple human raters.
I NTRODUCTION
The Gaussian Process (GP) (Rasmussen & Williams, 2006) expresses a prediction uncertainty that naturally increases for inputs further away from the training data.
As opposed to this, Neural Networks (NN) have been observed to yield overly confident predictions, even when the input is from a mismatched domain (Guo et al., 2017).
This behaviour of a GP may allow better explainability of the model’s predictions.
Having explainable predictions of uncertainty may be especially desirable for tasks that are subjective in nature.
In such subjective tasks, multiple human annotators may provide differing output labels for the same input.
A collection of human annotations for the same input may therefore be interpreted as a reference of uncertainty that an automatic model should also aim to compute.
In such settings, the uncertainties expressed by the model and the human annotators can be explicitly compared.
However, the standard GP formulation assumes that each input in the training set is paired with only a single output, which is treated as the ground truth.
This paper proposes to extend the GP formulation, to accommodate for situations where multiple samples of output labels for the same task are provided for each input.
The hyper-parameters can be optimised and the test set predictions can be inferred, with the consideration of having multiple training set output samples.
R ELATED WORK
The multi-output GP is formulated in a multi-task framework (Yu et al., 2005; Bonilla et al., 2007).
This treats the multiple outputs for each input as separate tasks.
On the other hand, this paper considers a single-output GP with a single output task, where multiple output samples for each input are present for that task.
This paper considers optimising the GP hyper-parameters using a criterion that is similar to minimising a distance to a reference output density function.
When training a NN, the reference output can be in the form of a distribution, as opposed to a scalar or single class.
Full-sum training in speech recognition (Yan et al., 1997) and handwriting recognition (Senior & Robinson, 1995) trains a NN toward the distributional reference formed by the soft forced alignment.
In BLIND, NNs for
Spoken Language Assessment (SLA) are trained toward the distribution represented by the scores from multiple human raters.
The distributional output from one NN can also be used as a reference to train another NN toward (Li et al., 2014; Hinton et al., 2014).
G AUSSIAN PROCESS REGRESSION
When given a collection of N input feature vectors of dimension D , X ∈ R N × D , a GP places a jointly Gaussian prior over latent variables, f ∈ R N , as
[Equation]
Here, p ( f | X ) is an abbreviation of p ( ˚ f = f | ˚ X = X ) , with the interpretation of being the likelihood that continuous random variables ˚ f take the values f .
A multi-variate Gaussian density function with mean µ and covariance V is represented as N ( f ; µ , V ) .
In a GP, the covariance of the latent variable is defined as a pair-wise distance between the inputs, with the notion of distance being defined by the kernel, K .
In this paper, the squared exponential kernel is used, with kernel matrix elements defined as
[Equation]
where i and j are the matrix indexes, l is a length hyper-parameter, and s is a scale hyper-parameter.
The GP makes the assumption that the outputs, y ∈ R N , are conditionally independent of the inputs when given the latent variables.
The outputs are Gaussian distributed about a mean of the latent variable, with a noise hyper-parameter, σ , to accommodate for observational noise,
[Equation]
where I is the identity matrix.
3. T RAINING
Training a GP involves estimating the hyper-parameters of the kernel, s and l , and of the observation noise σ .
One approach is to find the hyper-parameters that maximise the marginal log-likelihood of the training data,
[Equation]
where the training data comprises pairs of observed features and reference outputs.
The marginal likelihood can be computed as
[Equation]
3. I NFERENCE
When performing evaluation
[Equation]
a test set of input feature vectors, (cid:99) X , is given
[Equation]
and the task is to infer the predicted outputs, (cid:98) y .
Inference through a GP can be performed by first computing the density function over the predicted latent variables,
[Equation]
where
[Equation]
Then, the predictive density function for the output can be computed as
[Equation]
Finally, the predicted scalar output for each test sample, i , can be inferred from equation 12, using a decision rule.
One possible decision rule is to choose the mean of equation 12, (cid:98)
y ⋆i = (cid:98) µ i .
As a result of the symmetry and uni-modality of a Gaussian density function, the decision rules of choosing either the mean, mode, or median of equation 12 are equivalent.
M ULTIPLE OUTPUT SAMPLES
In the standard GP formulation, each input sample is paired with a single output sample.
As is described in section 5, situations may arise where modelling multiple output samples for each input sample may be beneficial.
This paper considers such a scenario, whereby all output samples for the same input sample are drawn from the same output density p ( y | f ) and kernel K .
It is thus assumed that these multiple output samples belong to the same task.
This differs from the multi-output GP formulation, which assumes that each separate output represents a different task.
4. R EPETITION OF INPUTS
A naive approach to accommodate these multiple output samples may be to simply stack the outputs together, and to repeat the inputs for each output sample.
For example, a dataset with two input samples, having two output samples for each input sample, may use a stacked kernel of (cid:20) k
[Equation]
x 1 , x
[Equation]
x
[Equation]
x
[Equation]
( x
[Equation]
x
[Equation]
x
[Equation]
x
[Equation]
However, the stacked kernel on the right-hand side of equation 13 is not full rank.
This may introduce numerical issues when computing the matrix inverses during inference.
Therefore, this approach is not adopted in this paper.
4. T RAINING
Let each input sample, x i , be associated with R output samples, y i =
[ y i 1 , · · · , y iR ] .
It is assumed here for simplicity that the number of output samples is the same for all inputs, but an analogous formulation can be derived to allow for varying numbers of output samples.
These outputs are stacked together across all input samples, to yield an output matrix, Y ∈ R N × R .
This paper proposes to optimise the GP hyper-parameters by maximising the marginal log-likelihood of all output samples,
[Equation]
This can be related to minimising the Kullback-Leibler (KL) divergence between the marginal likelihood and the reference density function over output samples.
To see this relation, consider only a single training sample, i , and ignore the correlations between multiple training samples.
The marginal log-likelihood for the multiple output references of this single training sample is
[Equation] [Equation]
If it is assumed that the multiple outputs are independently sampled from p ( y i | x i ) , then equationcan be expressed as
[Equation]
where the reference density function is
[Equation]
The Dirac delta function is defined as
[Equation]
The form of equation 17 is of an unnormalised KL divergence.
Thus, maximising equation 14 may be related to minimising a KL divergence between p (cid:0) Y ref (cid:12)(cid:12) X (cid:1) and the reference density function, under several assumptions.
Before the marginal log-likelihood in equation 14 can be computed, the output density function over multiple output samples first needs to be defined.
The diagonal covariance in the output density function of a standard GP in equation 3 implies that the outputs from different training samples are conditionally independent of each other when given the latent variables.
Thus when considering having multiple output samples for each input, this conditional independence allows the output density function to be factorised as
[Equation]
21) where
[Equation]
and
[Equation] [Equation]
The factorisation and diagonal covariance in equation 20 assume that the multiple output samples for the same input and across different inputs are independent of each other, when given the latent variable.
As a result, the output density does not depend on the order of the multiple output samples for each input.
This can be explicitly seen in the re-parameterisation of the multiple outputs as the empirical mean µ and empirical biased standard deviation η , both of which are independent of the ordering of the output samples.
Also, equation 21 may seem unnormalised over µ and η .
However, it should be noted that p ( Y | f ) is expected to sum to one when integrated over Y , and not over µ and η .
Using equation 21, the marginal likelihood can be expressed as
[Equation]
where
[Equation]
Therefore, when the application domain requires that the training set comprises multiple output samples for each input, all belonging to the same task, the GP hyper-parameters can be optimised by maximising this marginal log-likelihood.
4. I NFERENCE
When performing evaluation, one or more output samples can be predicted for each input test sample, inferred from the predictive density function.
To compute the predictive density function, the joint density between the training set outputs and the test latent variables first needs to be computed.
When the training set comprises multiple output samples per input, this can be expressed as
[Equation]
Analogously to equation
[Equation]
the predictive density over the test latent variables is
[Equation]
where
[Equation]
The predictive density over the test outputs is then
[Equation]
Finally, a single scalar output can be inferred for each test input by choosing the mean of this predictive density, similarly to how a score is chosen from equation 12.
Alternatively, multiple outputs can be generated by repeated sampling from this predictive density.
The covariance of the predictive densities in both equations 12 and 33 are independent of the training set outputs.
These covariances depend only on the training set inputs and the hyper-parameters.
As such, information about the proximity between the multiple output samples for the same input in the training set does not influence the resulting predictive covariance.
In fact, the GP formulation is such that whether the training set output values are near or far from each other, for two inputs that are near as measured by the kernel, does not influence the resulting predictive covariance for test inputs near to these training inputs. What does affect the predictive covariance is the distance between the test and training set inputs.
This implies that a GP may be more suited for computing distributional uncertainty, and less so for data uncertainty (Xu et al., 2021).
Perhaps, a future research direction may be to allow the GP to take training set output information into account for the predictive covariance.
S POKEN LANGUAGE ASSESSMENT
Section 4 proposes a modification of a single-output GP, to allow the training set to comprise multiple output samples for each input, where it is assumed that all output samples for the same input are related to the same task.
This formulation may be applicable for tasks that are not strictly deterministic.
For example, noise in the experimental observation may induce a variance in the output, a task may have multiple valid output values that can correspond to the same input value, or a task may be subjective such that different human experts may annotate the same input with different output values.
In such scenarios, including multiple output samples per input sample in the training set may be more informative than using only a single output sample for each input.
5. T ASK
Examples of subjective tasks are language assessment, emotion recognition (Busso et al., 2008), and medical diagnostics.
The experiments in this paper are applied to SLA.
In SLA, the input is an audio recording of speech from a student.
The task is to predict a score for that input that is similar to a score that an expert human rater would have given.
This score is meant to relate to aspects of the oral proficiency of the student.
Examples of potential assessed aspects include the pronunciation accuracy, intonation, fluency, and prosody.
In practice, examination committees often try to standardise the assessment criteria through a rubric.
However, it may be difficult to concisely document a rubric that takes all possible circumstances into account.
As such, much may still be left to the human rater’s subjective opinion when assigning a score.
To account for this subjectivity, training datasets for SLA may include the reference scores from multiple human raters for each input.
It is standard practice to compute a combined reference score, by taking either the majority vote (Lin & Wang, 2021), median (Zhang et al., 2021b), or mean (Zhang et al., 2021b) of these multiple scores.
An automatic SLA model may then be trained toward and evaluated against this combined reference.
Examples of such approaches are Metallinou &
Cheng (2014); Duan & Chen (2020); Lin & Wang (2021) using NNs, and van Dalen et al.
(2015) using GPs.
However, collapsing the scores from multiple human raters into a single score may result in a loss of information about the output uncertainty that should be associated with each input.
The proposed framework allows a GP to take into account multiple output samples for each input in the training set.
As can be seen from equation 31, the predictive mean under the proposed formulation is dependent on the mean of the multiple output samples, µ , and independent of η .
This is similar to a standard
GP formulation, when using a mean combination of the multiple rater scores to get the reference.
As mentioned previously, the predictive covariance is independent of the training set outputs.
Thus, during inference, the remaining difference between the standard GP and the proposed modification is in the hyper-parameters.
The proposed approach optimises the hyper-parameters by maximising the marginal log-likelihood of all training set outputs, which may be related to minimising the KL divergence between this marginal likelihood and the reference density function.
As opposed to this, the standard approach maximises the marginal log-likelihood of the combined output reference.
5. E VALUATION MEASURES
Previous works on SLA often evaluate a model’s performance by comparing the predicted output score from the model against the combined reference score.
Two evaluation measures are often used, the Pearson’s Correlation Coefficient (PCC) and the Mean Squared Error (MSE) (Zhang et al., 2021b).
These are written out in appendix A. Both measure how well the predicted score matches with the combined reference score.
This assumes that the combined reference score is correct.
However, comparing only against the combined reference score omits information about the agreement between the multiple human raters.
This agreement may be representative of a reference of the uncertainty that should be associated with each input.
It may be useful to assess how well the predictive density agrees with this reference uncertainty.
One approach may be to evaluate the model using the unnormalised continuous KL divergence between the predictive and reference densities,
[Equation]
[Equation] where M is the test set size.
5. D ISCRETE PREDICTION POSTERIOR
One of the few available public SLA datasets is speechocean762 (Zhang et al., 2021b).
In this dataset’s publication of Zhang et al.
(2021b), the standardised method of evaluating a model is established as first rounding both the predicted and reference scores to integers, before computing the PCC and MSE.
This shifts SLA toward a classification-like task, and further from being a fully regression task.
Abiding by this standardisation, the discretised predictive distribution can be com- puted by integrating over the scores that would have been rounded to each integer,
[Equation]
where c are the possible integer score classes [Equation] and ´ y represents a discrete random variable.
The unnormalised continuous KL divergence may have limited interpretability [Equation] because is it not lower-bounded by zero.
Instead, the agreement between the uncertainties expressed in the predictive and reference distributions can be evaluated using a discrete KL divergence,
[Equation]
which is lower-bounded by zero.
The discrete reference distribution can be computed from the multiple human rater scores as
[Equation]
where the Kronecker delta function is
[Equation]
This assumes that the scores from the multiple raters are provided as integers.
E XPERIMENTS 6. D ATASETS
Experiments were performed on the speechocean762 dataset (Zhang et al., 2021b) and an internal
Tamil language dataset.
The public speechocean762 dataset allows for experimental repeatability, while the internal Tamil dataset is used to verify that the observed trends are generalisable.
Speechocean762 comprises a training and test set, each with 2500 sentences and 125 disjoint speakers.
The native Mandarin speakers read the sentences in English.
Each sentence is annotated with a variety of score types at different linguistic levels, but only the sentence-level pronunciation accuracy was used in this paper.
Each sentence is annotated by 5 human raters, each assigning an integer score between 0 and 10.
The Tamil language dataset (Zhang et al., 2021a) comprises a total of 4211 sentences, read by 100 speakers, aged between 9 to 16 years, from Singapore.
This was split into training and test sets with 2118 and 2093 sentences respectively, with speaker disjointment.
Stratified sampling was used to split the data, to ensure similar demographics between the different educational levels and assessment scores across the two sets.
Each sentence was annotated with pronunciation accuracies by 4 human raters, with integer scores between 1 and 5.
For both datasets, combined reference scores were computed as a mean of the multiple rater scores.
This differs from Zhang et al.
(2021b), which instead computes the median score.
Using the mean may yield a baseline that is a more appropriate match for the formulation in equation 21.
Following the standard in Zhang et al.
(2021b), the predicted and combined reference scores were first rounded to the closest integers, before computing the PCC and MSE, for both datasets.
6. M ODELS
The feature extraction for a NN SLA model was similar to Zhang et al.
(2021a).
First, speech recognition models described in appendix B were trained.
These were then used to force align the audio with the transcriptions.
From the forced alignment, Goodness
Of Pronunciation (GOP) (Witt & Young, 2000), Log Phone Posterior (LPP)
(Hu et al., 2015), Log Posterior Ratio (Hu et al., 2015), and tempo (Zhang et al., 2021a) features were extracted for each phone.
Pitch features (Ghahremani et al., 2014) were extracted from the audio, and pooled across all frames within each phone.
A continuous skip-gram model (Mikolov et al., 2013), with one recurrent NN (Elman, 1990) hidden layer having 32 nodes, was trained on the training set non-silence phone sequences, and used to extract phone embeddings (Zhang et al., 2021a).
The GOP, LPP, LPR, tempo, pitch, and phone embedding features were concatenated to form one feature vector per phone.
In baseline NN SLA models, the sequence of feature vectors, with length equal to the number of phones in the sentence, were fed into a Bidirectional Long Short-Term Memory (BLSTM) layer (Graves & Schmidhuber, 2005), with 32 nodes per direction.
The output was then pooled across all phones in the sequence with equal weights, and fed through a linear layer.
Two forms of NN output layers were used.
In the first variation, referred to as NN scalar , the linear layer had a single dimensional output, which then passed through a sigmoid (Zhang et al., 2021a).
This sigmoid was then scaled to the bounds of the output score range, and the computed output was treated as the predicted score.
Such a regression model was trained by minimising the MSE toward the combined reference score.
In the second variation, referred to as NN categorical , the linear layer output dimension was equal to the number of integer scores, and the output was fed through a softmax (Duan & Chen, 2020).
This classification model was trained using the cross-entropy criterion toward the combined reference score.
Both models used dropout (Srivastava et al., 2014) with an omission probability of 60%, before the BLSTM and linear layers.
It may not be straight forward to allow a GP to utilise sequential inputs.
Work in Lodhi et al.
(2000) investigates designing kernels to operate on sequences, while van Dalen et al.
(2015) extracts handcrafted sentence-level features for SLA.
In this paper, the activations after the NN pooling layer were treated as bottleneck features (Konig et al., 1998), and used as sentence-level inputs to the
GP.
A principle component analysis whitening transformation, estimated on the training set, was applied to these features to better abide by the diagonal covariance assumption of the scalar length hyper-parameter in the kernel in equation
6. S TATISTICAL SIGNIFICANCE
The statistical significance, ρ , of the difference between either the MSE or KL divergence evaluation measures of two models was computed using a two-tailed paired t -test.
As seen in equation 39, the
PCC is not easily expressed as a sum over data points, making it difficult to apply the central limit theorem.
The significance was instead computed using the Z ⋆ 1 approach in (Steiger, 1980).
Here, an approximately normally distributed transformation (Dunn & Clark, 1969) was first computed from the two PCCs being compared.
Then the significance was computed as the two-tailed cumulative density of this transformed quantity.
6. B OTTLENECK FEATURES AS INPUTS TO A G AUSSIAN PROCESS
The first experiment investigates the soundness of using bottleneck features as inputs to a GP, and the impact that the training approach used for the NN bottleneck feature extractor can have.
The NN scalar and NN categorical models were trained as described in section 6.2.
Sentence-level bottleneck features from after the pooling layer were computed, and used as inputs to the GP.
The GP hyper-parameters were optimised separately for each of the feature types, by using gradient descent to maximise the marginal log-likelihood of the training set, as in equation 4.
The mean combined score between the multiple raters was used as the training set reference output.
The results in table 1 show that both the NN scalar and NN categorical models yield comparable PCC and
MSE performances, with no clear victor over both measures.
A GP that uses the NN scalar bottleneck features may yield better PCC and MSE performances than when using NN categorical features.
The statistical significance between GPs that use either type of features are ρ PCC = 0 . 033 and ρ MSE = 0 .
078 for speechocean762, and ρ PCC = 0 .
334 and ρ MSE = 0 .
617 for Tamil.
Thus the performance differences may not be significant.
Although not significant, the improvements from using bottleneck features from NN scalar are consistent across both datasets.
Therefore, all subsequent experiments use bottleneck features from NN scalar as inputs to the GP.
The GP and NN SLA models exhibit comparable performances.
[Table]
6. M ULTIPLE OUTPUT SAMPLES PER INPUT SAMPLE
The next experiment assesses the proposed extension [Table] of allowing the GP to use the separate output scores from multiple human raters in the training set.
When using multiple rater scores, the GP hyper-parameters were optimised by maximising equation [Table] and the predicted scores were inferred using equation [Table]
[Table]
Table 2 compares GPs that use either only the mean combined reference score or the separate scores from the multiple raters in the training set.
The PCC and MSE results may suggest that the proposed approach yields performance gains, but these may not be significant, with ρ PCC = 0 .
065 and ρ MSE = 0 .
036 for speechocean762, and ρ PCC = 0 .
624 and ρ MSE = 0 .
585 for Tamil.
The PCC and MSE compare the prediction against the mean combined reference, thereby assuming that this mean combined reference is correct.
Furthermore, this omits consideration of how well the output uncertainty expressed by the model matches the uncertainty between the human raters.
The KL divergence is used to assess this uncertainty matching.
As a baseline, the discrete KL divergence of the NN categorical model is 1.26 for speechocean762 and 0.70 for Tamil.
The results show that using the separate scores from multiple training set raters improves both the continuous and discrete KL divergences of the GP, with significances of ρ discrete-KL < 0 .
001 for the discrete KL divergence of both datasets.
This suggests that the proposed extension allows the GP predictive density function or distribution to better match that represented by the collection of scores from the multiple raters.
Reiterating from section 5.1, the primary difference between the standard and proposed approaches is in the estimation of the hyper-parameters during training.
C ONCLUSION
This work has proposed an extension to the GP, to allow for multiple output samples for each input in the training set.
The hyper-parameters of such a model can be optimised using a criterion that may be related to a KL divergence.
Inference is performed in a fairly similar way to a standard GP, with the main difference being in the hyper-parameters that were optimised differently.
This allowed the predictive density to better match the uncertainty expressed by the collection of multiple human annotations.
A A PPENDIX : S TANDARD EVALUATION MEASURES
The Pearson’s correlation coefficient is computed as
[Equation]
where the empirical mean is
[Equation]
The mean squared error is computed as
[Equation]
B A PPENDIX : S PEECH RECOGNITION MODELS
The speech recognition models used to compute the forced alignment between the audio and the transcripts for both datasets were phonetic hybrid (Bourlard & Morgan, 1994) models trained using the Kaldi toolkit (Povey et al., 2011).
The speech recognition model used to align speechocean762 was constructed following the steps that are used in Zhang et al.
(2021b).
This trained a model on the Librispeech 960 hours training set (Panayotov et al., 2015), following the standard Kaldi recipe.
In this recipe, a 5 layer Time Delay
Neural Network (TDNN) (Waibel et al., 1989) acoustic model with Rectified Linear Unit (ReLU) (Glorot et al., was trained toward the cross-entropy criterion.
This had about 19.1 million parameters.
The crossentropy targets were obtained from a forced alignment using a triphone Gaussian mixture model hidden Markov model (Juang, 1985).
The input features to the TDNN were 40 dimensional Mel frequency cepstral coefficients (Mermelstein, 1976) and 100 dimensional i-vectors (Dehak et al., 2011).
The acoustic model outputs were 5672 tied triphone states (Young et al., 1994).
The Tamil language speech recognition model was trained on 220 hours of closed-talking speech from 710 adults.
The acoustic model comprised 3 long short-term memory layers and 6 TDNN layers with ReLU activations, interleaved together (Peddinti et al., 2018), with a total right context of 20 frames.
This used 29 dimensional filterbank features, was trained using the lattice-free implementation of the maximum mutual information criterion (Povey et al., 2016), and had around 8. million parameters.
The outputs were 1888 tied left diphone states.