L ABEL R EFINING : A SEMI - SUPERVISED METHOD TO EXTRACT VOICE CHARACTERISTICS WITHOUT GROUND TRUTH
Anonymous authors
Paper under double-blind review
A BSTRACT
A characteristic is a distinctive trait shared by a group of observations which may be used to identify them.
In the context of voice casting for audiovisual productions, characteristic extraction has an important role since it can help explaining the decisions of a voice recommendation system, or give modalities to the user with the aim to express a voice search request.
Unfortunately, the lack of standard taxonomy to describe comedian voices prevents the implementation of an annotation protocol.
To address this problem, we propose a new semi-supervised learning method entitled Label Reﬁning that consists in extracting reﬁned labels (e.g. vocal characteristics) from known initial labels (e.g. character played in a recording).
Our proposed method ﬁrst suggests using a representation extractor based on the initial labels, then computing reﬁned labels using a clustering algorithm to ﬁnally train a reﬁned representation extractor.
The method is validated by applying Label Reﬁning on recordings from the MassEffect 3 video game.
Experiments show that, using a subsidiary corpus, it is possible to bring out interesting voice characteristics without any a priori knowledge.
I NTRODUCTION
Localization of a multimedia work, such as a video game or a motion picture, consists in making changes and adaptations to the original work so that international distributors can market the resulting product in a target country. For this purpose, companies need to translate their works into a target language. Various options are available to them to give substance to their translation. Subtitling is the simplest and cheapest option, but not the most practical one for spectators: a large audience is more comfortable with hearing speech, in general in their mother tongue, rather than reading subtitles while hearing speech in another language.
Dubbing, which is more expensive and takes longer to set up, better retains the audience’s immersion by replacing the original voices with dubbing comedian voices.
It involves a voice selection process, which means choosing among several candidate voices in a target language.
This selection is referred to as voice casting.
With the emergence of streaming platforms (e.g. Netﬂix, Disney+, Amazon Prime Video) and the rapid development of the video game industry, the number of works to be localized is increasing dramatically.
As a result, more and more actors are available on the voice market.
Vocal Casting is an artisanal process that requires a lot of precision and is carried out by human operators.
As it is, it cannot be applied to a very large number of actors.
Operators very often rely on their memory, which is limited to a small number of actors with whom they have worked and in whom they have conﬁdence.
However, this operation can degrade the immersion of the audience since the most oftused voices become easily recognizable.
They then become references and create cognitive biases in the public’s perception of the character. Automatic actor search and recommendation tools, based on voice processing, would help operators ﬁnd new actors who would then enrich the vocal diversity of the works to the delight of the audience. In this article, we wish to address some of the complexity of voice casting and we start with the voices of professional actors playing characters for the gaming industry. The articles Gresse et al.
(2020; 2019; 2017); Malik & Nguyen (2021); Obin & Roebel (2016); Obin et al. ; Quillot et al.
(2021b;a) cited in this paper are the only ones dealing with the speech-processing-based automation of Voice Casting in the literature to our knowledge. We rely on Gresse et al.
(2020), a recent work which deﬁnes the p -vector, a neural-network-based representation of the voice dedicated to the character aspects in acted voices.
The application context in Gresse et al. (2020) (like in related previous works Gresse et al. (2017; 2019)) is voice casting for audio dubbing.
The ﬁnal objective of this work is to assess how close an actor – acting in a target language – is to the voice of a given character, speaking in a source language. This can be referred to as character-based voice similarity.
In contrast to Obin et al. ; Obin & Roebel (2016), which proposed a voice similarity system for the acted voice based on data labeled with speech classes (age, gender, emotions ...) by an expert, the p -vector approach does not use any expert labels. This representation is learnt from audio associated with the played character label, without any additional information.
Also, the authors of Quillot et al.
(2021a) recently posed the following hypothesis: when a character is played, the actor adds to the voice some information that characterizes the character and is independent of the speaker identity. They then introduced SICI (Speaker-Independent Character Information) and showed that there is speaker-independent character information in the acted voice and that it is present in p -vectors pretrained on a speaker task Quillot et al. (2021b). Although these approaches deal with ﬁnal works (video games), another one consists in working directly on the decision data from Artistic Directors.
Recently, researchers from Warner Bros. evaluated their models on this kind of data. Unfortunately, these data are sensitive and their acquisition is not trivial since it requires to work on the critical voice casting process. This is why we decided to position ourselves in a task quite different from Malik & Nguyen (2021), since we do not use
Artistic Director decisions.
Voice similarity systems alone are not enough for Voice Casting operators. They do not trust the computed recommendations and need a more complete interface to interact with the system.
Many works propose explainability approaches Do ˇ silovi ´ c et al. but these are too complicated to be understood by a user who is not expert in computer science, and more particularly in machine learning. We strongly believe that the introduction of explanatory labels will help the operators understand the decisions by giving, for example, the list of speech characteristics shared by two recordings/actors.
Such labels could also be used to express queries in voice search systems. Unfortunately, voice casting is an underdeﬁned task Bonastre (2019). The operators themselves cannot formally deﬁne their decision criteria.
There is also no standard taxonomy in cinema to describe voices, making it impossible to create an annotation protocol. The question is then: how can we extract features from voices when we have no a priori knowledge of them?
Different representation learning approaches exist in the literature, but none of them corresponds to our expectations.
1) Feature extraction surveyed in Bengio et al. (2014) consists in representing an observation by a vector of reals but not in the form of binary vectors that we call characteristics in this paper. Supervised methods like Hu et al. (2017) require labels that we do not have access to in order to extract characteristics.
Unsupervised methods like Peri et al.
(2020) propose generic representations from which we could not extract characteristics dedicated to the domain of acted voice and character identity. Hence, to address the lack of method to extract characteristics without ground truth, this article proposes a totally new approach entitled Label Reﬁning.
It stats with training a representation extractor using non-expert initial labels. A clustering algorithm is then employed with the purpose to emphasizing characteristics shared by groups of observations which we term reﬁned labels. A last step consists in extracting a new representation from reﬁned labels and comparing it with the initial representation by evaluating it on a downstream task.
Unlike generic representations that use no label, this method has the advantage of guiding the extracted representations by using the non-expert initial labels. To validate this method, we apply it to dubbing and more speciﬁcally to the extraction of characteristics from actors’ voices. We therefore p -vector features from characters played in a video game.
We apply a clustering algorithm, k -means, in the p -vector space to extract reﬁned labels. In order to evaluate the quality of the reﬁned labels, we train a neural network to extract reﬁned p -vectors. We then subject them to a voice character similarity task to compare them with the initial p -vectors and thus measure the loss or gain of dedicated character information.
In this experiment, we test the following hypothesis: during the clustering step, using the recordings of the training corpus only highlights the initial labels, the characters. During this step, a second corpus (close to the domain) is needed to highlight the vocal features extracted by the p -vectors. Since distance plays an essen- tial role in the k -means algorithm, we also explore the use of cosine and Mahalanobis distances to compare with Euclidean one. Section 2 presents Label Reﬁning.
We deﬁne the term “reﬁned label” and present the method. The experimental protocol to apply Label Reﬁning is described on Section 3. In Section 4, we present experiments and discuss about the obtained results. For reproducibility reasons, scripts and models are available on GitHub 1 .
Finally, we conclude by presenting some takeaways and possible directions for future work in Section 5.
Before experimenting, we ﬁrst introduce the Label Reﬁning method.
It consists in extracting reﬁned labels from initial ones. In this section, we give a deﬁnition of reﬁned labels and we detail the method of extraction and evaluation.
2. T HE DEFINITION OF REFINED LABELS
In this work, we assume that labels are decomposable into various lower-level sublabels.
In our case, we wish to use the knowledge provided by the played character labels associated to the recordings with the aim to infer the characteristics such that the emotion played (e.g. agressive, happy), a voice color (e.g. white, red) or a timbre (e.g. husky, enlightened).
Equation 1 formalizes the deﬁnition of what a reﬁned label is.
Let R the set of all the reﬁned labels and a ( l ) i ∈ { r 1 , r 2 ...
r | R | } the i th reﬁned label associated to the initial label l .
Let us also specify that the number n of reﬁned labels is different depending on l .
Thus, from the knowledge given by initial labels in L we wish to ﬁnd the reﬁned labels a ( l ) i of each l .
[Equation]
Let us note the functions f : e (cid:55)→ l and g : e (cid:55)→ r where o is an observation (e.g. vocal recording).
The equation 2 gives ﬁctives examples.
It shows the expected images of the functions f and g according to two initial labels l 1 and l 2 .
In our model of the problem, we assume that a reﬁned label r may appear in observations associated to different initial labels.
Here, this is the case with r 1 which appears both in an observation associated to l 1 and an observation associated to l 2 .
l(cid:40)
f : o ( l 1 )(cid:55)→ lf : o ( l 1 )(cid:55)→ l, l(cid:40) g : o ( l 1 )(cid:55)→ rg : o ( l 1 )(cid:55)→ r
[Equation]
l(cid:40)
f : o ( l 2 )(cid:55)→ lf : o ( l 2 )(cid:55)→
l
[Equation]
l(cid:40) g
[Equation]
o ( l 2 )(cid:55)→ rg : o ( l 2 )(cid:55)→ r(2)
2. O UR PROPOSITION TO REFINE LABELS
Figure 1 presents the reﬁned label extraction method that we propose in three steps. 1 consists in training a system to extract a representation (i.e. vector, hash Roussev (2011)) of the observation o ∈
O using the initial labels l ∈ L (e.g. label of the character played in a recording).
2 consists in using a clustering algorithm in order to associate new labels r ∈ R to the observations
O .consists ﬁnally in training an new system to extract reﬁned labels by using labels r .
In order to evaluate the relevance of labels r , we use two methods.
The ﬁrst method, taking place at step 2 , consists in applying different clustering quality measures such as v -measure Rosenberg & Hirschberg (2007) or the purity Ajmera et al.
Applied to the step 1 and step 3 , the second method consists in extracting a representation of the observations O and evaluating it on paper is accepted. [Figure] [Figure] x
[Figure]
c t p
[Figure] v [Figure] t
[Figure]
r
[Figure]
r
[Figure]
r
[Figure]
s s o c
i a t e o n e g r o u p
[Figure]
e [Figure] a downstream task. The downstream task can be diverse: speaker recognition, speech recognition or, in our case, character voice similarity.
Thanks to the second method, it is possible to highlight the gain or loss of information by comparing the results obtained at steps 1 and 3.
In this work, we postulate that the reﬁned labels are pertinent if they do not lead to a loss of information when extracting reﬁned representation.
Depending on the clustering algorithm, it is likely to have to try different values of k . In the following experiments, we use k -means and test different values of k . In this context, we apply the method for all the k values and compare the obtained results. We postulate that Label Reﬁning is validated if the number of reﬁned labels k differs from the number of characters and if the character information is at least preserved in the reﬁned representation.
E XPERIMENTAL PROTOCOL
In the previous section, we introduced the generic method for extracting reﬁned labels.
We describe in this Section the protocol we follow to validate the method. We apply Label Reﬁning on acted voice recordings in order to extract voice characteristics dedicated to the character played. A resume of the protocol is presented on the ﬁgure 2. Each step is detailed on the next subsections.
3. M AIN (M ASS E FFECT ) AND SECOND (S KYRIM ) C ORPORA
We use MassEffect data as the main corpora to train feature extractor and evaluate our system on downstream task.
We also explore the use of a second dataset from the video game Skyrim during the second step (i.e. clustering algorithm) with the aim to emphasize voice characteristics models by previous trained p -vectors. This subsection present both MassEffect and Skyrim datasets.
3.1. M ASS E FFECT , THE DATASET USED TO TRAIN AND EVALUATE p - VECTORS
The main corpus is composed of voice recordings coming from the Mass-Effect 3 role-playing game.
Contrary to movies, voices of video games present some particularities explained in Gresse et al.
(2019) (radio effects are present in the original voices of our corpus).
They are also easier to collect since they are separated from the ambient sounds even in their ﬁnal form, contrary to the dubbed ﬁlm archives.
Originally released in English, the game has been translated and revoiced in several other languages.
In our experiments, we use the English and French versions of the audio sequences, representing about 7 . 5 hours of speech in each language.
Segments (or recordings) are 3 .
5 seconds long on average.
A character is then deﬁned by a unique French-English couple of two distinct speakers.
To avoid speaker identity biases, we consider only a small subset where we are certain that none of the actors play more than one character.
A single audio segment corresponds to a unique speaking slot from an actor in a particular language.
We then apply a ﬁlter that keeps only recordings longer than 1 second.
Finally, we only keep the 16 characters for which we have the largest number of recordings, as Gresse et al.
(2019) did.
In order to measure the ability of our systems to generalize, we propose a breaking down protocol.
We only keep 12 characters, and their 24 corresponding speakers, for the training phase and we use the remaining 4 characters for the testing phase.
We break the corpus down into training, validation and test sets.
The train and val sets are composed of different recordings, but coming from the same 12 characters.
To build the train , val and test subsets, we randomly select for each character 144 ,and 180 recordings respectively, while balancing the number of French and English recordings.
We then have a total of 1 , 728 ( train ), 432 ( val ) and 720 ( test ) recordings.
We name the three sets S n . S
KYRIM
Skyrim is also a known video game.
We use its recordings as a second corpus in addition to the one of MassEffect previously presented.
This dataset can be viewed as a validation set.
We propose to use this corpus to emphasize the characteristics of voices.
Indeed, the validation set of MassEffect suffers from a lack of variability since it contains the same characters as the training set.
Unfortunately, we do not have available MassEffect data with unknown characters to add to the validation set since they are used on the test.
Thus, we assume that using Skyrim as a second validation set will help to bring more variability and highlight voice characteristics extracted from the p -vectors trained on MassEffect.
This subset is composed by 18 , 250 recordings of different 32 characters.
As on MassEffect, each recording played in english is associate to a french one and vice-versa
but we don’t balance the number of recording by character.
3. E XTRACTING VOICE REPRESENTATION FROM CHARACTER LABELS
This part of the protocol corresponds to the step 1 described in the Figure 2.
In order to extract the p -vectors, we follow scrupulously the best architecture described in Quillot et al.
(2021b).
Training this architecture need to follow two steps: 1) training a speaker identiﬁcation system following a x -vector architecture on Voxceleb 2 corpora (Chung et al.
2) replacing the output softmax of this system with p -vector layers, and training it to recognise the characters of the MassEffect dataset.
3.2. R EFINING LABELS WITH K - MEANS ALGORITHM
This part of the protocol corresponds to the step 2 described in the Figure 2.
In this work, we wish to validate our method described in 2.2.
For this purpose, we focus on the k -means algorithm. Unlike others, this algorithm is easier to comprehend.
We use the implementation of the toolkit sklearn with the default parameters, namely: k − means + + as initializer of the barycentres in order to converge more quickly to the solution.
The max number of iterations is 300 of non convergence.
The algorithm is runned 10 times, then we select the clustering model with the best intertia. [Table]
3. E XTRACTING REFINED VOICE REPRESENTATION BY USING CLUSTERING GROUPS
This part of the protocol corresponds to the step 3 described in the Figure 2.
Once the clustering algorithm is complete, we build for each k -value a new neural network using the same neural architecture as used for the p -vector.
Instead of predicting the initial labels, this new network is trained to predict the reﬁned labels r ∈ R using the clustering algorithm.
After training, we can extract from its last layer a representation of the provided record that we name p -reﬁned-vectors.
The used architecture is exactly the same as for the step 1 since this neural network is also ﬁrstly pre-trained on a speaker identiﬁcation task by using Voxceleb 2.
Only the last layer of the p -vector network is changed to correspond to the number k of groups (of the reﬁned labels r ) instead of the number of character labels l .
3. E VALUATION
In this subsection, we present the two ways to evalute the quality of the reﬁning.
The further one is computing clustering metrics.
The second one is evaluating the representation extracted using the labels on a downstream task.
We give all the details in this subsection.
3.4. C OMPUTING CLUSTERING METRICS
To evaluate a clustering model, we use the following measures: entropy, v-measure Rosenberg &
Hirschberg (2007), class purity, group purity, purity-K Ajmera et al.
These measures require to know for each entity in the database the group to which the algorithm has associated it as well as its original label.
Since their formal deﬁnition are deﬁned in the cited papers, we only present below their meaning and behaviour.
These measures are calculated from two different perspectives.
The entropy, homogeneity and group purity measures check whether each group is associated with entities belonging to the same class.
They provide a group view.
The completeness and class purity measures check if each class is associated to entities belonging to the same group.
They provide a class view.
To analyze the behavior of these measures, let C a clustering model learnt on the observations X with parameter k , the a priori number of clusters.
Let f the function which measure the quality of the model C (e.g. entropy) on X .
The table 1 shows the behaviour of the measures by computing for each of them the limit of f i k
→ + ∞ et k → 0 .
These measures all have their meaning and are generally misinterpreted.
This is particularly the case for the entropy applied to clustering algorithms.
The latter is a measure similar to that of homogeneity that has been inverted.
Although it helps to compare two systems, it has a major drawback.
It is bounded between 0 and + ∞ .
Interpreting the measure of a single model is therefore difﬁcult unlike homogeneity which is bounded between 0 and 1 .
The latter thus gives a rate of homogeneity that entropy cannot give.
Without going into details here, the group purity is a measure very similar to the homogeneity.
The measures of completeness and class purity, in addition to being very similar, are the antithesis of the previous measures.
They are no longer measures of group homogeneity but measures of class homogeneity.
The closer the value is to 1 , the more the class in question is observed in only one group.
The closer the value is to 0 , the more the class in question is observed in a large number of groups at similar quantities.
Hybrid measures are weighted averages of class and group measures.
Their limitations in k → + ∞ and k → 0 are not clear as they depend on the balance of power between homogeneity (group purity) and completeness (class purity).
Demonstrating the limits of each function is quite simple.
For homogeneity and group purity, it is maximal when the elements of each group belong to the same class.
When we reach k = | X | , each element is then assigned to a cluster and each cluster is assigned to an element.
There is then only one class for each group.
The homogeneity is thus maximal, the group purity too.
In the opposite case, when k = 1 , all observations belong to the same group.
The group is then maximally heterogeneous and the values of homogeneity and group purity then tend towards 0.
For completeness and class purity, the reasoning is similar and then becomes trivial.
3.4. E VALUATING VOICE REPRESENTATION
As described in Quillot et al. (2021b), we evaluate p -vectors on a character voice similarity task using a character-similarity measure based on a Siamese network.
We generate Target trials composed of pairs of recordings belonging to the same character and Non-target trials made up of recordings belonging to two different characters.
To avoid any bias, the number of targets and non-targets is balanced, as well as the number of pairs between two actors.
So, we generate 124 , 416 ( train ), 7 , 776 ( val ) and 64 , 800 ( test ) trials using S n dataset (MassEffect).
Skyrim is not used during this evaluation step.
The threshold is set a posteriori at the Equal Error Rate (EER) point.
System performance is then expressed in terms of accuracy on the test.
E XPERIMENTS AND R ESULTS
In the experiments we present in the following sections, we decided to test different values of k in order to limit the number of calculations to be performed.
Several values seemed obvious to test such as 12, 24, 4 and 8 which are the number of characters and speakers respectively in the train and test.
We also assume that the number of speech features we are looking for is higher than the number of characters our system has to learn to model.
We therefore also tested the values 18, 32, 48 and 64 which are arbitrary values but greater than 12.
On the other hand, we have also tested k = 2 and k = 6 as values less than 4 and 12 as a counter-hypothesis.
With the aim to compare results obtained on the reﬁned p -vectors, we ﬁrst evaluate original pvectors. These p -vectors are trained by using S n . They are evaluated by training 10 siamese neural networks and selecting the best on the validation. The best model gives us 0 .
69 of accuracy on the test.
This corresponds to the actual state-of-art performance presented in Quillot et al.
(2021b). 4. R EFINING CHARACTER LABELS BY USING M ASS E FFECT In Table 2, we plot the observed metrics v-measure and purity-K on our clustering models computed on the training dataset of S n (MassEffect).
We then train 10 siamese neural networks and select the best on the validation.
Its accuracy score on the validation and the test are plotted on the last two lines.
For the three metrics presented in the table, the higher the value, the better the system.
On the validation set, the metrics ( v -measure and purity-K) highlight the 12 characters contained in it and known to the system.
Indeed, we naturally have higher scores for k = 12 with 0 .
91 for the v -measure and 0 .
90 for the purity-K.
On the test, we observe a somewhat different phenomenon.
Remember that the test is composed of 4 characters.
If the points – which are projections of the recordings into p -vector space – were perfectly discriminated by the neural classiﬁcation system, they would then be organized into groups of records belonging to the same character.
We would therefore have stronger measure scores for k = 4 .
In our case, we can observe this phenomenon with a loss due to the difﬁculty for the system to separate the records of characters that are unknown to it.
Indeed, the highest values are at k =for the v -measure with 0 .
58 and at k = 2 for the purity with 0 .
These groups are ﬁnally the closest to 4 and show a loss of quality of the representations on unknown data.
However, they do not show, a priori, the existence of features that could interest us. [Table]
Finally [Table] we are interested in the performances obtained on the Siamese networks that we have trained.
Recall that reﬁned p -vectors are provided as input to these Siamese networks which are trained on a character veriﬁcation task – character voice similarity.
These vectors are extracted from a neural network trained to recognize the clustering group (reﬁned label) corresponding the given recording (observation).
Without surprise, we observe a best accuracy for k = 12 .
This corresponds to the number of characters.
Contrary to the previous metrics, we do not observe a normal distribution in the results.
Indeed, when k is moving away from 12, the obtained accuracy is not necessarily lower.
For instance, at k = 64 the obtained accuracy is not signiﬁcantly lower than the one obtained at k = 12 .
Even more interesting, the accuracy obtained on the test subset is about 0 .
70 that is higher than the state-of-the-art system, 0 .
4. R EFINING CHARACTER LABELS BY USING S KYRIM AS A SUBSIDIARY CORPUS
In a second experiment, we wish to verify if it is possible to use a subsidiary corpus to extract voice characteristics.
For this purpose, we propose this time to apply the clustering algorithm on the pvectors representations extracted from the Skyrim corpus for step 2.
The p -vectors have learned to highlight the played character but it is impossible to bring out the vocal characteristics it extracts to perform this classiﬁcation.
We hope that the use of a secondary corpus such as Skyrim will create enough variability to bring out vocal characteristics from the initial labels.
For step 1, the p -vector extractor is exactly the same as the previous experiment.
It is trained on MassEffect with S n and the Skyrim data is completely unknown to it.
For step 2, we extract p -vectors using Skyrim data and perform on them the k -means algorithm.
For step 3, we ﬁnally associate a cluster to each recording of MassEffect and train reﬁned p -vectors as realized in the previous experiment.
Table 3 presents the obtained results.
We note × as accuracy result when labels are missing when we associate them to MassEffect data with the aim to train the step 3 neural network.
We can observe that this time the v -measure and the purity-K maximum scores move away from the number of characters that we originally had – 32 for Skyrim.
So we know that the semantics of these clusters differ from the semantics of the character labels.
In this example, purity-K highlights k = 2 .
In the k = 2 case, the two groupings primarily separate men and women.
Each group is therefore the gender characteristic of the voice.
For the v -measure, the score correlates with the number k of groups.
The best score is therefore for k = 64 , but it might have gone up by increasing k .
Unlike k = 2 , it is difﬁcult to understand the semantics of each group and to associate words that make sense to a human operator.
Note that these measures have no known bias unlike the previous experiment since here the number of characters in the Skyrim data is 32 and the k = 2 and k
=values are signiﬁcantly off.
4. A SSESSING THE IMPACT OF THE DISTANCE SELECTED FOR K - MEANS
As we know, distance in the k -means algorithm plays an essential role.
For this reason, we propose to realise the same experiment as presented in the previous section (4.2) but using cosine distance instead of Euclidian one. We also use the Mahalanobis distance k -means presented in Lapidot (2018) – without constraints to gain computing time.
The table 4 presents results obtained by using the cosine and the Mahalanobis distances.
We can observe that the Mahalanobis distance offers superior results on v -measure and purity-K compared to [Table]
[Table] the Euclidean distance and the cosine distance. Moreover, while the cosine distance tends to draw the same conclusions as the Euclidean distance through these measurements, the Mahalnobis distance differs in the number of clusters it highlights with the purity-K measure.
Indeed, the two other distances put forward k = 2 while this one puts forward k = 32 .
The training of Siamese networks seems to agree with the choice k = 32 for Mahalnobis.
However, this accuracy is signiﬁcantly lower than the one obtained with cosine distance, 0 . 64 < 0 .
Mahalanobis does not seem to ﬁt the data so as to conserve character information.
This article introduces Label Reﬁning, a semi-supervised method that consists in extracting reﬁned labels from initial ones. It meets the needs of characteristics extraction, especially vocal, without a priori knowledge. To validate this method, we applied it to dubbing and more speciﬁcally to the extraction of characteristics from actors’ voices.
A ﬁrst experiment showed that using the same data during all the steps of Label Reﬁning will only highlight the initial labels. A second experiment showed that using a subsidiary corpus during the clustering step permits to highlight reﬁned labels. It might also be noted that, in one case, our method brought to light gender characteristics. In the other case, giving labels intelligible meaning required further analysis.
A last experiment showed that the Euclidean distance seems to ﬁt the data better than cosine and Mahalanobis distances since it preserves the character information better.
With these experiments, we can conclude that Label Reﬁning permits to extract reﬁned labels that deviate from the original ones when using a subsidiary corpus.
These labels conserve or bring the wanted information, namely character information in our case.
While it is not straightforward to give meaning to these characteristics without a perceptive study, we propose in future works to build explainability methods based on Label Reﬁning, especially for voice similarity systems.