Optimal Transport-based Identity Matching for Identity-invariant Facial Expression Recognition
Author(s)
AffiliationAddress
Abstract
Identity-invariant facial expression recognition (FER) has been one of the challenging computer vision tasks.
Since conventional FER schemes do not explicitlyaddress the inter-identity variation of facial expressions, their neural network models still operate depending on facial identity.
This paper proposes to quantify theinter-identity variation by utilizing pairs of similar expressions explored througha specific matching process.
We formulate the identity matching process as an
Optimal Transport (OT) problem.
Specifically, to find pairs of similar expressionsfrom different identities, we define the inter-feature similarity as a transportationcost.
Then, optimal identity matching to find the optimal flow with minimum transportation cost is performed by Sinkhorn-Knopp iteration.
The proposed matchingmethod is not only easy to plug in to other models, but also requires only acceptable computational overhead.
Extensive simulations prove that the proposed FERmethod improves the PCC/CCC performance by up to 10% or more compared tothe runner-up on wild datasets.
Introduction
For sophisticated human-computer interaction (HCI), facial expression recognition (FER) must be able to accurately predict the emotions of people with different personalities or expressive styles.
On the other hand, to express the type and intensity of emotions in detail, VA FER,which annotates facial expressions in a continuous domainbased on two axes of Valence (V) and Arousal (A), is attracting attention [36].
Here, valence and arousal indicatethe positive/negative of emotions and the degree of activation, respectively.
Recently, many techniques for analyzingcomplex or fine-grained expressions have been developedto improve the regression or prediction performance of
VA FER [16, 27, 17].
However, it is still challenging toproperly learn facial expressions that differ from personto person due to heterogeneity in identity (ID) attributessuch as age, gender, and culture.
Thus, so-called ID-aware FER is receiving a lot of attention nowadays.
For example, [22, 2] learnedemotional factors inherent in strong facial expression samples through adversarial learning, andgenerated ID samples favorable to FER through GAN [14].
However, it is observed that conventionalmodels have difficulty learning the unique properties of facial expressions inherent in each ID, andabove all, they are still dependent on the ID change.
We believe that the inter-ID variation of facial expressions is the key to solving this problem.
Toanalyze the inter-ID variation, we propose a novel FER learning based on pair-wise similarityby finding pairs of similar expressions from different IDs.
This idea was inspired by cognitiveresearch [6, 48, 18] that analyzes facial expressions through comparison between IDs.
This paperrealizes our idea through a novel framework named Expression Learning with Identity Matching( ELIM ).
Specifically, ELIM finds ID pairs with similar expressions through ID matching based on theoptimal transport (OT) problem [8], and assigns relevance weights according to the similarity of IDpairs (see Fig. 1).
Next, based on the relevance weights, we compute and compare the tendency ofdifferences in facial expressions between IDs, i.e., the ID shift of facial expressions.
As discussedin Sec. 4, ID-invariant representations are generated by ID shift vectors quantified from ID pairs.
Finally, FER learning is performed through risk minimization [43].
Contributions of this paper are summarized as follows: (I) To realize ID-invariant FER, we propose an
FER framework ELIM that adaptively considers inter-ID variation of facial expressions.
Specifically,
ELIM learns facial expressions based on the similarity (or dissimilarity) between IDs obtained through
ID matching (see Fig. 4).
(II)
The state-of-the-art (SOTA) performance of ELIM is verified throughextensive experiments on various real-world datasets.
Especially, ELIM works more accuratelycompared to prior arts even for samples in which inconsistency between facial expressions and theemotion label predictions exists (see Fig. 3).
Related Work
Valence-Arousal (VA) FER is an approach based on the circumplex model [36] in which emotionsare distributed in a two-dimensional (2D) circular space.
VA FER has a wide research spectrum,from formulating mapping between features and continuous VA labels [17] to mapping complex andhigh-dimensional facial expression information on VA space [27, 37].
Recently, some interestingstudies have been reported on the so-called ID-invariant FER, which is designed to be robust toeven unseen IDs.
For example, [2] introduced a generative model to generate and analyze similarexpression examples from different IDs.
However, [2] is fundamentally dependent on the performanceof the generative model, and its performance is limited because only a single pair is compared toanalyze the expression difference between IDs.
From another point of view, several FER methodstried to learn intra-identity discrepancy of samples with different facial expressions [47, 39].
However,since they do not explicitly consider inter-ID variations, they inevitably suffer from performancedegradation for IDs that was never learned [41].
[5] proposed a personalized affective module basedon an adversarial autoencoder to learn features and generate samples having different expressions.
However, like [47, 39], this VA FER method also shows limited performance for unseen target ID.
On the other hand, since ELIM reflects the relative traits of expression between IDs in model learning,it can achieve reliable performance even on the test dataset including unseen IDs.
Distribution shift has been treated as very important not only in FER but also in various machinelearning tasks.
For example, distribution shifts between set of source domains and unseen targetdomains [32, 49] are seriously considered in the field of out-of-domain (OoD) generalization [34, 15].
In particular, normalization methods [52, 12] that generate domain-invariant representations byquantifying distribution shifts through domain-specific statistic are being actively studied in the OoDgeneralization field.
ELIM is differentiated from conventional normalization methods [52, 12] in thatit defines statistics from the OT problem that automatically matches pairs of similar facial expressions.
Note that ELIM is the first case applying the distribution shift concept to the field of VA FER (cf.
Problem 1).
Specifically, ELIM regards ID as a domain and quantifies ID shift vectors.
By normalizingfeatures with the corresponding ID vectors, ID-invariant representations can be generated.
Preliminaries
This section describes the problem formulation of ELIM and briefly reviews the OT problem responsible for optimal resource allocation.
3.1 Problem Formulation with Nomenclature
This paper focuses on the regression task to estimate VA labels y ( ∈ R 2 ) ⊂
Y from featuresz ( ∈ R d )
⊂ Z .
Here, Y and Z stand for the label and feature spaces, respectively.
The VA FER task generally assumes that labels depend on facial expressions rather than IDs.
However, this assumptiondoes not always hold due to ID shift, i.e., a phenomenon in which (marginal) distributions of IDs areshifted by their attributes even though the label prediction mechanism is the same: p 1 ( y | z )
=
p 2 ( y | z )and p 1 ( z ) ̸
= p 2 ( z ) .
Let p 1 and p 2 denote distributions of two different IDs.
We aim to find a predictorf : Z → Y that works robustly even when label predictions are inconsistent with the correspondingfacial expressions.
Formally,
Problem 1.
Let Z e = { z e } e ∈ supp( E tr ) be the training-purpose feature set collected from multiple IDs.
The corresponding VA label set Y e is defined similarly.
E tr is an index set of training environmentsand supp( · ) stands for its support set.
The cardinality of E tr is N .
Then, the risk of predictor f for
ID e is formulated as follows:
[Equation]
where L is a loss function based on mean-squared error (MSE).
Note that the goal of Problem 1 is to minimize the empirical risk [43] of f from as many IDs aspossible.
To minimize the risk of Eq.
1 derived to generalize ID shifts, we design a model to learnrepresentations of facial expressions invariant to ID shifts (cf.
Sec. 4.2).
3.2 Review of Optimal Transport
This section briefly review the OT problem that is being applied to various real-world applicationssuch as speech signal processing and biomedical technology [30, 46].
In this paper, the OT problemis applied to the VA FER task for a similar purpose to the above studies.
OT is known as a toolfor measuring the optimal distance between two distribution sets.
For example, suppose you aretransporting a product from a set of suppliers a = { a i | i
= 1 , 2 , · · · , n } to a set of demandersb = { b j | j = 1 , 2 , · · · , m } .
Also, assume that the i -th supplier a i has some unit(s) of products, andthe j -th
demander b
j requires some unit(s) of products.
Here, a i and b j mean the number of product(s)in a double sense.
The cost per unit transported from supplier i to demander j is C i,j ∈ R n × m + ,and the number of units transported is X i,j ∈ R n × m + .
The objective of the OT problem is to searchthe cheapest flow X ∗ =
{ X i,j | i = 1 , · · · , n, j = 1 , · · · , m } that can transport all products fromsuppliers to demanders at the minimum cost:
[Equation]
In order to find the solution X ∗ of the above problem, we can employ linear programming as atool.
However, since this tool does not necessarily provide a unique solution and it requires nearcubic complexity [1], it is difficult to apply to high-dimensional features.
So, we address this issuethrough a fast iteration solution, i.e., Sinkhorn-Knopp [8] (cf.
Appendix 1 ).
Sinkhorn-Knopp methodbased on the matrix scaling algorithm can obtain the unique optimal solution X ∗ with near-squarecomplexity [3].
Since this method computes X ∗ by aggregating the importance between all features,it can analyze both global and local distributions of data.
In this paper, we define a i and b j from thefeatures of each ID to capture the global representation of facial expressions, and perform the IDmatching process through the Sinkhorn-Knopp method.
Proposed Method: ELIM
This section proposes Expression Learning with Identity Matching ( ELIM ) for VA FER.
ELIMconsists of Sinkhorn-Knopp-based ID matching and ID-invariant expression learning.
Pre-processing.
Since ELIM regards ID as a domain, samples in the mini-batch need to be groupedaccording to ID.
One reference ID is randomly selected among N IDs in the mini-batch and theremaining N − 1 IDs are our target.
Then, a feature set Z e is constructed by the backbone ( f ϕ ) [29,19, 40] on an ID basis.
Next, ID matching process is performed between each target ID and thereference ID, thereby ID pairs corresponding to the target ID are constructed.
Early ID matching approach focused on calculating differences between features based on distancemetrics such as L2-norm and contrastive loss [20, 22].
However, since conventional methods basedon inter-sample matching can quantify even differences in attributes (e.g., hair, skin, etc.)
unrelated tofacial expressions, they cannot be free from the influence of ID.
In other words, previous works arenot robust to expression differences between IDs, i.e., inter-ID variation.
To overcome this drawback,even inter-ID variation should be learned.
To derive the information about inter-ID variation, reliable
ID pairs are required.
To acquire reliable ID pairs, we match samples from different IDs by using
Sinkhorn-Knopp iteration [8] which can consider both global and local distributions of samples.
As in Fig. 2, this section uses only a reference ID e 1 and a target ID e 2 as an example for convenience.
Let their feature sets be Z e 1 ∈ R d × n and Z e
2 ∈ R d × m , respectively.
Here, the cardinality of Z e 1and Z e 2 is n and m , respectively.
First, matching costs and relevance weights are generated from
Z e 1 and Z e 2 .
The cost matrix C is defined based on cosine similarity so that a pair of features with asimilar expression has a smaller matching cost: C i,j = 1 −
[Equation]
∥ .
Here, ∥·∥ indicates L2-norm.
Then, relevance weights a i and b j are computed to capture the global representation inherent in each
ID (cf.
Sec. 4.3).
Using the relevance weights, the Sinkhorn-Knopp iteration computes the optimalflow matrix X ∗ (cf.
Appendix 1 ).
As a result, based on C and X ∗ , the ID shift of the target ID iscalculated as follows:
[Equation]
µ e 2 j is a weighted cosine distance with the optimal transportation plan X ∗ between the j -th targetsample and the reference samples as a weight.
ID shift vectors µ e 2 , · · · , µ e N of all target IDscomputed in the same way are used for feature normalization process of the next section.
4.2 Identity-Invariant Expression Learning
This section describes the process of generating and learning the ID-invariant representation ofthe target feature set Z e using the ID shift vector µ e .
First, inspired by several normalizationtechniques [52, 12] that perform domain-specific adaptation or generalization of features, we definethe ID-invariant representation by using µ e and σ e as shifting and scaling parameters, respectively:
[Equation]
where σ ej =
[Equation]
, j = 1 , · · · , m .
Since µ e and σ e captured from a specific ID are used,the normalization of Eq.
4 can remove ID-specific information from the model.
Next, the normalized d -dimensional feature set ˆ Z e is mapped onto the 2D VA space:
[Equation] [Equation]
where θ = (cid:8) γ ∈ R d × 2 , β ∈ R 2 (cid:9) .
Affine parameter θ is trained to adapt to the normalized featuresof target IDs in f θ of Eq. 5, which behaves similarly to adaptive instance normalization [21].
That is,a given model is encouraged to learn an ID-invariant representation [31, 45, 12].
Finally, the risk loss[43] between ˆ Y e and the VA label set Y e is calculated:
[Equation]
Note that Eq. 6, which regresses predictions and labels in terms of MSE, calculates empirical riskloss through Monte-Carlo sampling.
Require:
# IDs N , learning rate lr , initialize L main to zero, initialize parameters ( ϕ, θ ) to Normal 1: while not converge ( ϕ, θ ) do 2:
Z e ← ID grouping and feature encoding via f ϕ , ∀ e ∈ { e 1 , · · · , e N } 3: for i = 2 , · · · , N do 4: µ e i ← ID matching and ID shift vector calculation (Eq. 3) 5:
ˆ Z e i , ˆ Y e i ← Normalization and mapping into VA space (Eqs.
4 and 5) 6:
L main :=
L main + MSE( ˆ Y e i , Y e i ) ← Update the main loss (Eq. 6) 7: end for 8:
L total =
L main + L va ← Calculate the total loss 9: ( ϕ, θ ) :
= ( ϕ, θ ) − lr ∇ ( ϕ,θ )
L total ← Update learnable model parameters ( ϕ, θ ) 10: end while
Training procedure.
Algorithm 1 describes the details of the losses of ELIM calculated every step.
Reference feature set Z e 1 and target feature sets Z e 2 , ··· ,N quantify ID shift vectors µ e 2 , ··· ,N through
ID matching.
ID-invariant representations are generated from these vectors and Z e 2 , ··· ,N , and then theempirical risk loss L main is calculated in the VA space.
With L va [50] calculated by MSE betweenunnormalized feature set Z e and label Y e , parameters ( ϕ, θ ) are updated until convergence.
4.3 Relevance Weight Design
Since the matching flow X is determined based on relevance weights, i.e., a and b , the design of therelevance weights is important for effective ID matching.
This section describes the design processwith the aforementioned Z e 1 ∈ R d × n and Z e 2 ∈ R d × m as examples.
Sampling reliable features.
If a reference ID has a weak facial expression, the difference in facialexpression with other IDs will not be noticeable, which indicates the difficulty in ID matching.
In otherwords, the feature of such a reference ID, i.e., z e 1 is not suitable for matching.
Therefore, we set thenorm ∥ y e 1 ∥ of the label corresponding to z e 1 to the sampling score π of z e 1 : π = ∥
y e 1 ∥ / ( (cid:80) i ∥ y e 1 i ∥ ) ,i = 1 , · · · , n .
z e 1 s corresponding to top k elements in terms of π are sampled.
However, since thesampling of z e 1 is non-differentiable, we employ a re-parameterized gradient estimator with the
Gumbel-topk [26]:
[Equation]
[Equation]
Retrieving the k ( ≤ n ) largest values topk( g 1 , · · · , g n ) will give us k samples without replacement .
Finally, Z e 1 , Z e 2 ∈ R
d
× k s retrieved in the same way are used to calculate relevance weights.
Computing relevance weights.
If weights are generated only from features of either reference IDor target ID, it will be difficult to capture the global representation of facial expression.
Instead, ifaggregated information of features between IDs is used for weight calculation, it will become easierto find ID pairs that are advantageous for FER.
This can be proven experimentally (cf.
Table 4).
Inspired by the feature representation of a few-shot classification task [38], we use the average ofz e 2 s as support of z e 1 to calculate the reference ID’s relevance weight a :
[Equation]
where [ · ] + indicates the clipping of negative value(s).
Another relevance weight b of the target ID iscalculated similarly.
The effect of the proposed relevance weight will be analyzed in Sec. 5.4.
Experiments
5.1 Datasets and Implementation Details
Datasets.
We adopted public databases only for research purposes, and informed consent was obtainedif necessary.
AFEW-VA [28] derived from the AFEW dataset [10] consists of about 600 short videoclips annotated with VA labels frame by frame.
Evaluation was performed through cross validation ata ratio of 5:1.
Aff-wild [50] is the first large-scale in-the-wild VA dataset that collects the reactions ofpeople who watch movies or TV shows.
Aff-wild with 298 videos amounts to over 30 hours in length,whose number of frames is about 1.18M. Since the test data of Aff-wild was not disclosed, this paperadopted the sampled train set for evaluation purpose in the same way as previous works [17, 22].
Aff-wild2 , which added videos of spontaneous facial behaviors to Aff-wild, was also released [25].
This database consists of about 1.41M frames.
Since subjects of different ethnicity and age groupswere newly added, Aff-wild2 can further enhance the reliability of FER performance.
Configurations.
All models were implemented in PyTorch, and the following experiments wereperformed on Intel Xeon CPU and NVIDIA RTX A6000 GPU.
Each experiment was repeatedfive times.
The light-weighted AlexNet (AL-tuned) [29], ResNet18 (R18) [19], and Mlp-Mixer(MMx)
[40] were chosen as f ϕ , respectively.
The mini-batch size for those models was set to 512,256, and 128, respectively (cf.
Appendix 2 for model details).
Learnable parameters ( ϕ, θ ) wereoptimized with Adam optimizer [23].
The learning rate (LR) of ϕ and θ was set to 5e-5 and 1e-5,respectively.
LR decreases 0.8-fold at the initial 5K iterations and 0.8-fold every 20K iterations.
ϵof Eq.
4 is 1e-6.
At each iteration, the number of ID, i.e., N was set to 10 by default.
The size d ofz is 64, and the size k of Gumbel-based sampling in Sec.
4.3 is 10.
Note that the cardinality of Z eis always greater than or equal to k .
For face detection, deep face detector [51] was used, and thedetected facial regions were resized to 224 × 224 through random cropping (center cropping whenevaluating).
ID grouping is performed based on folders in the database without separate clusteringtools.
Inference is performed through backbone ( f ϕ ) and projector ( f θ ) without Eq.
3 and Eq.
Evaluation metrics.
Root mean-squared error (RMSE) and sign agreement (SAGR) were used tomeasure the point-wise difference and overall degree of expressions.
In addition, Pearson correlationcoefficient (PCC) and concordance correlation coefficient (CCC) were employed to measure theexpression tendency (cf.
Appendix 3 for metric details).
Since the objective of FER learning is tosimultaneously achieve the minimization of RMSE and the maximization of PCC/CCC [22, 27],
L PCC and L CCC are used together to learn L va , where L C ( [Figure] P
[Figure]
5.2 Idea Verification of ELIM
To explain the mechanism of ELIM more clearly, we present experimental answers to the most curiousquestions: Q1.
Is ELIM robust when there is a discrepancy between facial expression and its labelprediction?
Q2.
Is ELIM successful in learning identity (ID)-invariant features?
For a fair experiment,we compared the proposed method with CAF [22] of SOTA performance as well as the baseline composed of only CNN and FC layers [29].
All methods were trained with AL-tuned backbone onthe AFEW-VA dataset under the same experimental condition as ELIM .
A1.
Analysis of inconsistency cases.
The main goal of ELIM is to accomplish robust performanceeven in cases where expressions and corresponding label predictions are inconsistent.
To verify this,we implement an ID shift case (cf.
Sec. 3.1).
In detail, we examine how robust each method is to IDshifts for facial images with the same GT but different IDs.
Fig.
3 illustrates the prediction resultsin the VA space.
Compared with CAF and baseline, ELIM not only estimates GT precisely, but alsooutputs relatively consistent prediction results.
For more examples, refer to Appendix 4 .
A2.
Visualization of ID-invariant representation ability.
We visualized feature distributions of ELIM and CAF through t-SNE [42].
Fig.
4 shows the result of comparing the distance between ananchor expression sample and two different expression samples with the same ID.
Note that in thecase of ELIM , samples with different IDs but similar expressions are located closer in the featurespace (see 1 ⃝ - 2 ⃝ and 4 ⃝ - 5 ⃝ in Fig.
4 ( a ) ).
Meanwhile, in CAF, samples with the same ID tend to belocated close to each other (see 2 ⃝ - 3 ⃝ and 5 ⃝ - 6 ⃝ in Fig.
4 ( b ) ).
ELIM adaptively considers inter-IDvariations of facial expressions through the OT problem, but CAF seems to have an ID-dependentfeature distribution because it has no means to explicitly consider them.
For visualization results andqualitative experiments of other feature distributions, refer to Appendix 4 .
[Table]
5.3 Comparison with State-of-the-art Methods
Table 1 compares ELIM and several existing methods in terms of various evaluation metrics for the
Aff-wild dataset.
Note that ELIM shows better CCC performance while having a smaller parametersize than other methods including CAF.
For example, ELIM with AL-tuned has about 21 times smaller parameter size than Hasani et al.
[16] but improves CCC(V) by about 26%.
Furthermore, ELIMshows 8.8% higher CCC(V) than a prior art CAF [22].
In addition, an experiment was conductedusing the recently popular Transformer Mlp-Mixer [40] as a backbone.
Although the parameter sizeincreased, a noticeable performance improvement was observed.
ELIM with MMx showed about11% better PCC(A) than CAF, and its RMSE and CCC were also improved significantly.
Table 2 shows the spontaneous expression recognition performance of short video clips througha total of four evaluation metrics.
Note that ELIM , which achieved SOTA performance, and CAF,which took 2nd place, are both FER methods designed to be robust against changes in identity’sexpression.
As a result, this suggests that it is more meaningful to extract ID-invariant featuresthan to focus only on changes in the expression itself.
And, ELIM showed better performance than
CAF in terms of CCC metric, which is important for expression tendency analysis.
For example,
ELIM with R18 showed CCC higher than CAF with R18 by about 2% and 7% in terms of Valenceand Arousal, respectively.
Considering that the Arousal axis indicating the degree of activation hashigher prediction difficulty than the Valence axis indicating positive and negative emotions, such aperformance improvement is sufficiently significant.
In addition, even in the most difficult Aff-wild2,
ELIM showed better CCC (mean) than the latest FER method AP [37] (cf.
Table 3).
These resultssuggest that further performance improvement is expected if temporal information is applied to staticimage-based ID-invariant methods.
[Table] [Table]
[Table]
5.4 Ablation Studies and Discussions
Our ablation studies aim to answer: Q3.
Is relevance weight effective?
Q4.
Is the number of training
IDs the larger the better?
Q5.
What other factors can be analyzed for ID shift?
All experiments in thissection were performed under the same setting as AL-tuned of Table 2.
A3.
Effectiveness analysis on relevance weights.
Sinkhorn-Knopp [8] depends on the size andnumber of relevance weights.
Table 4 shows the effectiveness of this fast iteration tool in threesettings.
First, let’s examine the effect of the optimization way on the OT problem.
As a benchmarkgroup of Sinkhorn-Knopp, we chose QPTH [4] that is a representative optimization tool based on theinterior point method of factorized KKT matrix.
Looking at the 3rd and 5th rows of Table 4, Sinkhorn provides approximately 3.7 times faster operation speed than QPTH while showing almost equivalent
CCC performance.
This proves that Sinkhorn, which has been often used to solve transportationproblems of high-dimensional features, has the advantage of fast operation speed because it is mainlybased on iterative updates of vectors.
Next, note that the relevance weight according to Eq.
8 showsbetter performance improvement than the self-side (non-relevance) weight despite a slight increase incomputation.
Here, ’self-side’ indicates determining the weights with the L2-norm of the individualfeatures of each ID.
Looking at the 3rd and 4th rows of Table 4, the relevance weight provides asignificant CCC performance improvement of about 1%.
This suggests that using relevance weightsis valuable enough.
Finally, Gumbel-based sampling has a processing speed of about 8.7 times fasterthan non-sampling, while providing higher CCC (mean) by 0.5% (see 2nd and 4th rows).
Therefore,we can argue that when calculating ID shifts, it is very important to select samples showing cleardifferences in facial expressions as in ELIM .
A4.
Influence of the number of training IDs.
ELIM ,which performs matching of ID pairs through the
OT problem, assumes a sufficient number of IDs.
To analyze the effect of the number of training IDs,we trained ELIM with or without L va .
Fig.
5 showsthat the performance of the two models increasesat almost the same rate as the number of training
IDs increases.
This prove that a sufficient number oftraining IDs is required, and that ELIM with moretraining IDs has the potential to bring about a higherperformance improvement. A
Other factors that can influence FER learning
Since inter-ID variation can be sufficiently shiftedby change of age or gender [13, 11], age, gender,race, etc. can also be used as indicators to measureinter-ID variation.
As a fragmentary example, thereis a technical report that analyzes the tendency of differences in facial expressions by domain of theage group of people [35].
Thus, starting with our study, various emotional change factors that can beobtained from humans will be grafted into deep learning models for inter-ID variation analysis for
FER.
Conclusion and Others
To analyze the inter-identity variation of facial expressions, we newly propose an identity matchingstrategy based on optimization theory.
The proposed method, which finds and utilizes optimal IDpairs based on similarity useful for facial expression learning, succeeded in explicitly dealing withidentity shifts, that is, the main cause of performance gap in the test set.
The results of verifyingidentity shifts and visualizing them from t-SNE perspective support the assumption of this paper thatit is effective to analyze inter-identity variation by considering identity as a domain.
Limitations.
The proposed method interacts with multiple target IDs based on a single reference ID.
Thus, model learning may depend on selection of a reference ID.
In order to overcome this limitation,dynamic connection methods of ID pairs along with more advanced relevance weight calculationmethods should be studied in the future.
Potential negative impacts.
FER may face with various privacy concerns.
For example, someonecan estimate and monitor an individual’s emotional response in social media through this FER systemwithout permission.
Although only an authorized database with informed consent is used in thisresearch field, someone’s facial images are likely to be used for learning of the FER system withoutpermission.
In order to solve this problem, the privacy-aware face data processing and expressionlearning framework should be urgently studied and introduced.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions
and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.1 and Section 5.2.
If you ran experiments...
(a) Did you include (b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)?
[Yes] See Section 5.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] Instead of error bars, we use the average values of multiple
experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.1.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.1.
(b) Did you mention the license of the assets?
[No] The license is added to the camera-ready
version.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
See Python scripts in the supplemental material.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes] See Section 5.1.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See Section 6.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]