DNN R EPRESENTATIONS AS C ODEWORDS : M ANIPULATING S TATISTICAL P ROPERTIES VIA P ENALTY R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Performance of Deep Neural Network (DNN) heavily depends on the characteristics of hidden layer representations.
Unlike the codewords of channel coding, however, the representations of learning cannot be directly designed or controlled.
Therefore, we develop a family of penalty regularizers where each one aims to affect one of representation’s statistical properties such as sparsity, variance, or covariance.
The regularizers are extended to perform class-wise regularization, and the extension is found to provide an outstanding shaping capability.
A variety of statistical properties are investigated for 10 different regularization strategies including dropout and batch normalization, and several interesting ﬁndings are reported.
Using the family of regularizers, performance improvements are conﬁrmed for MNIST, CIFAR-100, and CIFAR-10 classiﬁcation problems.
But more importantly, our results suggest that understanding how to manipulate statistical properties of representations can be an important step toward understanding DNN.
I NTRODUCTION
With a Deep Neural Network (DNN), information contained in the input data x is transformed into multiple representations over multiple layers.
Performance of machine learning tasks is known to heavily depend on the choice of representations over p ( x ) , but p ( x ) is almost always unknown and the representations cannot be directly controlled to match an arbitrary design even if p ( x ) was known.
As of today, the best that can be done is to indirectly affect the representations by adding constraints, modifying cost function, and tuning learning process, etc.
In Shannon’s information theory, channel coding theory deals with the problem of reliably sending the maximum amount of information through a given channel p ( y | x ) (Cover & Thomas, 2012).
Because the channel is known and ﬁxed, coding becomes a design problem where one needs to design codebook, encoder, and decoder.
Usually, the channel is used n times in a sequence to send a codeword of length n (expressed as x n ).
A codebook is a collection of all codewords that can be chosen and sent to the channel, and each message w with information of interest is mapped into one of the codewords during the design phase.
Because channel coding is a design problem, the optimal solutions are well understood for some of the important applications such as Gaussian channel and Binary Symmetric Channel (BSC).
Gaussian channel is the most important continuous alphabet channel problem.
It assumes that the received signal y n is a noisy version of x n , where the noise is independent of x and additive with an i.i.d.
Gaussian distribution over the n symbols.
Surprisingly, when n → ∞ , the optimal codebook turns out to be a collection of codewords that are generated by randomly drawing numbers from a Gaussian distribution.
Then, a codeword’s n symbols form an i.i.d.
Gaussian distribution (for instance, see Chap.
9 of Cover & Thomas (2012)).
BSC is one of the most popular discrete alphabet channel problem, and x can take a value of 0 or 1.
The received signal y is a corrupted version of x where it is ﬂipped with a ﬁxed probability.
For BSC, Hamming code is a well known solution where redundant bits are included in x n to resist the corruption (for instance, see Chap.
7 of Cover & Thomas (2012)).
The correction capability is dependent on the minimum Hamming distance among all pairs of two different codewords.
It would be helpful if the elegant channel coding theories can be applied to the design and control of DNN representation, but unfortunately the representation problem is clearly different from the channel coding problem.
First of all, a learning problem does not have a ﬁxed and known channel p ( y n | x n ) .
Secondly, we can design and control the DNN model to use, but we do not have the luxury of explicitly designing codebook, and representations can be formed in anyway.
Nonetheless, we can attempt to gain insights and ideas from the established coding theory.
In this work, we ﬁrst recognize that the optimal solution to Gaussian channel problem has i.i.d.
Gaussian codewords.
Although it is unclear if forcing representations of a layer to have an i.i.d.
Gaussian property will be helpful, we experiment the idea by expanding known penalty regularization strategies to include L1, variance, and covariance.
L1 and covariance (Cogswell et al., 2016) have been studied before (individually), but with our best knowledge, variance of a unit (neuron) and using combination of them have not been considered in the literature.
Secondly, we recognize that only a single codeword is assigned to a message (label for learning problems) for a well designed codebook.
When this idea is applied to learning problems via penalty regularization, the penalty term needs to be applied per-class such that we can shape the codeword of each label.
Note that almost all of the existing penalty regularization strategies have been applied to all classes together.
Thirdly, we recognize that Gaussian codebook and Hamming codebook are fundamentally different.
A Gaussian codebook uses continuous alphabets in an uncorrelated manner over n symbols, but Hamming codebook uses only binary values (0 and 1).
With the difference, it is inevitable for Gaussian code to utilize long codewords (very large n ) and distinguish codewords in a stochastic manner, while Hamming code needs to utilize carefully designed vector-space structures (orthogonality, null space, etc.) using relatively short codewords.
Because we are often interested in a relatively small number of neurons for representations, we consider a regularization strategy where each label’s activation for a unit is ‘hardened’ (by cw-VR regularizer that is introduced later) such that the representation vector is closer to a binary codeword than an i.i.d.
Gaussian codeword.
1. R ELATED W ORKS Regularization
The classical regularizers apply L2 (Hoerl & Kennard, 1970) and L1 (Tibshirani, 1996) penalties to the weights of models, and they are widely used for DNN as well.
Wen et al.
(2016) extended
L1 regularizer by using group lasso to regularize the structures of DNN (i.e., ﬁlters, channels, ﬁlter shapes, and layer depth).
Regularization has been applied to representations , too.
Srivastava et al.
(2014) devised dropout that randomly applies activation masking over the neurons.
While dropout is applied in a multiplicative manner, Glorot et al.
(2011) used L1 penalty regularization on the activations to encourage sparse representations.
XCov proposed by Cheung et al.
(2014) minimizes the covariance between autoencoding units and label encoding units of the same layer such that representations can be disentangled.
DeCov, developed by Cogswell et al. (2016), is also a penalty regularizer and it minimizes the off-diagonals of a layer’s representation covariance matrix.
DeCov reduces co-adaptation of units by encouraging units to be decorrelated.
It is called CR (Covariance
Regularizer) in this study for the purpose of consistent naming.
Statistics over mini-batch samples or in-layer activations have been used for regularization, too.
Batch normalization proposed by Ioffe &
Szegedy (2015) exploits mini-batch statistics to normalize activations.
It was developed to accelerate training speed by preventing internal covariate shift, but it was also found to be a good regularizer.
In line with batch normalization, weight normalization, developed by Salimans & Kingma (2016), uses mini-batch statistics to normalize weight vectors.
Layer normalization proposed by Ba et al.
(2016) is a RNN version of batch normalization, where they compute the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case.
There are many other publications on DNN regularization techniques, but we still do not have a sufﬁcient understanding on how they really work.
A recent work by Zhang et al.
(2016) shows that the traditional concept of controlling generalization error by regularizing the effective capacity cannot be applied to DNN.
Class-wise Learning
True class information is available for supervised learning problems.
Traditionally, the class information has been used only for evaluating correctness of predictions and the relevant cost function terms.
Some of the recent works, however, have adopted the class-wise concept in the learning algorithm itself.
In those works, class information is used as a switch or for emphasizing the discriminative aspects over different classes.
As an example, Li et al.
(2008) proposed a kernel learning method using class-wise information to model the manifold structure.
They modify locality preserving projection to be class dependent.
Jiang et al.
(2011) added label consistent regularizers for learning a discriminative dictionary.
As for DNN, a recent work by Belharbi et al.
(2017) uses a regularization function that penalizes the pair-wise distance of two representations that belong to the same class.
This approach is similar to cw-VR (class-wise Variance Regularizer) in our work, but they use the sum of pair-wise distances while we use representation variance of the same class samples.
T HREE S TATISTICAL P ROPERTIES AND C LASS -W ISE R EGULARIZATION
For channel coding problems, we can characterize the statistical properties of optimal codewords as discussed in Section 1.
Our goal is to make DNN representation vectors to have such statistical properties and analyze their effects.
Because an explicit design and control of representation vector is not possible for the learning problems, instead we utilize penalty regularizers to manipulate the statistical properties.
2. T HREE S TATISTICAL P ROPERTIES
Three of the most basic statistical properties are considered in this work - sparsity, variance, and covariance.
Sparsity over layer l ’s representation vector h l has been extensively studied in literature.
For variance, we are referring to the variance of a unit’s activation values over mini-batch samples.
When the variance is forced to be very small, the activation value needs to be close to the sample mean for all labels, and therefore the unit loses its discriminative power over multiple labels.
While this is undesirable, regularizing variance turns out to be meaningful because the cross-entropy cost function prevents the variance becoming zero and a healthy compromise can be achieved between cross-entropy and variance terms.
This is similar to the situation of classic weight regularization, where the weights actually never become zero by regularization.
For covariance, we calculate pairwise covariance over the unit activations of a layer.
When covariance is evaluated to be large for a pair of units (neurons) in the same layer, it indicates that the two are strongly correlated.
This is undesirable if we are pursuing i.i.d.
property over unit activations, and having a regularizer to control the level of correlation can be useful.
2. C LASS - WISE REGULARIZATION
To pursue statistical properties for each class, we adopt the concept of class-wise learning 1 .
For instance, it is undesirable if the variance becomes exactly zero for a unit’s activation as mentioned above.
Variance of zero for a class, however, can be desirable because it simply states that a consistent activation value will be observed over all samples with the same class label.
Note that overall variance over all labels can be still large while class-wise variance is zero - as long as inter-class difference exists, the overall variance will not be zero.
We combine this concept of class-wise regularization to the three concepts of sparsity, variance, and covariance.
Analytical formulations can be found in the following section.
P ENALTY L OSS F UNCTIONS
In this section, we provide the model for calculating basic statistics and formulate the penalty loss functions that are used for regularization.
3. B ASIC STATISTICS
For layer l , the output activation vector of a linear ﬁlter followed by ReLU is deﬁned as h l = max( W (cid:62) l h l − 1 + b l , 0) .
Because we will be focusing on layer l for most of the explanations, we drop the layer index and h is used to indicate h l instead.
Then, h i is the i th element of h (i.e. activation of i th unit), and w ki is the ( k, i ) element of W .
To use statistical properties of representations, we deﬁne mean of unit i , µ i , and covariance between unit i and unit j , c i,j , using the N samples in each mini-batch.
[Equation]
Here, h i,n is the activation of unit i for n th sample in the mini-batch.
From equation (2), variance of i unit can be written as below.
[Equation]
When class-wise statistics need to be considered, we choose a single label m and evaluate mean, covariance, and variance using only the data samples with label m in the mini-batch.
[Equation]
Here, S m is the set containing indexes of the samples whose label is m , and | S m | is the cardinality of the set S m .
3. P ENALTY LOSS FUNCTIONS
Using the notations in Section 3.1, the loss functions and their derivatives can be derived and summarized as in Table 1.
L1-weight and L2-weight are well-known, and they apply L1 and L2 penalties on the weights, respectively.
The rest in the table apply penalty on the representation.
L1-rep is similar to L1-weight, but the penalty is applied to the representation h .
Obviously, L2 can also be applied to representation, but it is excluded in this study because it tends to perform worse than L1 when applied to representation.
VR (Variance Regularization) calculates variance of each unit’s activation over mini-batch dataset and uses the calculated value as the penalty.
CR (Cross-covariance Regularization) uses off-diagonal terms of the mini-batch covariance matrix of activations as the penalty term.
As mentioned earlier, CR in this work is the same as DeCov presented by Cogswell et al.
but we use the term CR for the consistency of naming.
As in DeCov, we subtract variance terms and consider cross-covariance terms only (see penalty loss function in Table 1).
cw-VR and cw-CR are similar to VR and CR, respectively, except that the values are calculated for each class using the mini-batch samples with the same class label.
cw-L1-rep can be deﬁned, but its penalty loss function turns out to be the same as L1-rep’s loss function.
Therefore, cw-L1-rep is excluded in this study.
Interpretation of derivatives
While the penalty functions were chosen from the three distinct statistical properties and class-wise concept, their derivatives show that some of them are closely related.
For the derivatives of VR and
CR, it can be observed that they have similar structures.
If VR’s derivative ∂ Ω V R
∂h
i,n becomes zero for all i , then CR’s derivative ∂ Ω CR
∂h
i,n becomes zero as well.
The vice versa does not hold, but the effects of VR and CR can be expected to be similar or at least related for the learning process.
In the same way, the relationship between cw-VR and cw-CR is the same as the relationship between VR and
CR.
Therefore, we can expect cw-VR and cw-CR to have similar effects, too.
On the other hand, the derivative of L1-rep has a distinct formulation and it can be expected to have a distinct effect on learning.
There is another important effect that is not necessarily obvious from the derivative formulations.
For L1-weight and L2-weight, the derivatives are dependent on the weights w ki only, and they are independent of the activations
h i,n .
Therefore, the weights need to become smaller to reduce
[Table]
= (cid:88) k (cid:88) i | w ki | ∂ Ω L 1 - weight ∂w ki = sign ( w ki )
[Table]
L
[Table]
weight = (cid:88) k (cid:88)
i w 2 ki ∂ Ω L 2 - weight ∂w
ki = 2 w ki Ω L 1 - rep = (cid:88) n (cid:88) i | h
i,n | ∂ Ω L 1 - rep ∂h i,n = sign ( h i,n ) Ω V R = (cid:88) i v i ∂
Ω V R ∂h
i,n =
N ( h i,n − µ i )
Ω CR = (cid:88) i (cid:88)
j ( c i,j ) 2 − (cid:88) i ( v i )
∂
Ω CR ∂h i,n =
N (cid:88) j (cid:54) =
i c i,j ( h j,n − µ j ) Ω cw - V R = (cid:88) m (cid:88)
i v mi ∂ Ω cw - V R
∂h
i,n = | S m | ( h i,n − µ mi ) , n ∈ S m Ω cw - CR = (cid:88) m ( (cid:88) i (cid:88) j
( c i,j ) 2 − (cid:88) i ( v i ) 2 )
∂
Ω cw - CR ∂h i,n = | S m | (cid:88)
j (cid:54) =
i c mi,j ( h j,n − µ mj ) , n ∈ S m the regularization penalty.
For the other ﬁve representation regularizers, their derivatives are all dependent on activation h
i,n .
So, a simple way to reduce the regularization penalties is to scale the activations to small values (instead of satisfying the balances among the terms in the equation to reach zero gradient and force the desired statistical properties).
This scaling will not have any effect on prediction output as long as all the elements of h l are scaled together to α h l - the last softmax layer works as a normalization function for the output layer, and therefore the cross-entropy penalty term is not affected by such a scaling.
This means that there is a chance for the learning algorithm to squash activations just so that representation regularization terms can be ignored.
As we will see later, indeed activation squashing happens by learning, but the desired statistical properties are still sufﬁciently enforced.
Nonetheless, it must be possible to design better penalty regularizers that are immune to activation squashing, and such regularizers might be much more effective for manipulating statistical properties of representations.
E XPERIMENTS - MNIST
In this section, we consider 10 regularization strategies and compare them using the MNIST dataset (LeCun et al., 1998).
We use a Multilayer Perceptron (MLP) with ﬁve hidden fully connected layers and an output layer.
Each hidden layer has 100 units with Rectiﬁed Linear Unit (ReLU) activation function, and the output layer consists of 10 softmax units.
All experiments (in this work) were carried out using TensorFlow 1.3.
4. P ERFORMANCE RESULTS
For each regularization term, the level of regularization was determined by tuning the penalty loss weight using a validation dataset and a grid search.
Then, we trained each model ﬁve-times and calculated the test error performance as the average and one standard deviation over the ﬁve performance results.
In Table 2 and Table 3, the results show that representation regularizers outperform the popular regularizers, and that the representation strategies perform better when applied to upper layers of DNN.
Interestingly, the best performance is achieved by applying representation regularization to the output layer as shown in Table 3.
This might be because the regularizer directly affects only the regularizing layer and the layers below, or because manipulating statistical properties is more effective for the higher layer representations that have stronger or codeword-like structures.
To better understand the effect of layer, multiple layer results are shown in Table 4.
The best performance is achieved when output layer is regularized together with one or two upper hidden layers.
Among all the results in the three tables, CR performs best and achieves 2.43% of error.
[Table] [Table]
[Table]
4. S TATISTICAL PROPERTIES OF 10 REGULARIZATION STRATEGIES
We use 9 metrics to compare the statistical properties of the 10 regularization strategies.
Among the 9 metrics, ﬁrst seven of them are calculated by directly evaluating the penalty loss functions shown in Table 1.
The raw evaluation values, however, are difﬁcult to interpret because they have different scales.
So, we normalize the metrics as following (see the raw evaluation values shown in Table 10 and Table 11).
First, square-root is applied to L2-weight, VR, and CR because their units are not quadratic, and square-root of square-root is applied to cw-VR and cw-CR because their units are quartic.
Then, all the metrics of each regularizer are divided by the regularizer’s own (cid:112) Ω L 2 - weight such that all are normalized with respect to its 2-norm weight values. Finally, all the metrics are normalized by baseline’s metrics and 100 is multiplied such that we can focus on the relative change in percentage compared to the baseline’s metrics.
The remaining two metrics are the average number of activated classes per unit as the measure of sparsity and ratio of dead units.
Ω L 1 - weight and Ω L 2 - weight are calculated from the weights of all layers excluding biases, and the others are calculated from activations of test dataset in Layer 5.
We can observe two distinct groups of regularizers by investigating Table 5 and Table 6.
We can observe that the representation regularizers have much smaller values for the representation metrics.
This is because the representation regularizers squash activations in the way described in Section 3.2.
As mentioned in Section 3.2, VR and cw-VR are related to CR and cw-CR, respectively.
We can see that their values of metrics are similar to each other.
Despite of this similarity of the ﬁve representation regularization, L1-rep and cw-VR have unique characteristics.
L1-rep obviously enforces sparsity and causes much more dead units than the others.
The regularizer cw-VR always shows the smallest metric values among four strategies (VR, CR, cw-VR, and cw-CR).
This can be an evidence of the four regualizers’ close relationship.
The metric values of dropout and batch normalization (BN) are located somewhere between baseline and representation regularizers.
They cause similar effects on representation metrics as the representation regularizers, but much less effects are observed.
It is also interesting to note that both dropout and BN have only 0 or 1% of dead units (neurons).
Dropout and BN are implicit methods in the sense that they do not target any
[Table]
[Table]
particular statistical property, but they certainly seem to have distinct effects compared to the other regularizers.
4. V ISUALIZATION OF REPRESENTATIONS
Due to activation squashing, metrics of statistical properties can become misleading.
Therefore, we visualize the representation of Layer 5 to more intuitively understand the statistical properties that are affected by the regularizers.
Samples for three regularizers are shown in Figure 1 and Figure 2, and all ﬁgures for the ten regularizers are shown in appendix (Figure 3 and Figure 4).
Histogram of a single unit
We ﬁrst visualize the distribution of activation per unit in Figure 1 to observe sparsity and variance properties.
Activation histograms were generated by using 10,000 test data, and each color corresponds to a different class.
Since activations are generated as the output of ReLU activation function, many have zero value that can distort the histogram.
We, therefore, excluded zeros from activations when drawing the histogram plots.
In Figure 1(a), it can be seen that baseline has a large class-wise variance and inter-class overlaps.
The histogram of cw-VR in (b), however, shows the effect of separating the classes because class-wise variance is signiﬁcantly reduced.
For each class, the activation is ‘hardened’.
L1-rep in (c) can be conﬁrmed to have only one class that is activated, and this conﬁrms the sparsity.
As described in Table 6, Avg Act Class of L1-rep is close to zero, so most of its histograms show very few active samples.
Scatter plot of a unit pair
To show the relationship between a pair of two representation units, we drew a scatter plot of activations of two units.
As shown in Figure 2, baseline shows slight linearity, which is consistent with the high covariance value.
Since CR in (b) reduces cross-covariance per unit, it can be seen that overall linearity is reduced compared to the baseline.
In the same way, cw-CR has reduced class-wise cross-covariance.
Furthermore, its intra-class variance is small and thus end up having small ball-shaped concentrations of points. [Figure]
E XPERIMENTS - C I FAR-10/
While performance improvement is not the main focus of this work, we provide additional test results with performance evaluations and show that the representation regularizers are useful for pushing the accuracy performance to the next level.
In particular, we provide additional test results for CIFAR-100 and CIFAR-10 datasets (Krizhevsky & Hinton, 2009).
For CIFAR-100, we have chosen a toy CNN architecture to conﬁrm performance improvement of representation regularizers.
Concurrently using two of the regularizers is experimented as well.
For CIFAR-10, we have tested representation regularizers using Residual network (ResNet) that is known as one of the best performing deep neural networks for image data.
5. C OMBINING MULTIPLE STRATEGIES : CIFAR-
We use a toy CNN network for experimenting with CIFAR-100.
The CNN network consists of four convolution layers and a fully connected layer, all with 100 hidden units.
ReLU is used as the activation function.
The second, third, and fourth convolution layers are followed by a max pooling [Table] layer.
The last 10,000 instances of 50,000 training data were used as the validation data.
Using the validation data, validation performance was evaluated for the regularizer weight values of { 0.1, 0.01, 0.001, 0.0001 } .
The best weight value was found for each regularizer [Table] and the test performance was evaluated for the ﬁxed weight values.
For representation regularizers, regularization was applied to the fully connected layer.
The performance results are shown in Table 7.
From the table, it can be seen that the test error is improved from baseline 56.02% to 53.66% by using a single regularizer (VR) and to 53.30% by using two regularizers (CR and cw-VR).
Therefore, 2.72% of improvement is achieved by the best performing regularizer combination.
Aside from the performance improvement, it is interesting to observe that L1-rep consistently fails to train for the CIFAR-100 data.
Withlabels, too much sparsity might hurt the performance.
This is a plausible hypothesis considering that we have only 100 neurons to encode 100 labels.
A shared use of neurons over multiple classes might be a better direction to pursue.
In general, the relationship between the number of labels and the desired statistical properties of representation remains as a topic to be studied.
5. P ERFORMANCE IMPROVEMENT OF R ES N ET -
ResNet was ﬁrst proposed by He et al.
ResNet consists of multiple basic blocks that are serially connected, and shortcut connections to force residuals to be calculated.
We apply ﬁve regularization strategies without modifying the ResNet-32 architecture.
Regularization was applied to the output layer.
Experimental results in Table 8 show that performance is improved over the stateof-the-art ResNet-32 model.
This indicates that representation regularizers are compatible with
ResNet, and most likely also with other state-of-the-art models.
C ONCLUSION
In this work, we have investigated ﬁve different penalty regularizers for manipulating statistical properties of DNN representations.
The regularizers were conceived by examining optimal codewords of well-known channel coding problems, and the three statistical properties of sparsity, variance, and covariance were integrated into the regularizers along with the concept of class-wise regularization.
It was found that many statistical properties including cross-covariance, co-adaptation, intra-class variance, average number of active class per-neuron, and ratio of dead neuron can be [Table] manipulated.
Each regularizer, however, tended to manipulate multiple properties at the same time, making it difﬁcult to manipulate each property individually.
While manipulation was shown to be possible and helpful for improving performance of all three DNN classiﬁcation problems that were investigated, it is still unclear if any statistical property of representation is generally helpful when strengthened.
Due to the complicated nature of learning process where back-propagation affects not only the signal of interest but also other signals and irrelevant noise, it still remains as an open question on how to establish procedures that generally improves learning of any deep learning problems.
A CKNOWLEDGMENTS
To be added.
A PPENDIX A P ERFORMANCE OF POPULAR REGULARIZERS WHEN APPLIED TO EACH LAYER
[Table]
B E VALUATION OF STATISTICAL PROPERTIES
[Table]
[Table]
C M ETRICS Average number of activated classes
The number of activated classes in each unit is related to the class-wise sparsity.
ReLU is activated when the input of ReLU is positive.
Whether each unit is activated in a class is deﬁned by whether the ratio of activated ReLU exceeds the fuxed threshold.
The number of activated classes means the number of classes that each unit is activated.
The average of activation class is deﬁned by the average of the number of activated classes for all units.
The average of the activation class represents classwise sparsity by measuring how many classes are activated for each unit.
I is an indicator function and N u is the number of units in the layer.
[Equation]
Ratio of dead units
’Dead neuron’ is widely used to represent neurons that are not activated for ReLU.
However, the concept of dead neurons does not represent class-wise information.
So, we deﬁne the dead unit which is not activated on all classes.
The dead unit indicates that it is not signiﬁcant to any class.
The ratio of dead units measures the ratio of dead units over all units in the representation layer.
A higher ratio of dead units indicate that many units are not carrying any useful information.
[Equation]
D V ISUALIZATION OF REPRESENTATIONS D. R EPRESENTATION HISTOGRAMS OF 10 REGULARIZERS
[Figure]
D. R EPRESENTATION SCATTER PLOTS OF 10 REGULARIZATION STRATEGIES
⠀愀⤀ 䈀愀猀攀氀椀渀攀
⠀戀⤀ 䰀㄀ⴀ爀攀瀀 ⠀挀⤀ 䰀㄀ⴀ眀攀椀最栀琀 ⠀
搀⤀ 嘀刀
⠀攀⤀ 䰀㈀ⴀ眀攀椀最栀琀 ⠀昀⤀ 䌀刀 ⠀最⤀
䐀爀漀瀀漀甀琀 ⠀栀⤀ 挀眀ⴀ嘀刀 ⠀椀⤀
䈀一 ⠀樀⤀ 挀眀ⴀ䌀刀