F AIRNESS IN R EPRESENTATION FOR M ULTILINGUAL NLP: I NSIGHTS FROM C ONTROLLED E XPERIMENTS ON C ONDITIONAL L ANGUAGE M ODELING
A BSTRACT
We perform systematically and fairly controlled experiments with the 6-layer Transformer to investigate the hardness in conditional-language-modeling languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH).
We evaluate through statistical comparisons across 30 possible language directions from the 6 languages of the United Nations Parallel Corpus across 5 data sizes on 3 representation levels — character, byte, and word.
Results show that performance is relative to the representational granularity of each of the languages, not to the language as a whole.
On the character and byte levels, we are able to eliminate statistically significant performance disparity, hence demonstrating that a language cannot be intrinsically hard.
The disparity that mirrors the morphological complexity hierarchy is shown to be a byproduct of word segmentation.
Evidence from data statistics, along with the fact that word segmentation is qualitatively indeterminate, renders a decades-long debate on morphological complexity (unless it is being intentionally modeled in a word-based, meaning-driven context) irrelevant in the context of computing.
The intent of our work is to help effect more objectivity and adequacy in evaluation as well as fairness and inclusivity in experimental setup in the area of language and computing so to uphold diversity in Machine
Learning and Artificial Intelligence research.
Multilinguality is real and relevant in computing not due to canonical, structural linguistic concepts such as morphology or “words” in our minds, but rather standards related to internationalization and localization, such as character encoding — something which has thus far been sorely overlooked in our discourse and curricula.
I NTRODUCTION 1. B ACKGROUND AND MOTIVATION
Most current work on fairness in Machine Learning (ML) and Natural Language Processing (NLP) focuses on the societal biases encoded in natural language data that are propagated and amplified when they are used at scale for/as Artificial Intelligence (AI) solutions 1 .
But little has been said or questioned about the bias, as in, the favoring of certain outcomes, implicit in our theoretical/scientific assumptions that results in the varying performance of different languages in computing.
Disparity in machine translation results For instance, results reported in Junczys-Dowmunt et al. (2016) for Phrase-Based Statistical Machine Translation (PBSMT) (Koehn et al., 2003) and
Neural MT (Bahdanau et al., 2014) on the 6 official languages 2 of the United Nations (UN) Parallel
Corpus (Ziemski et al., 2016) indicate a disparity between EN/ES/FR and AR/RU/ZH in BLEU (Papineni et al., 2002) — translation performance in the latter group is generally worse, regardless of the MT algorithm used.
AR and RU are traditionally considered morphologically complex (see e.g. Minkov et al.
(2007), Seddah et al.
(2010) and proceedings of related workshops in subsequent years for Statistical Parsing of Morphologically Rich Languages ), and ZH morphologically frugal (for its lacking determiners and plural or tense markers) (Koehn, 2005).
While Koehn (2005) found translating into EN to be easier than into morphologically rich languages based on word-level BLEU scores from PBSMT systems of 110 language directions from the 11 Europarl languages then,
Bugliarello et al.
(2020) found it is easier to translate out of EN than into it based on 21 Europarl languages in BPEs (Byte Pair Encodings)
(Sennrich et al., 2016) with the Transformer (Vaswani et al., 2017) in a new metric, cross-mutual information.
Disparity in language modeling results Disparate performances across different languages seem to have been implicitly accepted in that it is often believed that some languages are harder to model than others.
Bender (2009) advocated the relevance of linguistic typology for the design of languageuniversal NLP systems due to differences based on crosslinguistic structural notions, such as parts of speech and morphological complexity.
Cotterell et al.
(2018) studied (monolingual) language models (LMs) on the 21 Europarl languages using a word-level 7-gram standard Kneser & Ney (1995) model and LSTM-LMs (Sundermeyer et al., 2012) with characters and lemmatized forms in information-theoretic terms, and found morphological complexity to be the primary culprit for the differences in performance.
Mielke et al.
(2019) extended the coverage to 69 languages with the multilingual Bible corpus (Mayer & Cysouw, 2014), tested on RNN-LMs (an implementation of
LSTM (Hochreiter & Schmidhuber, 1997)) with characters and BPEs, but concluded that basic data statistics in vocabulary size ( | V | ) and sequence length were the most predictive performance features.
We noticed, however, a discrepancy in the results from Mielke et al.
(2019) for ZH — it came out as the least difficult for the character model, but it is the 6 th most difficult language for the BPE model.
As different input representations have been tested with different architectures with divergent results in different metrics in previous studies, each of them only testing with one data size, we decided to investigate the matter more systematically once again with statistical comparisons of score distributions between languages.
1. R ESEARCH QUESTIONS AND CONTRIBUTIONS
Research questions Are there any statistically significant differences in hardness when it comes to Conditional-Language-Modeling (CLMing) languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH) with the 6-layer Transformer?
Is morphological complexity inherent in language?
When is the notion of morphological complexity relevant in computing?
Summary of findings and insights Based on our bilingual CLMing setup with the UN Parallel
Corpus in the data size range from 10 2 to 10 6 lines on the character, byte, and word levels, we find:
Language has many finer-grained dimensions with different representations and learning patterns.
Hardness in modeling is relative to its representational granularity ( representation relativity ).
There is neutralization of source language instances, i.e. there are no statistically significant differences between source language pairs.
Only pairs of target languages differ significantly.
On the character and byte levels, hardness is correlated with statistical properties concerning sequence length and | V | of a language, regardless of its morphological profile.
As it is possible to eliminate performance disparity by decomposing sequences into finer-grained units in characters and bytes, we show that morphological complexity is not an intrinsic property of language.
Unless word-based methods are used
or unless we implement/model it explicitly
the notion of morphological complexity is irrelevant in computing.
On the word level, hardness is correlated with | V | , and a complexity hierarchy arises through the manual preprocessing step of word tokenization.
This complexity/disparity effected by word segmentation can be improved by subword tokenization but cannot be eliminated due to the fundamental qualitative differences in the definition of a “word” being one that neither holds universally nor is suitable/consistent for fair crosslinguistic comparisons.
Representational units of finer granularity can help close the gap in performance disparity.
Orthogonal to our main research questions, we also observed 2 types of sample-wise non-monotonicity — Double Descent (Belkin et al., 2019; Nakkiran et al., 2020) and erraticity .
For reasons due to length and scope for this paper, we will defer discussions and analyses of these beyond what is addressed in § 5 to future work.
Outline of the paper In § 2, we define our method and experimental setup.
We present our results and analysis on the primary representations in § 3 and those from the secondary set of controls in §in a progressive manner to ease understanding.
Meta analysis on performance disparity and other discussions are in § 5.
M ETHOD AND DEFINITIONS
Conditional language modeling CLMing is the modeling of the probability of the next token, given the history of the preceding tokens and conditioning context.
In our case, such conditioning context is a line from the source language.
To explicitly focus on modeling the complexities that may or may not be intrinsic to the languages, we study the more fundamental process of CLMing without performing any translation.
This allows us to eliminate confounds associated with generation and other evaluation metrics.
One could think of our setup as estimating conditional probabilities with the Transformer, with a bilingual (one-to-one) setup where the perplexity (PP) of one target language ( l trg ) is estimated given the parallel data in one source language ( l src ), where l src ̸ = l trg .
We focus on the very basics and examine the first step in our pipeline — input representations, holding everything else constant.
Instead of measuring absolute cross-entropy scores at one data size, we evaluate the relative differences between development (dev) set score distributions between languages.
Controlled experiments as basic research for scientific understanding of language data Using the UN Parallel Corpus, the data from which the MT results in Junczys-Dowmunt et al.
(2016) stem, we perform a series of controlled experiments with the Transformer, holding the hyperparameter settings for all 30 one-to-one language directions from the 6 languages constant.
We control for size (from 10 2 to 10 6 lines) and language with respect to representational granularity.
We examineprimary representation types/levels — character, byte (UTF-8), and word, and upon encountering some unusual phenomena, we perform a secondary set of controls with 5 alternate representations — on the character level: Pinyin and Wubi (ASCII representations for ZH phones and character strokes, respectively), on the byte level: code page 1256 (for AR) and code page 1251 (for RU), and on the word level: BPE.
These symbolic variants allow us to manipulate the statistical properties of the representations, while staying as “faithful” to the language as possible.
We adopt this symbolic datacentric 3 approach because we would like to more directly interpret the confounds, if any, that make language data different from other data types.
We operate on a smaller data size range as this is more common in traditional language sciences and one of our higher goals is to bridge an understanding between language sciences and engineering (the latter being the dominant focus in NLP), and between traditional symbolic sciences and ML.
We run statistical tests to identify the strongest correlates of performance and to assess whether the differences between the mean performance of different groups are indeed significant.
We are concerned not with the absolute scores, but with the differences between score distributions from different languages.
Fair evaluation with multitexts Multitexts are multiway parallel corpora.
The UN Parallel Corpus is a 6-way parallel corpus consisting of manually translated UN documents from the 25-year period between 1990 and 2014.
We use the UN Parallel Corpus because it contains languages conventionally regarded as morphologically rich and poor, has quality and size sufficient for evaluation, and more importantly, it comes as raw texts (untokenized), unlike both of the corpora that Mielke et al.
(2019) used.
Detokenization (esp.
the evaluation thereof) is not a trivial task.
Fair information-theoretic evaluation metric Most sequence-to-sequence models are optimized using a cross-entropy loss, defined as:
[Equation]
where t is the sequence of tokens to be predicted, t i refers to the i th token in that sequence, s is the sequence of tokens conditioned on, and N = | t | .
It is customary to report scores as PP, which is 2 1 N H ( t , s ) , i.e. 2 to the power of the cross-entropy averaged by the number of tokens in the dev algorithm. data.
Cotterell et al.
(2018) proposed to use “renormalized” PP to evaluate LMs tokenwise fairly by dividing the overall bits per utterance/sequence by one constant token count in any one arbitrary language (e.g. so to arrive at “bits per character” in one language to evaluate all languages).
But we find that it is not necessary to assign a perspective that is centered on any one particular language, when we can evaluate simply by the total number of bits for a larger portion of texts/sequences.
This can be a fairer, more general and flexible way of evaluating data that has not been or cannot be perfectly segmented or aligned line by line.
We hence used instead unnormalized PP, i.e. the total number of bits needed to encode the dev set (3,077 lines per language, after length filtering, in our case).
As the implementation we used only reports PP, we transformed it back to entropy as defined above via H ( t , s )
= log 2 PP ( t | s ) × N .
Disparity/Inequality
In the context of our CLMing experiments, we consider there to be “disparity” or “inequality” between languages l 1 and l 2 if there are significant differences between the performance distributions of these two languages with respect to each representation.
Here, by performance we mean the number of bits required to encode the held-out data using a trained CLM.
With 30 directions, there are 15 pairs of source languages ( l src 1 , l src 2 ) and 15 pairs of target languages ( l trg 1 , l trg 2 ) possible.
We compare the source languages among each other, and the target languages among each other.
Each l src or each l trg consists of scores from all models trained across various sizes and directions.
To assess whether the differences are significant, we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different.
Upon testing for normality with the Shapiro-Wilk test (Shapiro & Wilk, 1965; Royston, 1995), we use the parametric unpaired two-sample Welch’s t-test (Welch, 1947) (when normal) or the non-parametric unpaired Wilcoxon test (Wilcoxon, 1945) (when not normal) for the comparisons.
We use the implementation in R (R Core Team, 2014) for these 3 tests.
To account for the multiple comparisons we are performing, we correct all p-values using Bonferroni correction (Benjamini & Heller, 2008; Dror et al., 2017) and follow Holm’s procedure 4 (Holm, 1979; Dror et al., 2017) to identify the pairs of l 1 and l 2 with significant differences after correction.
We report all 3 levels of significance ( α ≤ 0 .
for a more comprehensive overview.
In contrast to Dror et al.
(2017), which aimed to compare the performance of different algorithms, we compare languages (in the context of computing).
Experimental setup The systematic, identical treatment we give to our data is described as follows with further preprocessing and hyperparameter details in Appendices A and B, respectively.
After filtering length to 300 characters maximum per line in parallel for the 6 languages, we madesubsets of the data with 1 million lines each — one having lines in the order of the original corpus (dataset A) and two other randomly sampled (without replacement) from the full corpus (datasets
B & C).
Lines in all datasets are extracted in parallel and remain fully aligned for the 6 languages.
For each run and each representation, there are 30 pairwise directions (i.e. one l src to one l trg ) that result from the 6 languages.
We trained all 150 (for 5 sizes) 6-layer Transformer models for each run using the SOCKEYE Toolkit (Hieber et al., 2018).
We optimize using PP and use early stopping if no
PP improvement occurs after 3 checkpoints up to 50 epochs maximum, taking the best checkpoint.
Characters and bytes are supposed to mitigate the out-of-vocabulary (OOV) problem on the word level.
In order to assess the effect of modeling with finer granularity more precisely, all vocabulary items appearing once in the train set are accounted for (i.e. full vocabulary on train, as in Gerz et al.
(2018a;b)).
But we allow our system to categorize all unknown items in the dev set to be unknown (UNK) so to measure OOVs (open vocabulary on dev (Jurafsky & Martin, 2009)).
To identify correlates of performance, we compute Spearman’s correlation (Spearman, 1904) with some basic statistical properties of the data (e.g. length, | V | , type-token-ratio, OOV rate) as metrics — a complete list is provided in App.
C. See App.
D for sample construction for statistical comparisons.
E XPERIMENTAL RESULTS OF PRIMARY REPRESENTATIONS
Subfigures 1a, 1b, and 1c show the mean results across 12 runs of the 3 primary representations — character, byte, and word, respectively.
The x-axis represents data size in number of lines and the y-axis the total conditional cross-entropy, measured in bits (Eq. 1).
Each line connects 5 data points corresponding to the number of bits the CLMs (trained with training data of 10 2 , 10 3 , 10 4 ,
1e+ 1e+05 number of lines nu m be r o f b i t s
TRG
ARENESFRRUZH
SRC
ARENESFRRUZH (a) CHAR     1e+ 1e+05 number of lines nu m be r
o f b
i t s
TRG
ARENESFRRUZH
SRC
ARENESFRRUZH (
BYTE     1e+ 1e+05 number of lines nu m be r o f b i t s
TRG
ARENESFRRUZH
SRC
ARENESFRRUZH (c) WORD AR
EN
ES
FR
RU
ZH    1e+03 1e+05 1e+03 1e+05 1e+03 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH
AR
EN
ES
FR
RU
ZH        1e+03 1e+05 1e+03 1e+05 1e+03 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH (
AR
EN
ES
FR
RU
ZH      1e+03 1e+05 1e+03 1e+05 1e+03 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH (f
WORD by target (g) (
[Figure]
10 5 , and 10 6 lines) needed to encode the target language dev set given the corresponding text in the source language.
These are the same data in the same 30 language directions and 5 sizes with the same training regime, just preprocessed/segmented differently.
This confirms representation relativity — hardness in modeling is relative to its representational granularity.
Languages (or any objects being modeled) need to be evaluated relative to their representation.
“One size does not fit all” (Durrani et al., 2019).
Our conventional way of referring to “language” (as a socio-cultural product or with traditional word-based approaches or even for most multilingual tasks and competitions) is too coarse-grained for computing (see also Fisch et al.
and Ponti et al.
Subfigures 1d, 1e, and 1f display the corresponding information sorted into facets by target language, source languages represented as line types.
Through these we see more clearly that results can be grouped rather neatly by target language — as implicit in the Transformer’s architecture, the decoder is unaware of the source language in the encoder.
As shown in Table 1 in § 5 summarizing the number of source and target language pairs with significant differences, there are no significant differences across any source language pairs .
The Transformer neutralizes source language instances.
This could explain why transfer learning or multilingual/zero-shot translation (Johnson et al., 2017) is possible at all on a conceptual level.
In general, for character and byte models, most language directions do seem to converge at 10 4 lines to similar values across all target languages, with few notable exceptions.
There are some fluctuations past 10 4 , indicating further tuning of hyperparameters would be beneficial due to our present setting possibly working most favorably at 10 4 .
On the character level, target language ZH (ZH trg ) shows a different learning pattern throughout.
And on the byte level, AR trg and RU trg display highly unstable behavior, which we refer to as erratic .
Word models exhibit Double Descent across the board (note the spike at 10 3 ), but overall, difficult/easy languages stay consistent, with AR and RU being the hardest, followed by ES and FR, then EN and ZH.
A practical takeaway from this set of experiments: in order to obtain more robust training results, use bytes for ZH (as suggested in Li et al. (2019a)) and characters for AR and RU (e.g. Lee et al.
— also if one wanted to avoid any “class” problems in performance disparity with words.
Performance disparity for these representations is reported in
Table 1 under “CHAR”, “BYTE”, and “WORD”.
Do note, however, that the intrinsic performance of ZH with word segmentation is not particularly subpar.
But this often does not correlate with its poorer downstream tasks results (recall results from Junczys-Dowmunt et al.
Since the notion of word in ZH is highly contested and ambiguous — i) it is often aimed to align with that in other languages so to accommodate academic theories and manual feature engineering 5 , ii) there is great variation among different conventions, and iii) native ZH speakers identify characters as words — there are reasons to rethink this procedure now that fairer and language-independent processing in finer granularity is possible.
Li et al.
(2019b) questioned the necessity of CWS in Deep Learning (DL)-based ZH NLP and presented evidence in favor of character-based processing, including results from downstream NLP tasks.
In Linguistics, Duanmu (2017) presented a summary on the contested nature of wordhood in (Mandarin) ZH in relation to EN.
A more native account of ZH, however, despite a couple of dialects/varieties of it being considered a high-resource language, has not yet been fully recognized and accepted in NLP.
U NDERSTANDING THE PHENOMENA WITH ALTERNATE REPRESENTATIONS
To understand why some languages show different results than others, we carried out a secondary set of controlled experiments with representations targeting the problematic statistical properties of the corresponding target languages.
1e+ 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH (a) Wubi
AR
EN
ES
FR
RU
ZH [Figure] nu m be r o f b i t s
TRG [Figure] SRC
[Figure] [Figure] [Figure] 1e+ 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH ( [Figure] ) Pinyin
AR
EN
ES
FR
RU
ZH
[Figure]
1e+03 1e+05 1e+03 1e+
[Figure] 1e+ [Figure] 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH
[Figure]
Character level We reduced the high | V | in ZH with representations in ASCII characters — Pinyin and Wubi.
We replaced the ZH data in these formats only on the target side and reran the experiments involving ZH trg on the character level.
Results in Figure 2 and Table 1 show that the elimination of
) has been a decades-long issue in text processing.
But even in EN, for computing, the variability in “word” counts (from the trivial convention of whitespace tokenization) results in different bit counts, affecting file sizes.
In NLP, such method of “word” counting brings about a high | V | , hence different tokenization schemes have been designed to mitigate this problem.
For humans, there is no consensus about the definition of “words”.
Even for a purely academic account, it is held to be indeterminate (see Haspelmath (2011) and references therein from the past century).
Kilgarriff (1997; 2014) pointed out that “words” and “word senses” and the number thereof, in terms of lexical entries for dictionaries, are contextual and arbitrary.
1e+ 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH (
[Figure] AR EN ES
FR
RU
ZH      1e+03 1e+05 1e+03 1e+05 1e+03 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH
[Figure]
1e+ 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH (c) BPE
AR
EN
ES
FR
RU
ZH      1e+03 1e+05 1e+03 1e+05 1e+03 1e+05 number of lines nu m be r o f b i t s
TRG ARENESFRRUZH
SRC ARENESFRRUZH (
[Figure]
disparity on the character level is possible if ZH is represented through Pinyin (transliteration), as in
Subfigure 2c, though at the cost of the native script information.
Models represented through Wubi, an input algorithm that decomposes character-internal information into stroke shape and ordering and matches these to 5 classes of radicals (Lunde, 2008), display a behavioral tendency unlike those with other (phonetic) alphabetic scripts 6 (Subfigure 2a), suggesting that this script/stroke pattern decomposes differently. But ZH the language is not an outlier all around.
Byte level Length is the most salient statistical attribute that makes AR and RU outliers.
To shorten their sequence lengths, we tested with alternate encodings on AR trg and RU trg — code pageand 1251, which provide 1-byte encodings specific to AR and RU, respectively.
Results are shown in
Subfigures 3a and 3b.
Not only is erraticity resolved, the number of 15 possible target language pairs with significant differences reduces from 8 with the UTF-8 byte representation to
(Table 1 under “ARRU t ”), indicating that we eliminated disparity with this optimization heuristic.
Since our heuristic is a lossless and reversible transform, it shows that a complexity that is intrinsic and necessary in language 7 does not exist in computing, however diverse they may be, as our 6 are, from the conventional linguistic typological, phylogenetic, historical, or geographical perspectives.
Word level The main difference between word and character/byte models is length not being a top contributing factor correlating with performance, but instead | V | is.
This is understandable as word segmentation neutralizes sequence lengths.
To remedy the OOV problem, we use BPE, which learns a fixed vocabulary of variable-length character sequences (on word level, as it presupposes word segmentation) from the training data.
It is more fine-grained than word segmentation and is known to better model subword units for morphologically complex languages.
We use the same vocabulary of 30,000 as specified
This reduced our averaged OOV token rate by 89-100% across the 5 sizes.
The number of language pairs with significant differences reduced tofrom 8 for word models.
While BPEs are still not as effective as our character/byte variants, their results show how finer-grained modeling contributes positively to closing the disparity gap .
M ETA RESULTS , ANALYSES , AND DISCUSSION
Performance disparity Table 1 lists the number of language pairs with significant differences under the representations studied.
Since it is possible for our character and byte models to effect no performance disparity for the same languages on the same data, a complexity intrinsic to language does not exist.
In fact, the customary expectation that languages ought to perform differently is created through our word segmentation practice.
Furthermore, the order of AR/RU > ES/FR > EN/ZH (Figure 1c) resembles the idea of morphological complexity. Considering there are character-internal
[Table] [Table]
meaningful units in languages with logographic script such as ZH (cf.
Zhang & Komachi (2018)) that are rarely captured, studied, or referred to as “morphemes”, this goes to show that linguistic morphology, along with its complexity, as it is practiced today 8 and that which has occurred in the
NLP discourse thus far, has only been relevant on the “word” level, conceptually constrained by unstandardizable units such as “words” (and “sentences”).
The definition of word, however, has been recognized as problematic for a very long time in the language sciences (cf.
Footnote 5).
While the lack of significant differences between pairs of source languages would signify neutralization of source language instances, it does not mean that source languages have no effect on the target.
For our byte solutions with code pages, we experimented also with source side optimization in the directions that involve AR/RU as source.
This affected the distribution of the disparity results for that representation — with 2 pairs being significantly different (see Table 1 under “ARRU s,t ”).
We defer further investigation on the nature of source language neutralization to future work.
Target language pairs with significant differences are summarized in Table 2.
We show that morphological complexity can be empirically eliminated in this one-setting-for-all configuration with a 6-layer network, no hyperparameter tuning, and a maximum line length of 300 characters (and its corresponding equivalence in other representations) as constrained by our hardware and compute time listed in App.
A and current data availability.
A more analytical solution can be obtained through data statistics (see App. E).
A conceptual solution lies in the definition of “words” and morphology.
Sample-wise Double Descent (DD) Sample-wise non-monotonicity/DD (Nakkiran et al., 2020) denotes a degradation followed by an improvement in performance with increasing data size.
We notice word models and character models with ZH trg , i.e. models with high target | V | , are prone to exhibit a spike at 10 3 .
A common pattern for these is the ratio of target training token count to number of parameters falls into O( 10 − 4 ) for 10 2 lines, O( 10 − 3 ) at 10 3 , O( 10 − 2 ) at 10 4 , and O( 10 − 1 ) for 10 5 lines and so on.
But for more atomic units such as alphabetic (not logographic) characters (may it be Latin, Cyrillic, or Abjad) and for bytes, this progression instead begins at O( 10 − 3 ) at 10 2 lines.
Instead of considering this spike of 10 3 as irregular, we may instead want to think of this learning curve as shifted by 1 order of magnitude to the right for characters and bytes and/or the performance at 10 2 lines for words and ZH-characters due to being overparameterized and hence abnormal.
This would also fit in with the findings by Belkin et al.
(2019) and Nakkiran et al.
(2020) attributing DD to overparameterization.
While almost all work attribute DD to algorithmic reasons, findings from Chen et al.
(2020) corroborate our observation and confirms that DD arises due to “the interaction between the properties of the data and the inductive biases of learning algorithms”.
Other related work on the DD phenomenon and its development can also be found in their work.
Erraticity We observe another type of sample-wise non-monotonicity, one that signals irregular and unstable performance across data sizes and runs.
Within one run, erraticity can be observed directly as changes in direction on the y-axis.
Across runs, large variance can be observed, even with the same dataset.
Erraticity can also be observed indirectly through a negative correlation between data size and performance.
Much work on length bias in NMT have focused on solutions related to search, e.g. Murray & Chiang (2018).
Our experiments show that a kind of length bias can surface already with CLMing, without generation taking place.
Additional related work That basic data statistics are the driver of success in performance in multilingual modeling has so far only been explicitly argued for in Mielke et al.
We go beyond their work in monolingual LMs to study CLMs and evaluate also in relation to data size, representational granularity, and quantitative and qualitative fairness.
To the best of our knowledge, there has been no prior work on demonstrating the neutralization of source language instances through statistical comparisons, a numerical analysis on DD for sequence-to-sequence models, the meta phenomenon of a sample-wise non-monotonicity (erraticity) being related to length.
C ONCLUSION
Summary We investigate whether the performance disparity between languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH) in the 6-layer Transformer
CLM due to morphological complexity is justified and find that it is not.
Performance disparity can be explained by data statistics and in the context of computing, it can be eliminated by optimization on length and | V | through character/byte representations.
In fact, morphological complexity is not a necessary concept in computing because “word” is not a necessary concept in computing, unless we make it so through word segmentation.
A morphological complexity hierarchy can result simply through word segmentation.
Furthermore, there are many possible interpretations to “words” for humans and since morphology is defined with the concept of “word”, there is no stable ground for assessing this complexity.
Representational units of finer granularity were shown to help eliminate performance disparity though at the cost of longer sequence length, which can have a negative impact on robustness.
In addition, we found all word models and character models with ZH trg to behave similarly in their being prone to exhibit a peak (as sample-wise DD) around 10 3 lines in our setting.
Outlook ML has enabled greater diversity in NLP (Joshi et al., 2020).
Fairness, in the elimination of disparity, does not require big data.
This paper made a pioneering attempt to bridge research in
NNs/DL, language sciences, and language engineering through a data-centric perspective.
Multilinguality is real and relevant in computing not due to canonical, structural linguistic concepts such as morphology or “words” in our minds, but rather standards related to internationalization and localization, such as character encoding — something which has thus far been sorely overlooked in our discourse and curricula.
We also believe that a more fine-grained statistical data science can well complement algorithmic analyses with a view that is more empirically robust (i.e. experimentally verifiable) and more relevant to machine processing, contributing to a more generalizable and interpretable pool of knowledge for ML/NNs/DL.
A more comprehensive study can lead us not only to new scientific frontiers, but also to better designs and evaluation, benefitting the development of a more general, diverse and inclusive AI.
E THICS S TATEMENT : F AIRNESS CONCERNS FOR MULTILINGUALITY
Clearer nomenclature If/When the intent is not to explicitly model linguistic morphology in computing, one can simply describe languages and their statistical profiles with respect to their representational granularity in characters or bytes (which are and/or can be exhaustively standardized in computing), or refer to sequences as longer/shorter or having a higher/lower vocabulary size when comparing them with each other, rather than “richer”/“poorer” based on concepts (e.g. “words”, “sentences”) that can be ambiguous, contested, and inaccessible to many.
Accessibility Language communities who are unfamiliar with languages similar to dominant languages or those who are reluctant to conform to one structurally similar interpretation should not have to feel inadequate in processing their own language “from scratch”, if they so choose.
As technologists, we can help take equitable measures to make fairer data representations and infrastructure available.
A “word”-free view of language preempts linguistic/cultural hegemony and such an interpretation would also help make the analyses of language data more objective and clearer.
Scarcity of quality and/or multiway data for science, evaluation, and documentation With the rise of multilingual models comes an alleged decrease in reliance on parallel corpora for MT.
But data, esp.
high-fidelity/quality 9 textual and multimodal multiway parallel data, play not only an important role in scientific research, but also one for historical/cultural documentation.
And as this paper shows, they can also serve as evaluation data for ML models for better understanding and interpretation.
As challenge sets, data for machine processing would need to be statistically diverse and challenging.
Parallel data from previous years have often come in the form of bitexts (2-way parallel text data), usually “word”-tokenized where real length information has been compromised.
(The Bible data from Mayer & Cysouw (2014) came with another confound — the ZH numeral ‘ 一 ’ (“one”) is recognized as a dash (punctuation) and hence tokenized with surrounding whitespaces.)
At the time of our present study (2019-present), the UN Parallel Corpus was the only unperturbed, fully multiway parallel data sufficient for reliable evaluation for our size range.
Data for science, evaluation, and documentation require long-term, stable platform(s) and support.
There are many forms of data science.
But in terms of having a sustainable practice to collect and curate good data, and exercising enough of a science with that data so to improve collective intelligence and mutual understanding, instead of looking at data from an utilitarian point of view only for consumer application purposes, there seems to be room for improvement still.
We hope our work could help effect a positive change in this direction.
A CKNOWLEDGMENTS
The author thanks all anonymous reviewers and PCs as well as the following individuals for their informative comments on previous version(s) of this paper (names in alphabetic order by last name):
Benjamin Börschinger, Jordan Boyd-Graber, Christian Buck, Kyunghyun Cho, Kenneth Church,
Miryam de Lhoneux, Rotem Dror, San Duanmu, Chris Dyer, Roman Flury, John Goldsmith, Yannic
Kilcher, Sandra Kübler, Thomas McColgan, Jason Naradowsky, Ryokan Ri, Anna Rogers, Mark
Rowan, Nikunj Saunshi, Rico Sennrich, Ekaterina Vylomova, Jason Weston, and Kie Zuraw.
We also thank Timothy Baldwin, Kevin Duh, Daniela Gerz, Kenneth Heafield, Felix Hieber, Marcin
Junczys-Dowmunt, Gal Kaplun, Dan Klein, Sabrina Mielke, Mathias Müller, Nakkiran Preetum, and
Annette Rios for their kind responses in correspondence concerning related work and software.
We thank Ismail Moukadiri for his help with the Arabic language.
Some initial experiments were supported by the computing resources at the Department of Informatics and Computational Linguistics at UZH and the Institute of Neuroinformatics at UZH and ETH Zurich (for which we especially thank Richard H.R. Hahnloser, Nikola I. Nikolov, Yuhuang Hu, and Pawel
Pyk).
A PPENDICES
A D ATA SELECTION AND PREPROCESSING DETAILS
The UN Parallel Corpus v1.0 (Ziemski et al., 2016) consists of manually translated UN documents from 1990 to 2014 in the 6 official UN languages.
Therein is a subcorpus that is fully aligned by line, comprising the 6-way parallel corpus we use.
We tried to have as little preprocessing or filtering as necessary to eliminate possible confounds.
But as the initial runs of our experiment failed due to insufficient memory on a single GPU with 12 GB VRAM 11 , we filtered out lines with more than 300 characters in any language in lockstep with one another for all the 6 languages such that the subcorpora would remain parallel, thereby keeping the material of each language semantically equivalent to one another.
8,944,859 lines for each language were retained as our training data which cover up to the 75 th percentile in line length for all 6 languages.
In order to monitor the effect of data size, we made subcorpora of each language in 5 sizes by head ing the first 10 2 , 10 3 , 10 4 , 10 5 , 10lines 12 .
We refer to this as dataset A.
In addition, to better understand and verify the consistency of the phenomena observed, we made 2 supplemental datasets by shuffling the 8,944,859 lines two different times randomly and head ing the number of lines in our 5 sizes for each language, again in lockstep with one another (datasets B and C).
The systematic training regime that we gave to our language directions is identical for all and we controlled also for seeds.
For each of the 3 primary representations — character, byte, and word, we performed:
• 5 runs in 5 sizes ( 10 2 − 10 6 lines): A0 (seed=13), B0 (13), C0 (9948), A1 (9948), A2 (265), and • 7 more runs in 4 sizes ( 10 2 − 10 5 lines): A3 (777), A4 (42), A5 (340589), A6 (1000), A (83146), B1 (9948), & C1 (13).
Figure 1 shows results from all 12 runs in all sizes for the primary representations.
For the alternate/secondary representations, we performed 3 runs each in 5 sizes ( 10 2 - 10 6 lines) (A0,
B0, & C0).
Due to limitations in computing resources, we were not able to perform as many runs as the primary representations.
But important for our statistical comparisons is that we evaluate based on an equal number of runs and on the same data for all candidates.
Tables 1 & 2 are the results.
For each run and each size, there are 30 pairwise directions (i.e. 1 source language to 1 target language, e.g. AR-EN for Arabic to English) that result from the 6 languages.
We trained all 150 jobs ( directions x 5 sizes) for each run and representation using the Transformer model (Vaswani et al., 2017) as supported by the SOCKEYE Toolkit (Hieber et al., 2018)
(version 1.18.85), based on MXNet (Chen et al., 2015).
A detailed description of the architecture of the Transformer can be found in (Vaswani et al., 2017).
The same set of hyperparameters applies to all and its values are listed in
Appendix B.
For character modeling, we used a dummy symbol to denote each whitespace.
For byte, we turned each UTF-8-encoded character into a byte string in decimal value, such that each token is a number between 0 and 255, inclusive.
For word, we followed (Junczys-Dowmunt et al., 2016) and used the Moses tokenizer (Koehn et al., 2007) as is standard in NMT practice when word tokenization is applied and Jieba 13 for segmentation in ZH.
Pinyin is a romanization of ZH characters based on their pronunciations and Wubi an input algorithm that decomposes character-internal information into stroke shape and ordering and matches these to 5 classes of radicals (Lunde, 2008).
For Pinyin, we used the implementation from https: //github.com/lxyu/pinyin in the numerical format such that each character/syllable is followed by a single digit indicating its lexical tone in Mandarin.
For Wubi, we used the dictionary from the implementation from https://github.com/arcsecw/wubi .
We have implemented all representations such that they would be reversible even when the sequence contains code-mixing.
Additional code will be made available at https://github.com/ dadasci .
We used the official dev set as provided in (Ziemski et al., 2016), 3,077 lines per language remained from 4,000 after filtering line length to 300 characters.
Data statistics is provided in Appendix E for reference.
Notes on training time Each run of 30 directions in 5 sizes took approximately 8-12 days for character and byte models.
Byte models generally took longer — hence training time is positively correlated with length (concurring with observations by Cherry et al.
(2018) as they compared character with BPE models).
A maximum length of 300 characters entails a maximum length of at least 300 bytes in UTF-8.
Each run of word models (30 directions, 5 sizes) took about 6 days (excluding the training of some 7-9 directions out of 30 per run involving AR trg or RU trg at 10 6 on word level which took about 12-18 hours each direction to train on a CPU as these required more space and would run out of memory (OOM) on our GPUs otherwise).
These figures do not include the additional probing experiments described in § 4.
B H YPERPARAMETER SETTING
• encoder transformer; • decoder transformer; • num-layers 6:6; • num-embed 512:512; • transformer-model-size 512; • transformer-attention-heads 8; • transformer-feed-forward-num-hidden 2048; • transformer-activation-type relu; • transformer-positional-embedding-type fixed; • transformer-preprocess d; transformer-postprocess drn; • transformer-dropout-attention 0.1; • transformer-dropout-act 0.1; • transformer-dropout-prepost 0.1; • batch-size 15; • batch-type sentence; • max-num-checkpoint-not-improved 3; • max-num-epochs 50; • optimizer adam; • optimized-metric perplexity; • optimizer-params epsilon: 0.000000001, beta1: 0.9, beta2: 0.98; • label-smoothing 0.0; • learning-rate-reduce-num-not-improved 4; • learning-rate-reduce-factor 0.001; • loss-normalization-type valid; • max-seq-len 300 for character, word, and BPE, 672 for all bytes, 688 for Wubi, 680 for Pinyin; • checkpoint-frequency/interval 4000.
(For smaller datasets, the end of 50 epochs is often reached before the first checkpoint.
Since SOCKEYE only outputs scores at checkpoints, we adjusted the checkpoint frequency as follows to get a score outputted by the end of 50 epochs: 1000 for 100 lines for all character & byte instances, 400 for 100 lines for word and 500 for 100 lines BPE, 3450 for 1000 lines for word & BPE.
For the very few cases that this default does not suffice due to bucketing of similar length sequences, we manually set the checkpoint frequency to the last batch.)
C C ORRELATION STATISTICS
Best correlating metrics, i.e. the union of top 3 metrics for all representations.
For each representation, the top 3 metrics are boldfaced.
All correlations are highly significant ( p < 10 − 30 ), except for min source length for WORD ( p ≈ 0 . 0001 ) and min target length for WORD ( p ≈ 0 . 3861 ).
[Table]
The full list of metrics used for the correlation analysis is:
minimum length (source), 2.
minimum length (target), 3.
maximum length (source), 4.
maximum length (target), 5.
median length (source), 6.
median length (target), 7. mean length (source), 8.
mean length (target), 9. length std (source), 10.
length std (target), 11.
data size in lines, 12.
number of parameters, 13. number of types ( | V | ) (source), 14. number of types ( | V | ) (target), 15.
number of tokens (source), 16.
number of tokens (target), 17.
type-token-ratio (TTR) (source), 18.
type-token-ratio (TTR) (target), 19.
OOV type rate (source), 20.
OOV type rate (target), 21.
OOV token rate (source), 22.
OOV token rate (target), 23.
token ratio, 24.
target type-to-parameter ratio, 25.
target token-to-parameter ratio, 26. distance between the TTRs of source and target =
(1-TTR src /TTR trg ) 2 , 27.
token-to-parameter ratio (i) =
(median length source * median length target * num_lines) / num_parameters, 28.
token-to-parameter ratio (ii) =
(num_source_tokens * num_target_tokens) / num_parameters.
D S TATISTICAL COMPARISONS
Recall the definition and method for our Disparity/Inequality assessment from § 2:
In the context of our CLMing experiments, we consider there to be “disparity” or “inequality” between languages l 1 and l 2 if there are significant differences between the performance distributions of these two languages with respect to each representation.
Here, by performance we mean the number of bits required to encode the held-out data using a trained CLM.
With 30 directions, there arepairs of source languages ( l src 1 , l src 2 ) and 15 pairs of target languages ( l trg 1 , l trg 2 ) possible.
We compare the source languages among each other, and the target languages among each other.
Each l src or each l trg consists of scores from all models trained across various sizes and directions.
To assess whether the differences are significant, we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different.
Upon testing for normality with the Shapiro-Wilk test (Shapiro & Wilk, 1965; Royston, 1995), we use the parametric unpaired two-sample Welch’s t-test (Welch, 1947) (when normal) or the non-parametric unpaired Wilcoxon test (Wilcoxon, 1945) (when not normal) for the comparisons.
We use the implementation in R (R Core
Team, 2014) for these 3 tests.
To account for the multiple comparisons we are performing, we correct all p-values using Bonferroni’s correction (Benjamini &
Heller, 2008; Dror et al., 2017) and follow Holm’s procedure 14 (Holm, 1979; Dror et al., 2017) to identify the pairs of l 1 and l 2 with significant differences after correction.
We report all 3 levels of significance ( α ≤ 0 .
for a more comprehensive overview.
In contrast to Dror et al.
(2017), which aimed to compare the performance of different algorithms, we compare languages (in the context of computing).
To get samples for the statistical comparison results for the Disparity Table (Table 1):
For each representation, we used 3 runs (A0, B0, C0) in 5 sizes ( 10 2 - 10 6 lines) for each l src and each l trg .
There are:
[Equation]
src (AR src , EN src , ES src , FR src , RU src , ZH src ) and 6 l trg (AR trg , EN trg , ES trg , FR trg , RU trg , ZH trg )
[Equation]
We compare pairwise among the l src .
The 15 pairs are:
[Equation]
Likewise 15 pairs among the l trg .
For example , for the character (primary) representation to compare between AR src and EN src , we construct the sample for AR src (sample ARsrc ) and the sample for EN src (sample ENsrc ) as follows:
Out of the 30 CHAR directions, there are 5 directions involving AR src trained for each run and data size (i.e. the directions: AR-EN, AR-ES, AR-FR, AR-RU, AR-ZH).
For each direction, there are 15 models trained (3 runs x 5 sizes).
We take all 75 CHAR models ( models x 5 directions) involving AR src as sample ARsrc .
That’s a sample of size 75.
Likewise, for EN src (5 directions: EN-AR, EN-ES, EN-FR, EN-RU, EN-ZH), we also have 75 data points for sample ENsrc .
(Likewise also for all 6 l src and all 6 l trg .)
For the comparisons, we compare pairwise, i.e. with two samples each time, but with unpaired two-sample Welch’s t-test (when normal) or the non-parametric unpaired Wilcoxon test (when not normal) because sample ARsrc and sample ENsrc have one direction that is not paired: AR-EN and
EN-AR.
Other directions can be seen as paired, e.g. AR-ES and EN-ES as both having the same l trg .
[Table]
o f t yp e s a nd t
ok e n s i n t h
e
d a t a .
T h
i s i s a r oughp r oxy f
o r l e x i ca l d
i v e r
s it
y i n t h a t a v a l u e o f 1 w ou l d i nd
i ca t e t h a t
no t yp e i s e v e
r s ee n t w i ce , a nd a v a l u e v e r y c
l o s e t o 0 w ou l d i nd
i ca t e t h a t v e r
[Table]
d i s ti n c tt yp
e s acc oun
t f o r a l m o s t a ll o f t
h e d a t a .
• L i n e l e ng t h ( e x c l .
E O
S / B O S m a r k e r) : m ea n ± s t a nd a r dd e v i a ti on , a nd t h e 0 //
50 / 75 / 100 - t hp e r ce n til e .
S t a ti s ti c s f o r d a t a s e t A
R e p r e s e n t a ti on
C HA R
B Y TE
W O R D
B P E
N u m b e r o f li n e s 101 ,10 ,100 ,1 , 000 , 1 ,
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
N u m b er o f T
Y P E S
A R           5 ,26 ,97 ,304 , 4 ,18 ,29 ,29 ,
E N           3 ,12 ,37 ,122 , 2 ,10 ,26 ,29 ,
E S           3 ,15 ,49 ,142 , 3 ,12 ,27 ,29 ,
F R           3 ,13 ,42 ,118 , 3 ,11 ,27 ,29 ,
R U           5 ,23 ,81 ,236 , 4 ,17 ,29 ,29 ,
Z H 601 ,2 ,3 ,4 ,      3 ,13 ,46 ,134 , 2 ,10 ,27 ,29 ,
Z H _p i ny i n
Z H _ w ub i
A R _ c p
R U _ c p
N u m b er o f T
OK E N S
A R 9 , 07123 ,1 , 083 ,10 , 625 ,102 , 064 , 2316 ,227 ,1 , 985 ,19 , 487 ,186 , 171 ,1 ,23 ,206 ,2 , 035 ,19 , 410 ,2 ,28 ,223 ,2 , 218 ,21 , 344 ,
E N 11 ,159 ,1 , 344 ,13 , 132 ,123 , 491 , 8711 ,159 ,1 , 345 ,13 , 158 ,123 , 705 ,2 ,27 ,236 ,2 , 339 ,21 , 943 ,2 ,29 ,243 ,2 , 369 ,22 , 351 ,
E S 12 ,171 ,1 , 484 ,14 , 549 ,138 , 596 , 0312 ,175 ,1 , 513 ,14 , 821 ,141 , 276 ,2 ,29 ,263 ,2 , 588 ,24 , 654 ,3 ,32 ,271 ,2 , 644 ,25 , 264 ,
F R 12 ,179 ,1 , 490 ,14 , 528 ,138 , 049 , 1812 ,185 ,1 , 542 ,15 , 055 ,143 , 495 ,2 ,32 ,273 ,2 , 684 ,25 , 595 ,3 ,34 ,280 ,2 , 727 ,26 , 077 ,
R U 11 ,168 ,1 , 436 ,14 , 151 ,134 , 706 , 1221 ,309 ,2 , 636 ,25 , 990 ,247 , 098 ,1 ,24 ,216 ,
[Table]
H _
[Table]
i
[Table]
i n
[Table]
[Table] [Table]
N
[Table]
N
[Table]
S
[Table]
U
[Table]
H
[Table] [Table]
Z H _p i
[Table]
i n
[Table] [Table]
[Table]
Z H _
[Table]
i
[Table] [Table]
A R _ c
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
U
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Z H 33 .
18 ± 18 .42 .
57 ± 16 .37 .
20 ± 18 .36 .
60 ± 19 .34 .
67 ± 19 .85 .
59 ± 51 .116 .
67 ± 43 .102 . 00 ± 50 .99 .
90 ± 51 .94 .
27 ± 52 .17 .
51 ± 10 .23 .
57 ± 9 .20 .
77 ± 10 .20 .
39 ± 10 .19 .
36 ± 10 .22 .
[Table]
± 12 .25 .
46 ± 10 .
[Table]
52 ±
[Table] [Table]
±
[Table]
Z H _p i
ny i n
11 ± 65 .148 .
08 ± 55 .127 .
96 ± 63 .125 .
31 ± 65 .118 .
47 ± 66 .7 / 43 / 98 //7 / 109 / 149 / 188 /1 / 82 / 129 / 175 /1 / 76 / 126 / 174 /1 / 63 / 119 / 169 /
Z H _ w ub i 92 .
45 ± 55 .127 .
19 ± 48 .110 .
45 ± 54 .108 .
86 ± 56 .102 .
91 ± 57 .8 / 39 / 86 //7 / 94 / 129 / 158 /1 / 71 / 111 / 150 /1 / 67 / 110 / 150 /1 / 55 / 103 / 146 /
A R _ c p 90 .
79 ± 54 .123 .
83 ± 47 .108 .
35 ± 53 .106 .
25 ± 54 .102 .
06 ± 57 .8 / 33 / 91 //5 /
R U _ c p 119 .
80 ± 72 .168 .
16 ± 62 .143 .
61 ± 71 .141 .
52 ±
[Table]
71 ±
S t a ti s ti c s f o r d a t a s e t B
R e p r e s e n t a ti on
C HA R
B Y TE
W O R D
B P E
N u m b e r o f li n e s 11 ,10 ,100 ,1 , 000 , 1 ,10 ,100 ,1 , 000 , 1 ,10 ,100 ,1 , 000 , 1 ,10 ,100 ,1 , 000 ,
N u m b er o f T
Y P E S
A R 8         1 ,6 ,28 ,97 ,311 , 4 ,18 ,29 ,29 ,
E N 7          4 ,13 ,42 ,140 , 3 ,10 ,26 ,29 ,
E S 8          4 ,16 ,51 ,154 , 3 ,12 ,28 ,29 ,
F R 8          4 ,15 ,45 ,132 , 3 ,12 ,27 ,29 ,
R U 1         1 ,6 ,25 ,83 ,251 , 4 ,18 ,29 ,29 ,
Z H 71 ,2 ,3 ,4 ,      4 ,14 ,46 ,140 ,1 ,3 ,11 ,27 ,29 ,
Z H _p i
ny i n
Z H _ w ub i
A R _ c p
R U _ c p
N u m b er o f T
OK E N S
A R 9 , 7100 ,1 , 019 ,10 , 248 ,102 , 481 , 8117 ,181 ,1 , 842 ,18 , 517 ,185 , 172 ,1 ,18 ,190 ,1 , 915 ,19 , 138 ,3 ,25 ,210 ,2 , 090 ,20 , 986 ,
E N 11 , 8119 ,1 , 201 ,12 , 087 ,120 , 969 , 1611 ,119 ,1 , 202 ,12 , 091 ,121 , 006 ,2 ,21 ,211 ,2 , 122 ,21 , 234 ,3 ,24 ,220 ,2 , 159 ,21 , 669 ,
E S 13 , 5135 ,1 , 363 ,13 , 709 ,137 , 180 , 5413 ,137 ,1 , 389 ,13 , 967 ,139 , 753 ,2 ,23 ,241 ,2 , 431 ,24 , 326 ,3 ,28 ,253 ,2 , 485 ,24 , 933 ,
F R 13 , 1134 ,1 , 358 ,13 , 662 ,136 , 811 , 8713 ,139 ,1 , 404 ,14 , 121 ,141 , 403 ,2 ,24 ,247 ,2 , 485 ,24 , 874 ,3 ,28 ,257 ,2 , 527 ,25 , 374 ,
R U 13 , 1130 ,1 , 325 ,13 , 356 ,133 , 699 , 4724 ,239 ,2 , 433 ,24 , 512 ,245 , 396 ,1 ,19 ,200 ,2 , 014 ,20 , 153 ,3 ,26 ,218 ,2 , 170 ,21 , 792 ,
Z H 3 , 333 ,340 ,3 , 419 ,34 , 206 ,9 ,92 ,935 ,9 , 402 ,94 , 064 ,1 ,18 ,189 ,1 , 906 ,19 , 073 ,2 ,21 ,199 ,1 , 946 ,19 , 574 ,
Z H _p i ny i n 11 , 3115 ,1 , 166 ,11 , 732 ,117 , 367 ,
Z H _ w ub i 10 , 0101 ,1 , 019 ,10 , 242 ,102 , 460 ,
A R _ c p 9 ,100 ,1 , 019 ,10 , 248 ,
[Table]
N
[Table]
S
[Table] [Table]
R
[Table]
[Table] [Table]
U
[Table]
N
[Table]
n
[Table]
n
[Table]
Z H _p i ny i n 113 .
91 ± 64 .115 .
58 ± 69 .116 .
62 ± 68 .117 .
32 ± 67 .117 .
37 ± 67 .8 / 66 / 109 / 174 /2 /
Z H _ w ub i 100 .
35 ± 55 .101 .
05 ± 60 .101 .
91 ± 58 .102 .
43 ± 58 .102 .
46 ± 58 .8 / 59 / 99 //2 / 48 / 100 / 146 /1 / 52 / 102 / 147 /1 / 53 / 102 / 147 /1 / 54 / 103 / 146 /
A R _ c p 97 .
98 ± 55 .100 .
60 ± 59 .101 .
97 ± 59 .102 .
49 ± 58 .102 .
48 ± 58 .9 / 53 / 94 //2 / 49 / 101 / 147 /1 / 51 / 101 / 147 /1 / 53 / 103 / 147 /1 / 53 / 102 / 147 /
R U _ c p 131 .
06 ± 76 .130 .
29 ± 77 .132 .
58 ± 77 .133 .
57 ± 77 .133 .
70 ± 77 .10 / 71 / 123 / 195 /2 / 62 / 130 / 193 /1 / 65 / 133 / 195 /1 / 67 / 134 / 194 /1 / 67 / 134 / 195 /
S t a ti s ti c s f o r d a t a s e t C
R e p r e s e n t a ti on
C HA R
B Y TE
W O R D
B P E
N u m b e r o f li n e s 101 ,10 ,100 ,1 , 000 , 00 1 ,10 ,100 ,1 , 000 , 1 ,10 ,100 ,1 , 000 , 1 ,10 ,100 ,1 , 000 ,
N u m b er o f T
Y P E S
A R          1 ,6 ,28 ,97 ,310 , 4 ,18 ,29 ,29 ,
E N           4 ,13 ,42 ,141 , 3 ,10 ,26 ,29 ,
E S           4 ,16 ,51 ,154 , 3 ,12 ,28 ,29 ,
F R           4 ,15 ,45 ,132 , 3 ,12 ,27 ,29 ,
R U          1 ,6 ,26 ,84 ,250 ,1 ,4 ,19 ,29 ,29 ,
Z H 781 ,2 ,3 ,4 ,      4 ,14 ,46 ,139 ,1 ,3 ,11 ,27 ,29 ,
Z H _p i ny i n
Z H _ w ub
i
A R _ c p
R U _ c p
N u m b er o f T
OK E N S
A R 10 ,104 ,1 , 021 ,10 , 231 ,102 , 448 , 318 ,188 ,1 , 845 ,18 , 482 ,185 , 105 ,1 ,19 ,190 ,1 , 912 ,19 , 137 ,3 ,25 ,210 ,2 , 087 ,20 , 981 ,
E N 12 ,123 ,1 , 208 ,12 , 089 ,120 , 933 , 912 ,123 ,1 , 209 ,12 , 093 ,120 , 971 ,2 ,21 ,212 ,2 , 123 ,21 , 231 ,3 ,25 ,221 ,2 , 160 ,21 , 666 ,
E S 14 ,139 ,1 , 369 ,13 , 701 ,137 , 133 , 414 ,142 ,1 , 395 ,13 , 958 ,139 , 702 ,2 ,24 ,242 ,2 , 430 ,24 , 319 ,3 ,28 ,254 ,2 , 483 ,24 , 925 ,
F R 13 ,138 ,1 , 365 ,13 , 661 ,136 , 754 , 314 ,143 ,1 , 411 ,14 , 119 ,141 , 343 ,2 ,24 ,248 ,2 , 485 ,24 , 864 ,3 ,28 ,258 ,2 , 527 ,25 , 362 ,
R U 13 ,136 ,1 , 339 ,13 , 369 ,133 , 671 , 325 ,251 ,2 , 457 ,24 , 535 ,245 , 338 ,2 ,20 ,201 ,2 , 016 ,20 , 156 ,3 ,26 ,219 ,2 , 172 ,21 , 787 ,
Z H 3 ,34 ,341 ,3 , 419 ,34 , 200 , 59 ,95 ,938 ,9 , 404 ,94 , 052 ,1 ,19 ,190 ,1 , 907 ,19 , 073 ,2 ,22 ,199 ,1 , 947 ,19 , 573 ,
Z H _p i
ny i n 11 ,119 ,1 , 171 ,11 , 732 ,117 , 356 ,
Z H _ w ub i 10 ,103 ,1 , 022 ,10 , 245 ,102 , 450 ,
A R _ c p 10 ,104 ,1 , 021 ,10 , 231 ,102 , 448 ,
R U _ c p 13 ,
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
E S
[Table]
±
[Table] [Table]
±
[Table]
[Table] [Table]
[Table] [Table]
±
[Table]
[Table] [Table]
[Table] [Table]
±
[Table]
[Table] [Table]
[Table] [Table]
±
[Table]
[Table] [Table]
±
[Table]
[Table] [Table]
±
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
±
[Table] [Table]
[Table] [Table]
Z H _p i ny i n 116 .
05 ± 67 .119 .
26 ± 68 .117 .
16 ± 67 .117 .
32 ± 67 .117 .
36 ± 6 7 .11 / 58 / 11 7 // 263 3 / 61 / 122 / 173 /3 / 61 / 117 / 169 /1 / 60 / 118 / 169 /1 / 60 /
Z H _ w ub i 100 .
71 ± 57 .103 .
94 ± 59 .102 .
29 ± 57 .102 .
46 ± 58 .102 .
45 ± 5 8 .13 / 54 / 10 1 // 217 3 / 54 / 105 / 150 /3 / 54 / 102 / 147 /1 / 53 / 102 /
A R _ c p 102 .
12 ± 57 .104 .
14 ± 57 .102 .
11 ± 58 .102 .
31 ± 58 .102 .
45 ± 58 .5 / 57 / 96 //4 / 54 / 106 / 149 /3 / 53 / 102 / 147 /1 / 53 / 102 / 147 /1 / 53 / 102 / 147 /
R U _ c p 137 .
63 ± 80 .136 .
67 ± 78 .133 .
93 ± 77 .
[Table]
70 ±
[Table] [Table]
± 77 .11 / 71 / 130 / 212 /3 / 63 / 139 / 200 /1 / 67 / 135 / 196 /1 / 67 / 134 / 195 /1 / 67 / 134 / 195 /
S t a ti s ti c s f o r d e v e l op m e n t ( d e v ) s e t
A s a d i ff e r e n t s e t
o f
vo ca bu l a r y
i s l ea r n e d fr o m eac h t
r a i n i ngd a t a s e t a ndd a t a s i ze , B
P E h a s a d
i s ti n c t
d
e v s e t
f
o r eac h .
R e p r e s e n t a ti on
C HA R
B Y TE
W O R D
B P E _ A B P E _ B
B P E _ C
N u m b e r o f li n e s i n t r a i n s e t
[Table]
N u m
[Table]
er o f T
Y P E S
A R
[Table]
E N
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
E S
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
F R  8 , 2 ,6 ,8 ,8 , 3 ,7 ,8 ,8 , 3 ,7 ,8 ,8 ,
R U  12 , 3 ,10 ,12 ,12 ,1 ,4 ,11 ,12 ,12 ,1 ,4 ,11 ,12 ,12 ,
Z H 1 , 7 ,3 ,4 ,6 ,7 ,7 ,3 ,4 ,6 ,7 ,7 ,3 ,4 ,6 ,7 ,7 ,
Z H _p i ny i n
Z H _ w ub
i
A R _ c p
R U _ c p
N u m b er o f T
OK E N S
A R 334 ,605 ,61 ,167 ,115 ,83 ,76 ,70 ,149 ,97 ,73 ,68 ,68 ,149 ,97 ,73 ,68 ,68 ,
E N 391 ,391 ,67 ,155 ,101 ,77 ,71 ,70 ,140 ,90 ,73 ,69 ,69 ,140 ,90 ,73 ,69 ,69 ,
E S 443 ,452 ,
[Table] [Table]
[Table] [Table]
[Table]
F R 438 ,452 ,78 ,166 ,
[Table]
[Table] [Table]
R U
[Table]
[Table] [Table]
Z H 107 ,
[Table] [Table]
Z H _p i ny i n 376 ,
Z H _ w ub i 330 ,
A R _ c p 334 ,
R U _ c p 431 ,
TT R ( % )
A R 0 .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
E N 0 .
[Table] [Table]
U _
[Table]
n li
[Table]
l
[Table]
n g t h ± s t d
[Table]
[Table] [Table]
Z H
[Table]
±
[Table]
85 ±
[Table]
±
[Table]
±
[Table]
±
[Table]
±
[Table]
±
[Table]
±
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Z H _p i ny i n 122 .
52 ±
[Table]
Z H _ w ub i 107 .
49 ± 56 .4 / 60 / 108 / 151 /
A R _ c p 108 . 66 ± 58 .3 / 60 / 110 / 153 /
R U _ c p
[Figure]
[Figure]
nu m be r o f b i t s
[Figure]
AR
EN
ES
FR
RU
ZH
SRC
AR
EN
ES
FR
RU
ZH
[Figure]
nu m be r o f b i t s
TRG
AR
EN
ES
FR
RU
ZH
SRC
AR
EN
ES
FR
RU
ZH
[Figure]
[Figure]
nu m be r o f b i t s
TRG
AR
EN ES
FR
RU
ZH
SRC AR
EN
ES
FR
RU
ZH
[Figure]
nu m be r o f b i t s
[Figure]
[Figure]
nu m be r o f b i t s
TRG
AR
EN
ES
FR
RU
ZH
SRC
AR
EN
ES
FR
RU
ZH
[Figure]
nu m be r o f b i t s
[Figure]
[Figure]
nu m be r o f b i t s
TRG
AR
EN
ES
FR
RU
ZH
SRC
AR
EN
ES
FR
RU
ZH
[Figure]
m be r o f b i t s
[Figure]
[Figure]
nu m be r o f b i t s
TRG
AR
EN
ES
FR
RU
ZH
SRC
AR
EN
ES
FR
RU
ZH
[Figure]
m be r o f b i t s
[Figure] [Figure]
be
[Figure]
i
[Figure]
AR
EN
ES
FR
RU
ZH
SRC
AR
EN
ES
FR
RU
ZH
[Figure] [Figure]
nu
[Figure]
be r o f b i t s
[Figure]
RU
ZH
[Figure]
AR
EN
[Figure]
FR
RU
ZH
[Figure]
nu m be r o f b i t s
TRG
AR
EN
ES
FR
RU
ZH
SRC
AR
EN
ES
FR
RU
ZH
[Figure]
nu m be r o f b i t s
[Figure]
[Figure]
nu m be r o f b i t s
TRG
AR
EN
ES
FR
RU
ZH
SRC
AR EN
ES
FR
RU
ZH
[Figure]
nu m be r o f b i t s
[Figure]
on
Version 1.
ICLR 2022 camera-ready copy (20220510)