BERT Cannot Align Characters (2021)

Abstract: In previous work, it has been shown that BERT can adequately align cross-lingual sentences on the word level. Here we investigate whether BERT can also operate as a char-level aligner. The languages examined are English, Fake English, German and Greek. We show that the closer two languages are, the better BERT can align them on the character level. BERT indeed works well in English to Fake English alignment, but this does not generalize to natural languages to the same extent. Nevertheless, the proximity of two languages does seem to be a factor. English is more related to German than to Greek and this is reflected in how well BERT aligns them; English to German is better than English to Greek. We examine multiple setups and show that the similarity matrices for natural languages show weaker relations the further apart two languages are.

## introduction for the many sweeping successes bert has had in the field of natural language processing, the model's alignment capabilities have been lacking and under-explored. work in this area is picking up and it has been shown that bert can operate with adequate efficiency in word alignment tasks @xcite . the question whether or not bert can perform character-level alignment, though, has not been answered yet. with our work we show that even though bert is not able to align languages on the character level, the closer these languages are the better the alignment. in the trivial case of english to fake-english alignment, the model successfully learns to align characters. for english to german performance drops substantially and it drops even more for english to greek. languages seem to be put on an intuitive scale by bert, with more similar languages having better alignment than highly dissimilar ones. ## related work research has been conducted to uncover elements of multilinguality in mbert. in @xcite , an analysis of mbert is presented, while in conneau and lample (2019); wu and dredze (2019); artetxe et al. (2020) zero-shot cross-lingual transfer is analyzed. dufter and schütze (2020) further analyzes mbert's capabilities with bert's architecture and the structure of languages examined. the authors performed their experiments on a pairing of english with fake-english, as proposed by @xcite in their rigorous empirical study of mbert where linguistic properties of languages, architecture and learning objectives are investigated. in @xcite ; @xcite , it is shown that transformers @xcite can achieve similar performance with sequenceto-sequence approaches based on recurrent neural networks @xcite for characterlevel tasks such as transliteration and graphemeto-phoneme conversion. further work to develop character-level bert-based models is conducted in el @xcite ; @xcite . och and ney (2003); legrand et al. (2016) worked towards representation-based word alignment, with implementations of aligners proposed in @xcite ; @xcite . we also note recent efforts towards unsupervised word alignment. in @xcite , an extension to the usual machine translation encoderdecoder is proposed to jointly learn language translation and word alignment in an unsupervised manner. bert has also been shown to be able to perform word alignment @xcite through embedding matrix similarities. ## ablation studies we also performed some minor experiments with static embeddings and adjustments to the mlm task, as well as a variation to our engf ake setup. apart from the bert embeddings, we also examined static embeddings, to introduce another baseline. specifically, we used the fasttext algorithm @xcite . data was the same as the previous experiments. results were seemingly random, with fasttext unable to capture any meaningful representations. cosine similarities were generally very low and alignments incoherent. for the mlm task, we experimented with different probabilities for token masking. in the origi-nal bert paper, the chosen token is replaced by [mask] 80% of the time, while it remains the same 10% of the time and gets replaced by a random token the rest 10%. we experimented with the following distributions: 80/20/0, 60/20/20 and 50/50/0 with no noticeable change in performance. we also experimented with overlapping fake-english mappings in the form 'a' → '100 101 102'. that is, each character is mapped to the tokens corresponding to itself and its next two characters (eg. 'a' is mapped to the tokens for 'a @xcite . this is an extreme case of the engf ake f 1 f 2 setup, where each character is mapped to three (instead of 'f' getting mapped to 'f1' and 'f2'). results deteriorated to random performance. finally, we conducted another engf ake experiment. namely, we restricted the context where we present each character. for the previous experiments each word was split in characters. now, we further split each word into trigrams. so, instead of 'e x a m p l e' we end up with multiple overlapping trigram entries: 'e x a', 'x a m', 'a m p', etc. the motivation for this experiment is that since the transliteration of a character in a word doesn't need the entire word but instead only its direct neighbors, we should examine a setup with more restricted context foot_3 . in this case, results deteriorated heavily and are seemingly random with no meaningful representation captured. ## conclusion in our work bert is shown to be unable to create consistently good cross-lingual spaces on the character level. we train models on english, german, greek and fake-english and we compare characterlevel alignments between them. cosine similarity matrices between the target and source alphabets were examined and we found that the closer two languages are, the better bert does in aligning them. fake-english is the easiest to align with english, whereas german is worse, with greek trailing far behind. we conclude that bert is not able to perform adequate character alignment. (a) engdeu (b) engell (c) deuell figure 2: showing heatmaps for (a) engdeu, (b) engell and (c) deuell. the lighter green cells show lower cosine similarity, the darker green cells show higher cosine similarity.