The Z ipfian Challenge: Learning the statistical fingerprint of natural languages (2023)

Abstract: Human languages are often claimed to fundamentally differ from other communication systems. But what is it exactly that unites them as a separate category? This article proposes to approach this problem – here termed the Zipfian Challenge – as a standard classification task. A corpus with textual material from diverse writing systems and languages, as well as other symbolic and non-symbolic systems, is provided. These are subsequently used to train and test binary classification algorithms, assigning labels “writing” and “non-writing” to character strings of the test sets. The performance is generally high, reaching 98% accuracy for the best algorithms. Human languages emerge to have a statistical fingerprint: large unit inventories, high entropy, and few repetitions of adjacent units. This fingerprint can be used to tease them apart from other symbolic and non-symbolic systems.

## introduction however, the challenge posed in the quote above is still only partially addressed by research into statistical laws. namely, a statistical pattern might universally occur across languages, but this does not entail that it is a unique feature of languages. the zipfian challenge is ultimately the search for a statistical fingerprint: a feature, or set of features, which uniquely identify human languages. this is related to an age-old controversy of the language sciences: what makes human language special -if anything? this challenge is here broken down into a standard classification task. assume you are provided with strings of characters: foot_0 aallaqqaasiuta sssscsofspppfpp (1) is there an algorithm which robustly classifies these into "writing" and "non-writing"? -if yes, how? -if no, why not? beyond pure scientific curiosity, there would be concrete applications for such an algorithm: a) cleaning of contaminated corpora, especially when large and automatically crawled @xcite ; b) measuring similarity of undeciphered scripts to known writing systems in order to help decipherement @xcite @xcite ; c) providing tools to systematically compare human language with animal communication @xcite . in the following, a corpus of character strings labelled as "writing" and "non-writing" is introduced in section 2. given this corpus, a sampling procedure is defined to retrieve strings of predefined lengths (10, 100, 1000). subsequently, features from quantitative linguistics and information theory are described an calculated on the strings (section 3). a series of classification algorithms are trained on a subset of the feature values. section 4 then gives the results in terms of performance of the algorithms on the test sets. section 5 discusses the results with regards to the original research question of a statistical fingerprint, as well as some follow-up questions which arise from the results. ## data the data stems from a corpus of overall 377 files, split into "writing" (170 files) and "non-writing" (207). 2 the standard definition of writing is applied here. it refers to the tight link between spoken language structure and the graphemes representing it: "broadly defined, writing represents speech. one must be able to recover the spoken word, unambiguously, from a system of visible marks in order for those marks to be considered writing," @xcite . however, some transcriptions of sign languages are also included here. arguably, unique structural features of a given sign language can be identified in a transcrip-2 files and code can be found at https://github.com/ christianbentz/nalafi . ## results for all classification algorithms the accuracy, precision, recall, and f1 score on the test set are reported alongside the respective hyperparameters. a condensed overview of classification results are given in ## discussion overall, the classification results suggest that the zipfian challenge is indeed a solvable problem. namely, given strings of characters of length 100, knns and mlps reach performance values of 0.92 and 0.93 respectively. with 1000 characters, they are almost at the ceiling of performance. in fact, it is questionable whether humans would be able to correctly classify the respective strings with 100% accuracy. mind you that more than 36 different scripts and 90 different languages are represented in this data sample. it would be an interesting project for future research to establish human performance on this task. in the following, some further follow-up questions are briefly discussed. ## conclusions compared to other symbolic and non-symbolic systems, natural languages seem to exhibit a unique fingerprint: relatively large unit inventories, relatively high entropy, and relatively few repetitions of adjacent units. this statistical fin-gerprint can be used to identify written language with high accuracy when more than 100 characters are provided. interestingly, this seems to hold not only for writing reflecting spoken language but also for transcriptions of sign languages (though only small samples of asl and sln were used here). this suggests that humans have evolved the capacity of encoding information with a diverse, non-repetitive succession of units in three modalities: speech, manual signs, and graphical codes. if these results hold true, then it is not a single feature, and not a single modality, which defines human language, but a set of features related to rapid information transmission in the face of space and time limitations.