Cuaċ: Fast and Small Universal Representations of Corpora (2025)

Abstract: The increasing size and diversity of corpora in natural language processing requires highly efficient processing frameworks. Building on the universal corpus format, Teanga, we present Cuaċ, a format for the compact representation of corpora. We describe this methodology based on short-string compression and indexing techniques and show that the files created with this methodology are similar to compressed human-readable serializations and can be further compressed using lossless compression. We also show that this introduces no computational penalty on the time to process files. This methodology aims to speed up natural language processing pipelines and is the basis for a fast database system for corpora.

## introduction the size and scope of corpora used in natural language processing (nlp) applications have grown massively in the past few years and as such the efficient storage and retrieval of large-scale linguistic corpora are critical for these applications. the growth of textual data means the traditional storage and annotation formats can present significant challenges to real-world applications. we recently proposed the teanga format @xcite , which provides a universal method of annotating corpora, principally through a metamodel serialized in yaml and other formats. while this model addresses key challenges in the accessibility and interoperability of corpora, the yaml format does not provide an efficient method for working with corpora in this data model. in this paper, we present cuaċ foot_0 (compression of universal annotated corpora), a new serialization method, which addresses these challenges by providing a compact, high-performance representation of annotated corpora. cuaċ is designed to reduce the size of corpora as stored on disk, while still allowing full searchability of the corpus and to avoid increasing processing times. it is built on top of the teanga data model, and integrates annotation layers using multiple compression techniques in order to minimize the redundant representation of information. it also incorporates lightweight text compression methods, alongside indexed integer representations to reduce the storage size of the data. in this paper, we present the cuaċ format and its implementation within the teanga framework. we discuss the compression strategies and the indexing mechanisms. we evaluate its performance in terms of the file size reduction as well as the time taken to process the records. our results show that cuaċ not only outperforms conventional formats such as yaml and json, which are similar to the xml and conll-u formats used originally for the corpora, but also strongly outperforms parquet, a binary format that is widely used for sharing datasets including corpora. as such cuaċ is a practical solution for handling large-scale annotated corpora in nlp research and applications. ## related work effective compression and representation of large data sets are essential for scalable storage and retrieval. brotli @xcite , a compression algorithm developed by google, provides high compression ratios through a static dictionary and transformation techniques, making it ideal for web and textual data compression. in contrast to traditional methods like deflate, brotli greatly enhances both compression density and decompression speed, positioning it as a strong candidate for compact corpus representation. when dealing with structured data, especially rdf (resource description framework), compression strategies concentrate on removing structural redundancies. the hdt (header-dictionary-triples) format @xcite is a commonly used binary serialization that streamlines rdf data by replacing textual terms with numeric identifiers @xcite . advanced optimizations like hdt++ build on this by utilizing schema-based redundancies, such as families of predicates and typed subjects, effectively reducing storage needs by half compared to standard hdt serialization @xcite . another innovative method, k²-triples, enhances rdf graph compression by dividing rdf triples into predicate-specific binary matrices, which are then encoded using k²-trees. this strategy improves structural compression while ensuring efficient query performance. furthermore, rdf-tr enhances rdf compression by reorganizing triples to reflect recurring structural patterns, which leads to better space efficiency and faster retrieval speeds @xcite . advancements in rdf data compression have led to the development of grammar-based techniques, such as grdf, which utilizes the grepair algorithm to identify and compress repetitive patterns within rdf datasets, achieving substantial reductions in data size while preserving structural integrity @xcite . additionally, compressed indexing methods, including trie-based layouts and circular suffix sorting, have been introduced to compactly represent rdf triples, enabling efficient pattern-matching operations and enhancing query execution speeds @xcite . these innovations address the challenges posed by the increasing volume of rdf data, facilitating more efficient storage and retrieval processes. beyond grammar-based and indexing techniques, researchers have explored estimation-based optimizations for compressing rdf knowledge bases. these methods analyze input and intermediate data to improve compression efficiency, reducing storage overhead while preserving query performance @xcite . additionally, machine learning-driven compression has gained traction, with inductive autoencoders learning compact representations of rdf graphs by identifying latent structures and redundant patterns, leading to improved storage efficiency and faster retrieval @xcite . such techniques signal a shift toward hybrid approaches that integrate statistical learning with structural compression, paving the way for more scalable rdf management solutions. these advancements in compression and serialization demonstrate the potential for fast and small universal representations of corpora, balancing storage efficiency with rapid access and processing capabilities. ## results to evaluate the effectiveness of the cuaċ format we evaluate it on two main measures. firstly, we consider the file sizes and show that these are reduced 3 the string length is given in preference to the null termination, so that the null character is available for text compression by the use of this model. secondly, we consider the time to convert a file into and from cuaċ. this is important as it shows that the format does not introduce significant runtime overheads compared to using a more verbose format, and in fact, shows that our more compressed model is faster to read than the uncompressed version. as a baseline, we consider the formats proposed for teanga in @xcite serialized in either yaml or json. as these are formats meant for human consumption, we also compare to parquet @xcite as a binary format that is used by many projects including hadoop, and pandas. we use the python implementation of arrow foot_2 to convert data into this format. in addition, we consider the effect of further compressing files using lossless compression algorithms. we consider the following algorithms: deflate deflate @xcite ) is a lossless compression algorithm that combines lz77 and huffman coding, used in formats like gzip and png to efficiently reduce file sizes. zstd zstandard @xcite is a fast, lossless compression algorithm, introduced by meta, that provides high compression ratios and low latency, making it efficient for real-time and large-scale data compression. bw the burrows-wheeler transform @xcite is a reversible block-sorting compression algorithm that improves redundancy for better entropy encoding, forming the core of bzip2, which enhances compression efficiency with bwt, huffman coding, and run-length encoding. we evaluate our methodology across a wide range of corpora. firstly, we evaluate a small section of the colossal common crawl corpus (raffel et al., 2019, c4) in a section identified as english web texts. we chose the first file from this corpus for our experiments and applied linguistic analysis with spacy foot_3 to this corpus. secondly, we have converted all the nltk corpora @xcite into the teanga format foot_4 . this collection includes a wide range of corpora from plain-text corpora to tagged and parsed corpora. these are primarily english-language corpora but also include corpora in a wide range of languages. we also converted the universal dependencies @xcite corpora collection in its entirety, which is a homogeneous corpus collection in terms of its annotations, however, it covers a wide range of languages and is useful to study the performance of the system across languages. lastly, we converted the xl-wa @xcite and the europarl corpus @xcite to the teanga format. both datasets are provided in parallel text formats. xl-wa is a manually-curated benchmark for word alignment, covering 14 language pairs, including english and languages such as arabic, chinese, and russian. it includes gold-standard word alignment annotations. for the europarl corpus, which is derived from european parliament proceedings and offers parallel texts across 21 european languages, we generated word alignment annotations using fast_align @xcite . in in order to measure the effect of text compression presented in section 3.2, we compare the size of compressed files on the ud corpora. again we present the absolute file size for the serialization without compression and the relative size for other compression methods. we see that for the english corpus, all methods provide similar size reduction in file sizes. for other languages using the latin script, we generally see that the default tables provide effective text compression for most languages, except for some languages such as turkish and vietnamese that are typologically different from english. we also present old irish as an example of a very small corpus, and see that for this small corpus, generating a language-specific table introduces more overhead than reduction in file size. for non-latin languages, we see that the default tables are not suitable and can substantially increase the file size. we see that it is effective to generate a language-specific table for languages that use a small set of letters, such as arabic or russian. however, for languages with a large number of characters, such as chinese and japanese, the short text compression is not able to reduce the length of the strings. finally, we again see that no approach is effective for a small corpus language, namely bengali, that does not use the latin alphabet. finally, we consider the time to convert the resources into and from the cuaċ format. in ## discussion the cuaċ tooling is developed in rust and in addition is compiled to web assembly @xcite , so it can be run on any platform or language using a tool like wasmer foot_6 . this allows corpora to be used with the teanga library, giving a data science interface for corpora similar to how pandas uses parquet as the underlying data storage. further, it is used as the core of a database engine for teanga corpora which is currently under development. the cuaċ format is ideal for such a format as it allows data to be accessed quickly without overly burdening the database engine with a large amount of data. in particular, the use of text compression technologies is designed to still allow full-text search over the corpus due to the nature of the compression, in a way that would not be possible with the other lossless compression algorithms. as corpora in nlp grow larger and larger more efficient methods for handling such corpora are required. large-scale nlp datasets (e.g., common crawl, wikipedia, or domain-specific corpora) take up terabytes of space. a specialized compression format can significantly reduce storage requirements, making dataset management more cost-effective. loading large text datasets from disk or transferring them over networks can be a bottleneck. a compressed format optimized for fast decompression can accelerate data loading, benefiting both training and inference workflows. ## conclusion in this work, we have presented cuaċ, a format for efficient and compact representation of largescale corpora. cuaċ substantially reduces the storage requirements while maintaining fast processing speeds. our evaluation demonstrated that cuaċ achieves superior compression better or similar to applying lossless compression to human-readable formats and strongly outperforming other binary formats not designed for corpus information. in this way, cuaċ will enhance the computational efficiency of nlp applications and improve processing speeds across a range of nlp and machine learning applications. future work will explore further optimization to target a wider range of corpora including multimodal corpora, when support for multimodal corpora is added to the teanga data model.