calaman C y: A T agalog Natural Language Processing Toolkit (2023)

Abstract: We introduce calamanCy, an open-source toolkit for constructing natural language processing (NLP) pipelines for Tagalog. It is built on top of spaCy, enabling easy experimentation and integration with other frameworks. calamanCy addresses the development gap by providing a consistent API for building NLP applications and offering general-purpose multitask models with out-of-the-box support for dependency parsing, parts-of-speech (POS) tagging, and named entity recognition (NER). calamanCy aims to accelerate the progress of Tagalog NLP by consolidating disjointed resources in a unified framework.The calamanCy toolkit is available on GitHub: https://github.com/ljvmiranda921/calamanCy.

## introduction tagalog is a low-resource language from the austronesian family, with over 28 million speakers in the philippines @xcite . despite its speaker population, few resources exist for the language @xcite . for example, universal dependencies (ud) treebanks for tagalog are tiny ( 20k words) @xcite , while domain-specific corpora are sparse @xcite . in addition, tagalog language models (lms) @xcite are few, while most multilingual lms @xcite underrepresent the language @xcite . thus, consolidating these disjointed resources in a coherent framework is still an open problem. the lack of such framework hampers model development, experimental workflows, and the overall advancement of tagalog nlp. to address this problem, we introduce cala-mancy, foot_0 an open-source toolkit for tagalog nlp. it is built on top of spacy @xcite and offers end-to-end pipelines for nlp tasks such as dependency parsing, parts-of-speech (pos) tagging, and named entity recognition (ner). cala-mancy also provides general-purpose pipelines in three different sizes to fit any performance or accuracy requirements. this work has two main contributions: (1) an open-source toolkit with outof-the box support for common nlp tasks, and (2) comprehensive evaluations on several tagalog benchmarks. evaluations on tagalog nlp tasks structured evaluations for core nlp tasks, such as dependency parsing, pos tagging, and ner, are meager. however, we have access to a reasonable amount of data to conduct comprehensive benchmarks. for example, tlunified (cruz and cheng, 2022) is a pretraining corpus that combines news reports @xcite , a preprocessed version of com-moncrawl @xcite , and several other datasets. however, it was evaluated on domainspecific corpora that may not easily transfer to more general tasks. in addition, tagalog has two universal dependencies (ud) treebanks, tagalog reference grammar (trg) @xcite and ugnayan (aquino and de leon, 2020), both with pos tags and relational structures for parsing grammar. this paper will fill the evaluation gap by providing structured benchmarks on these core tasks. ## implementation the best way to use calamancy is through its trained pipelines. after installing the library, users can access the models in a few lines of code: import calamancy as cl @xmath0.load("tl_calamancy_md-0.1.0") @xmath1.") here, the variable nlp is a spacy processing pipeline 2 that contains trained components for pos tagging, dependency parsing, and ner. applying this pipeline to a text will produce a doc object with various linguistic features. calamancy offers three pipelines of varying capacity: two static word vector-based models (md, lg), and one transformerbased model (trf). we will discuss how we developed these pipelines in the following section. ## evaluation architectures we used spacy's built-in architectures for each component in the calamancy pipeline. the token-to-vector layer uses the multihash embedding trick @xcite to reduce the representation size. for the parser and named entity recognizer, we used a transition-based parser that maps text representations into a series of state transitions. as for the text categorizer, we utilized an ensemble of a bag-of-words model and a feed-forward network. ## discussion the calamancy pipelines are competitive across all core nlp tasks while maintaining a smaller compute footprint. as shown in the text categorization and ner results, users with low compute budgets can attain similar performance to multilingual lms by using medium-or large-sized cala-mancy models. the transformer-based calamancy pipeline is the best option for users who prioritize accuracy. however, we were surprised that most alternative approaches perform better in dependency parsing. we attribute this performance to the added strength of multilingual and crosslingual information, which we don't have when training solely on a smaller treebank. we plan to improve dependency parsing performance by building a larger treebank within the universal dependencies framework. for practical applications, we recommend users to start with a medium-or largesized calamancy model before trying out gpuintensive pipelines. only then can they switch to a transformer-based pipeline to get accuracy gains. ## conclusion in this paper, we introduced calamancy, a natural language processing toolkit for tagalog. our work has two main contributions: (1) an opensource toolkit containing general-purpose multitask pipelines with out-of-the-box support for common nlp tasks, and (2) comprehensive benchmarks that compare against alternative approaches, such as cross-lingual or multilingual finetuning. we hope that calamancy is a step forward to improving the state of tagalog nlp. as a low-resource language, consolidating resources into a unified framework is crucial to advance research and improve collaboration. in the future, we plan to create a more fine-grained ner benchmark corpus and extend calamancy to natural language understanding (nlu) tasks. finally, the project is hosted on github ( https://github.com/ljvmiranda921/ calamancy ) and we are happy to receive community feedback and contributions.