P o S to UP o S Conversion and Creation of UP o S Tagged Resources for A ssamese Language (2023)

Abstract: This paper addresses the vital task of transitioning from traditional Part-of-Speech (PoS) tagging to Universal Part-of-Speech (UPoS) tagging within the linguistic context of the Assamese language. The paper outlines a comprehensive methodology for PoS to UPoS conversion and the creation of UPoS tagged resources, bridging the gap between localized linguistic analysis and universal standards. The significance of this work lies in its potential to enhance natural language processing and understanding for the Assamese language, contributing to broader multilingual applications. The paper details the data preparation and creation processes, annotation methods, and evaluation techniques, shedding light on the challenges and opportunities presented in the pursuit of linguistic universality. The contents of this research have implications for improving language technology in the Assamese language and can serve as a model for similar work in other regional languages. Mapping of standard PoS tagset applicable for Indian languages to that of the primary categories of the UPoS tagset is done with respect to the Assamese language lexical behaviour. Conversion of PoS tagged text corpus to UPoS taged corpus using this mapping, and then utilizing a Deep Learning based model trained on such a dataset to create a sizable UPoS tagged corpus, are presented in a structured flow. This paper is a step towards a more standardized, universal understanding of linguistic elements in a diverse and multilingual world.

## introduction assamese language, a rich and diverse member of the indian linguistic landscape, has been a subject of interest in recent years in the field of natural language processing (nlp) and linguistic analysis. an integral component of language analysis is part-of-speech (pos) tagging, a fundamental linguistic process that classifies words into grammatical categories. in the context of assamese, the pos tagging has predominantly relied on the bis (bureau of indian standards) pos tagset (bureau of indian standards.(2021) for indian languages, which has served as a valuable framework for linguistic analysis. the bis pos tagset for indian languages, designed to be applicable across numerous indian languages, has played a pivotal role in linguistic research and language technology development for assamese. it has provided a structured foundation for categorizing and understanding the grammatical elements of this language. however, as language technology continues to evolve, there is an increasing need to align with universal standards such as the universal pos (upos) tagging system @xcite . upos tagging facilitates cross-linguistic comparisons and fosters interoperability in nlp applications. this paper embarks on the task of transitioning from the conventional bis pos tagset to upos tagging for the assamese language. this transition not only reflects the evolving landscape of linguistic analysis but also opens doors to more comprehensive and standardized linguistic resources. it aims to harmonize assamese language analysis with broader linguistic universality while respecting the unique intricacies and characteristics that define assamese. in doing so, it addresses the need for a pos tagging system that balances local relevance and global linguistic standards. the research presented here not only contributes to the advancement of language technology for assamese but also serves as a model for similar efforts in other regional languages. part-of-speech tagging, also known as pos tagging, involves the assignment of grammatical categories (e.g., nouns, verbs, adjectives) to individual words in a text. this process is crucial in understanding the structure and meaning of a sentence. accurate pos tagging is the cornerstone of many nlp applications and linguistic analysis. it enables computers to comprehend the syntactic and semantic properties of a language. the accurate tagging of parts of speech (pos) is a fundamental aspect of natural language processing (nlp) and linguistic analysis. it forms the basis for various language technology applications, including machine translation, sentiment analysis, and information retrieval etc. while pos tagging has been extensively developed and standardized for many major languages, languages with fewer resources, such as assamese, face challenges in this regard. ## challenges in pos tagging for low resource language-assamese indian state of assam and neighboring regions. it boasts a rich linguistic heritage with its own set of grammatical rules, dialects, and nuances. however, the development of linguistic resources for assamese, such as standardized pos tagging, has been relatively limited compared to more widely spoken languages. one of the primary challenges in pos tagging for assamese is the absence of a standardized and widely accepted pos tagset. while major languages like english and hindi have established pos tagsets, assamese lacks a universally recognized set of pos tags. the bis superset, designed to be a common primary tagset across the indian languages, is yet to be customized for assamese, and this has forced the nlp researcher to confine all works limited to the bis superset for indian languages without having the assamese linguistic features embedded. this poses a hindrance to linguistic research, language technology development, and the interoperability of nlp tools for assamese. while pos tagging has seen extensive development and standardization for major languages like english, challenges persist for languages with fewer linguistic resources. low-resourced languages, such as assamese, often face different difficulties: i. lack of standardization: many low-resourced languages lack standardized pos tagsets, making it challenging to develop linguistic resources and nlp tools. ## quest for universality -upos tagging to address the challenges posed by the absence of a standardized and inclusive pos tagset for assamese, this research embarks on the journey of transitioning from traditional pos tagging to universal part-of-speech (upos) tagging. upos tagging follows universal annotation standards that aim to provide a common framework for tagging parts of speech across languages. the transition to upos tagging is driven by the goal of achieving linguistic universality while retaining the unique linguistic characteristics of assamese. the current endeavor seeks to develop upos tagged resources that are compatible with global linguistic standards, making assamese more accessible for linguistic analysis, cross-linguistic research, and the development of language technology applications. this research paper aims to achieve the following objectives: i. pos to upos conversion: develop the mapping of pos tagset to the upos tagset, and conversion of existing pos tagged data in assamese to upos tagging, aligning the language with universal standards. ii. creation of upos tagged resources: create upos tagged linguistic resources, a structured multidomain text corpora annotated with assamese upos tagset, to facilitate linguistic analysis and language technology development. ## background and related works part-of-speech tagging, often abbreviated as pos tagging, is a fundamental task in natural language processing (nlp). it involves the annotation of words in a text with their corresponding grammatical categories, such as nouns, verbs, adjectives, and adverbs. pos tagging serves as a critical component of various language technology applications and linguistic analysis. by assigning grammatical labels to words, computers can discern the syntactic and semantic structure of sentences, enabling them to understand language more effectively. in the context of pos tagging, a tagset is a predefined collection of pos tags, each representing a specific grammatical category. different languages often have their own unique tagsets, which can make it challenging to develop unified nlp tools and linguistic resources. accurate pos tagging is essential for several reasons: i. language understanding: pos tags help in disambiguating word meanings. for example, the word "bank" can refer to a financial institution or the notion of dependency, depending upon on what sense the word is used in the sentence. by assigning different tags (e.g., "nn" for noun and "vb" for verb), the meaning becomes clear in the context. ii. syntactic parsing: pos tagging aids in syntactic parsing, which is crucial for understanding the grammatical structure of sentences. this, in turn, supports various nlp tasks, including machine translation and information retrieval. iii. semantic analysis: by identifying the parts of speech, it becomes easier to perform semantic analysis, extracting information about the roles words play in a sentence. this is invaluable for tasks like sentiment analysis and questionanswering systems. historically, the brown corpus tagset @xcite , the penn treebank pos tagset @xcite , and the ibm watson project simplified pos tagsets ( https://www.ibm.com ) were the fundamental standardizations in the english language tagset evolution. the british national library bnc corpus thereafter, was a huge collection of texts tagged with a rather elaborate larger pos tagset ( http://www.natcorp.ox.ac.uk ). indian language pos tagset evolves with multiple efforts, including indian language machine translation (ilmt) project, iiit hyderabad, and ciil mysore initiatives. a convergence has been seen lateron with unification and standardizing effort by the technology development in indian languages (tdil) program of ministry of communication and information technology, govt. of india (tdil). the evolution of indian languages pos tagset has got stability with a published standard by the bureau of indian standards (bis) through an inclusive and structured methodology (bureau of indian standards.( 2021 )). this standard pos tagset consists of annotations for 11 core categories, and 45 subcategories, applicable as a superset for all indian languages, which is added with language specific subset for few major languages. bis tagset has taken the dominating role in indian languages pos tagging endeavors in recent years, and multiple experiments for almost all indian languages could be traced. @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite . among the major indian languages, north eastern languages are relatively new for nlp research works. in recent years, efforts have been intensified by different researchers and groups. preliminary resources have been created, including corpus, wordnets etc. @xcite ; @xcite ; @xcite ; @xcite . experimentations for various nlp tasks have been undertaken including pos tagging @xcite ; @xcite ; @xcite with conventional as well as machine learning techniques. more complex modelling like machine translation, @xcite information retrieval, neural machine translation @xcite ) also have been observed with limited scopes and performances. ## methodology for conversion methodology employed for the conversion of part-of-speech (pos) tagging to universal partof-speech (upos) tagging in the context of the assamese language is outlined here. it serves as the foundation for the subsequent chapters that describe the creation and evaluation of upos tagged resources. ## creation of upos tagged resources here, the details of the process of creating universal part-of-speech (upos) tagged resources for the assamese language is presented. building upon the methodology outlined in the previous chapter, the focus now shifts to converting existing part-of-speech (pos) tagged data into upos format and leveraging this transformed data to develop invaluable linguistic resources. ## contribution and resource accessibility the transition from pos to upos tagging in assamese and the creation of upos tagged resources are pivotal steps in advancing linguistic analysis and language technology. this work contributes to the ongoing journey of language standardization and holds the promise of broader applications in multilingual systems and linguistic research. by aligning assamese with upos standards, the language becomes more adaptable to multilingual systems, fostering interoperability and collaboration. as we reflect on the accomplishments of this research, we are reminded that the journey towards language standardization is ongoing. the creation of upos tagged resources marks a significant milestone, but there are further horizons to explore. key findings from this research include-successful conversion from pos to upos tagging, aligning assamese with universal linguistic standards, and creation of upos tagged resources. another contribution of the current work has been the rigorous evaluation measures, combining quantitative and qualitative analyses, to assess the accuracy and reliability of the upos tagging system, resulting the resource trusted as high quality. for the benefit of the research community and language technology developers, it is paramount that these upos tagged resources are made widely accessible. resource accessibility is the cornerstone of collaborative research and development. by making these linguistic resources openly available, we foster a community of researchers and developers who can leverage them for various applications. the upos tagged assamese text corpora is now sharable, and available for the research community. the open nature of these resources ensures transparency, fosters innovation, and paves the way for advancements in language technology. the resources created in this endeavor unlock opportunities for assamese to participate in the global linguistic landscape and benefit from cross-linguistic analysis and interoperability. ## conclusion the research embarked on the task of transitioning from traditional part-of-speech (pos) tagging to universal part-of-speech (upos) tagging in assamese, with the primary aim of achieving linguistic universality while retaining the unique linguistic characteristics of the language. it presents a tangible outcome in the form of upos tagged resources, which are invaluable assets for linguistic analysis, research, and the development of language technology. the adoption of upos tagging in assamese signifies a leap forward in linguistic analysis. it aligns assamese with global linguistic standards, making cross-linguistic comparisons and research more accessible. the upos tagged resources, created as a result of this conversion, enhance the quality and depth of linguistic analysis in assamese. researchers and linguists can now explore assamese with a newfound level of universality and precision. this advancement opens doors to deeper insights into the language's structure, grammar, and semantics, and also on the study of linguistic variations across different languages. the impact of upos tagging is not limited to linguistic analysis; it extends to the realm of language technology. with upos tagged resources at their disposal, language technology developers can design more sophisticated and accurate natural language processing applications for assamese. machine translation, sentiment analysis, information retrieval, and various other language-based technologies can benefit from the standardized linguistic data. as we move forward, the path is illuminated by the understanding that linguistic diversity and universality can coexist, enriching our global linguistic tapestry. future research and development in this field are poised to unlock deeper insights into assamese, enhance language technology, and contribute to the wider world of linguistic analysis. this research stands as a testament to the importance of maintaining a balance between linguistic universality and language-specific relevance. the journey towards linguistic universality is ongoing. the creation of upos tagged resources for assamese represents a significant step, but it is not the final destination. the road ahead holds promise and beckons for further exploration.