Towards Enhancing Knowledge Accessibility for Low-Resource I ndian Languages: A Template Based Approach (2024)

Abstract: In today’s digital age, access to knowledge and information is crucial for societal growth. Although widespread resources like Wikipedia exist, there is still a linguistic barrier to breakdown for low-resource languages. In India, millions of individuals still lack access to reliable information from Wikipedia because they are only proficient in their regional language. To address this gap, our work focuses on enhancing the content and digital footprint of multiple Indian languages. The primary objective of our work is to improve knowledge accessibility by generating a substantial volume of high-quality Wikipedia articles in Telugu, a widely spoken language in India with around 95.7 million native speakers. Our work aims to create Wikipedia articles and also ensures that each article meets necessary quality standards such as a minimum word count, inclusion of images for reference, and an infobox. Our work also adheres to the five core principles of Wikipedia. We streamline our article generation process, leveraging NLP techniques such as translation, transliteration, and template generation and incorporating human intervention when necessary. Our contribution is a collection of 8,929 articles in the movie domain, now ready to be published on Telugu Wikipedia.

## introduction wikipedia has been a source of reliable structured information for much time now. although english wikipedia has abundant articles(6.8 million approx.) to cover many domains of knowledge, the scarcity of articles in many indian languages is evident. this highlights the need for a robust knowledge base accessible to native speakers. in response, our research focuses on generating wikipedia articles with consistency and structure, addressing this gap in information accessibility.our chosen language for this work is telugu, belonging to the dravidian language family foot_3 . it is spoken by approximately 100 million people across telangana and andhra pradesh in india. after exploring various category pages on wikipedia and identifying reliable data sources on the internet, we chose the "movies" domain. this decision was based on the popularity of movierelated pages in india and the abundance of reliable sources available for this category. one of such sources is the imdb, which holds a vast collection of movie data across languages. leveraging data scraping techniques, we extract the necessary information from this webpage. we build a dataset comprising 8900 movies with 52 attributes each. using jinja foot_4 template creation, we transform this dataset into an xml dump of wiki articles. each article is manually verified before being published on the global wikipedia platform. our templatedriven method can be re-used to generate articles in any indian language for many simple domains with little complexity with just changing the template. ## related work significant progress has been made in the domain of natural language generation(nlg) within the nlp community, ranging from rule-based methods to statistical methods @xcite to neural-network based techniques @xcite . however, for structured data, nlg techniques such as template-based generation, rule-based approaches, and data-to-text systems are mostly used @xcite . @xcite have utilized a templatebased approach to generate cohesive summaries from laboratory findings to help young doctors. in @xcite ) work, wikidata is be used as a reliable source of information for generating wikipedia articles in hindi. in recent history, sverker johansson, a swedish linguist, worked on creating lsjbot foot_5 , a program that automatically generated wikipedia articles about living organisms and geographical entities (such as rivers, dams, and mountains). we take inspiration from the many works that have utilized template-based text generation approaches and we focus our efforts on generating articles in telugu wikipedia, we employ a similar approach that would help in maintaining the consistency and reliability of wikipedia articles. our streamlined process is a comprehensive, endto-end approach for generating articles across any language and domain, as long as the template used remains simple yet consistent. this flexibility and consistency make our methodology unique and sets it apart from other approaches. ## methodology our automated article generation process operates by populating a predefined text template with various attributes. the detailed methodology is illustrated in figure 1 . ## extending our work expanding our research scope, we have extended our efforts in generating articles across various domains of knowledge. our contributions are listed as follows: ## results & conclusion to assess the accuracy of the translated text, we employed bertscore @xcite , a metric that evaluates both the semantic meaning and syntactic structure of sentences. given the significant risk of meaning loss when translating to or from indian languages, we prioritized measuring semantic similarity to evaluate the quality of machinetranslated text.we achieved an average bertscore of 0.79 for articles generated using a reference of their corresponding articles from global wikipedia. the quality & human-like characteristics of the text generated were evaluated by 4 annotators who rated each article within a range of 1-5. to demonstrate consistency & inter-annotator agreement, we also report fleiss' kappa score to be at 0.82. as of early 2023, telugu wikipedia hosts around 95,599 articles 18 . our work has produced approximately close to 25000 articles ready for publication in global telugu wikipedia across multiple domains. this makes our contribution to the telugu community further significant. furthermore, we believe this work serves as motivation for the research community to take initiative in developing and enhancing natural language resources for various low-resource languages. such efforts are crucial in bridging existing gaps and making a meaningful impact on society. in conclusion, our paper has demonstrated how employing data scraping techniques coupled with template generation can produce articles in telugu wikipedia, thereby significantly enhancing the accessibility of knowledge for millions of native telugu speakers. we show how nlp can be leveraged to bridge linguistic and knowledge gaps across low-resource languages, ultimately creating a more inclusive society. we remain committed to our goal of enhancing the digital footprint of many more indian languages.