Resources in Underrepresented Languages: Building a Representative R omanian Corpus (2020)

Abstract: The effort in the field of Linguistics to develop theories that aim to explain language-dependent effects on language processing is greatly facilitated by the availability of reliable resources representing different languages. This project presents a detailed description of the process of creating a large and representative corpus in Romanian – a relatively under-resourced language with unique structural and typological characteristics, that can be used as a reliable language resource for linguistic studies. The decisions that have guided the construction of the corpus, including the type of corpus, its size and component resource files are discussed. Issues related to data collection, data organization and storage, as well as characteristics of the data included in the corpus are described. Currently, the corpus has approximately 5,500,000 tokens originating from written text and 100,000 tokens of spoken language. it includes language samples that represent a wide variety of registers (i.e. written language - 16 registers and 5 registers of spoken language), as well as different authors and speakers

## metadata storing information that describes the properties of the linguistic resource and variables containing information about every individual file is very essential for both accessing information of interest for specialized studies (e.g. psycholinguistic studies looking at gender differences and language use) but also indexing and searching the corpus. for the creation and storing of the metadata for the romanian corpus, we are using the arbil tool, developed at max planck institute for psycholinguistics, nijmegen, the netherlands @xcite , available at ( http://tla.mpi.nl/tools/tla-tools/arbil/ ). this tool allows the usage of standardized profiles and schemas for both spoken and written language resources, cmdi (component metadata infrastructure) framework. we are continually editing these as new data is added. ## corpus annotation aims we are in the process of starting to annotate the corpus, using the text preprocessing module ttl @xcite , developed in perl. the module is available at http://ws.racai.ro/ttlws.wsdl and offers a sentence splitter, tokenizer, tagger, lemmatizer, and a chunker procedures for romanian @xcite . we also envision parse tree annotation, semantic labeling and affix annotation in future steps. ## conclusions and further directions computerized corpora in the different languages spoken around the world has important implication for linguistic theory. although not all questions can be answered by studying language as represented in corpora, they can greatly expand our understanding of language and its complex facets. evidence from corpora allows researchers to document natural language, study language typology and the effects of language-dependent factors on language processing. it has also important applications in the development of natural language processing systems. thus, building resources in languages that are understudied is crucial. this project's goal was to enable linguists to study romanian and its unique grammatical characteristics, by creating a reliable repository of text that represents a wide variety of registers of this language. although the corpus presented in this project is one of the largest available for romanian, we want to continue enlarging this resource, specifically the spoken data; compiling large samples of spoken data is not an easy task. accurately transcribing spoken language can be time consuming and expensive; it also requires native speaker knowledge. however, spoken language is by far the mode in which language is most frequently used, and it has its distinct characteristics. in written language, the authors tend to clean the text in somewhat unnatural ways: the ideas are well formed, and well organized. while when producing language, the speakers tend to make many repetitions (i.e. same words are said twice, or even more than two times), utter unfinished thoughts @xcite , and produce various speech errors. spoken language tends to have generally shorter sentences, with words that may not appear at all in written language (i.e. aha is used a lot in conversational romanian to show 'agreement' or 'approval', while it is not a word that appears in written text). language dialects are yet another reason why spoken language may have different characteristics than written language. @xcite , found that frequencies that are calculated from movies' subtitles and television were better than the ones found on written text. thus, a further direction for the development of the corpus is adding and including an equal amount of spoken data along with the written samples. another step is adding text samples representing different time periods in the history of romanian language development (i.e. current romanian orthography is little more than a century old @xcite , thus writings representing the original language forms may be found particularly in print). further work is yet needed in order to make the resource as easy to use as possible; the current corpus is at its initial stage of annotation. many corpora used in various linguistic studies are still unannotated; however, working with annotated corpora makes the process of information retrieval easier and faster. one further steps of the present project is annotating the text and transcribed spoken language for various types of grammatical information. some of the target annotations are word-class annotation as well as morphological annotation, including affix annotation for different types of affixes of the language; in addition, parse tree annotation and semantic labeling are also considered. the brc project, aims to create a new, reliable resource in romanian -a language that has unique structural characteristics, still remains understudied.