Automatic Extraction of the R omanian Academic Word List: Data and Methods (2023)

Abstract: This paper presents the methodology and data used for the automatic extraction of the Romanian Academic Word List (Ro-AWL). Academic Word Lists are useful in both L2 and L1 teaching contexts. For the Romanian language, no such resource exists so far. Ro-AWL has been generated by combining methods from corpus and computational linguistics with L2 academic writing approaches. We use two types of data: (a) existing data, such as the Romanian Frequency List based on the ROMBAC corpus, and (b) self-compiled data, such as the expert academic writing corpus EXPRES. For constructing the academic word list, we follow the methodology for building the Academic Vocabulary List for the English language. The distribution of Ro-AWL features (general distribution, POS distribution) into four disciplinary datasets is in line with previous research. Ro-AWL is freely available and can be used for teaching, research and NLP applications.

## introduction since academic language differs from everyday social language and is an essential acquisition target in current education, extracting salient features contributes to linguistic, register, genre and disciplinary feature identification that can benefit students, teachers and educational app developers alike. compiling an academic word list (awl) is an effective solution to support both language teaching and nlp tasks. from the didactic perspective, awls reflecting either the l1 (i.e. mother tongue) or the l2 (i.e. foreign language) academic vocabulary can be used to offer linguistic support to novice academic writers in the form of disciplinespecific and general lexical prompts. teachers of all disciplines can integrate awls into teaching materials to help students write better (see, for example, @xcite ). nlp studies can exploit awl datasets on topics such as text classification @xcite and topic modelling @xcite . for example, field-specific academic lists can be used to automatically classify texts into disciplinary areas. the same can be applied for the automatic distribution of texts in academic versus non-academic batches. in machine learning methods for language modelling tasks, awls are essential in training models to generate accurate academic writing samples. by combining nlp tasks with linguistic approaches in relation to awls, important advances can be achieved in the frame of lexical and syntactic analyses that evaluate the use of collocations and phraseology specific to the academic varieties. for the romanian language, there have been few attempts to extract a valid romanian word list @xcite and only one study has extracted and analysed multiword units using academic writing corpora @xcite . in recent years, researchers have worked to create several academic writing corpora. expres -corpus of expert writing in romanian and english @xcite is one of them. it is the only bilingual multidisciplinary corpus capturing the romanian academic writing context. by combining datasets representing the romanian frequency list @xcite based on the rombac corpus @xcite , and expres disciplinary datasets @xcite , we were able to generate an empirically based romanian academic word list. ro-awl is made publicly available foot_0 and can be used for teaching, text classification and language modelling. ## related work most academic vocabulary lists have been developed in the context of english for academic pur-poses (eap). on the whole, two categories of lists exist. one list type aims to identify academic words commonly used in eap across disciplines, which students could be made aware of. the studies aiming to provide cross-disciplinary academic word lists usually use large corpora containing expert academic writing from various disciplines. the widely used lists of this type are the academic word list (awl) @xcite and the academic vocabulary list (avl) @xcite . the second type of list seeks to identify discipline or field-specific words worth teaching. various specialised lists have been developed for fields such as veterinary medicine @xcite or nursing @xcite . while there is a growing interest in building cross-disciplinary academic word lists for languages other than english, these academic word lists remain few. see, for example studies conducted for french @xcite , persian @xcite , portuguese @xcite , swedish @xcite , and norwegian @xcite ). an explanation for this scarcity might be that academic language data sets are rare and often not freely available due to copyright. this can be especially true for lowresource languages, such as romanian. access to a representative corpus is crucial, as the validity and reliability of an academic word list highly depend on the quality of the data set. apart from the limited availability of academic writing corpora, an additional challenge may be that there is no standard procedure for extracting academic word lists. scholars are still exploring and testing various methodologies. for example, some studies build on the methods used for the awl or the avl @xcite . one study uses the translated version of the avl in portuguese as a starting point for its investigation @xcite . another study proposes a new word list extraction method different from previous ones @xcite . in the case of romanian, no previous studies have compiled specialised or general academic word lists. although in the last 10-15 years, several research institutions and projects have been involved in developing corpus resources in romanian, relatively few have focused exclusively on general academic writing. some of the most significant corpora recently compiled, such as rombac (romanian balanced annotated corpus, see ion et al. (2012)), with more than 30 million words, corola (corpus of contemporary romanian language, see mititelu et al. (2014)), or the balanced romanian corpus (brc, see midrigan-ciochina et al. (2020)) cover only few disciplines or subsets: 5 sections for rombac (journalism, literature, medical texts, legal texts, biographies), uneven and unfiltered distribution of resources in corola (the collection of academic writing texts is based on agreements with publishing houses and journals, without filtering of the content on quality criteria) and brc (literary text samples, research articles, news, spoken data). the rombac corpus (excluding the medical subcorpus) was already used to develop the romanian word list (rwl, see szabo (2015)), targeted at romanian l2 learners (e.g. from the hungarian minority in romania). the list is a general list of words, not focused on academic language. as far as discipline-specific corpora are concerned, smaller corpora such as simonero (medical corpus, mitrofan et al. (2019)), bioro (mitrofan and tufis ¸, 2018), parseme-ro (news articles), legalnero (legal, pȃis , et al. (2021)), marcell (legal, multilingual, see váradi et al. (2020)), curlicat (multilingual, containing several domains: economics, education, health, sciences, etc., see váradi et al. ( 2022 )) have been compiled. however, apart from compiling the datasets and conducting a series of descriptive studies, no special attention is given to the lexical level. in this context, the expres corpus (corpus of expert writing in romanian and english) is the first corpus of discipline-specific academic writing in the romanian context (academic writing in romanian l1 and academic writing in english l2 produced by romanians) @xcite . covering four disciplines -linguistics, economics, political sciences, information technology -, the romanian subset contains 200 open-access research articles from each domain, published in the past 5-10 years in peer-reviewed journals (see @xcite ). the rigorous selection criteria @xcite contribute to the representativeness of the corpus, making it a suitable candidate for testing a possible romanian word list and narrowing it down to an academic word list. furthermore, the expres corpus is the first romanian expert academic corpus available on an open-access query platform. unlike other romanian corpora, which offer limited access to third parties and poor resources for downloading search results or statistics, the expres corpus support platform has been implemented as a cross-platform distributed web application @xcite . ## data this work uses two different corpora: the academic corpus expres and the romanian academic word list @xcite compiled from the general corpus rombac. the romanian language sub-corpus of expres 2 @xcite consists of 800 research articles, 200 articles for each of the four fields: linguistics (lg), economics (ec), information technology (it) and political sciences (ps). the articles from the corpus were manually processed to preserve the anonymity of the authors (e.g., the name of the authors were replaced with author name) and the beginning and end of the title, abstract and sections are annotated with corresponding xml tags (e.g., <title>, </@xmath0. data preprocessing. the romanian academic word list, with words from the rombac corpus, provides the lowercase lemma for each word in the corpus and its frequency. therefore, no preprocessing step was done on this data. even if we use the word frequencies from the romanian academic word list, we will refer to this data as the rombac corpus, given that the list contains all the words from rombac. the expres corpus is organised in multiple .txt files, one for each article from the four domains lg, it, ps, and ec. for each document, we removed specific tags used for article anonymisation, such as journal title, au-thor name, etc., and the specific xml tags used to mark the beginning or end of the title (<title>, </@xmath1, abstract (<abs int>, </abs @xmath2, or different sections of the article (<introd>, </@xmath3, etc. the ex-pres corpus statistics regarding the words and word types in the corpora are shown in building the academic word list. for constructing the academic word list, we follow the methodology for building the academic vocabulary list for the english language @xcite , comprising different frequency measures for lemmas. we chose to use the methodology from gardner and davies (2014) instead of the procedure from @xcite because the former method provides an academic list with almost twice the latter's coverage. the approach from @xcite is based on word families, while the method from gardner and davies (2014) relies on lemmas. a word family is represented by the base word from which other words are derived with suffixes and prefixes. this can be problematic in the case of academic words, as the base of a word family can be an academic word, but their derivations might not be academic @xcite . the methodology is based on four measures: ratio, range, dispersion and discipline measure. the ratio is used to exclude general high-frequency words from the corpus, while the other three metrics exclude technical or discipline-specific terms. we further expand on each metric below. ratio. similar to @xcite , general high-frequency words (in our case, lemmas) are removed from the academic word list. the ratio is computed to keep in the list words with a higher frequency in the academic corpus than in the general non-academic corpus. we computed the normalised frequency per million words of each word in the two corpora, expres and rombac. the ratio is calculated by dividing the academic corpus's normalised frequency by the general corpus's normalised frequency for each word. gardner and davies (2014) use the frequency ratio of 1.5 in their method, but mention that the measure is not a gold standard. we experimented with values between 1.2 and 2.0 for ratio, and, in our case, the 1.2 ratio was a suitable value, to not have important academic words excluded from our list, such as "metodologic" (en: "methodological"), "clasificare" (en: "classification"), "activitate" (en: "activity"), "distinge" (en: "distinguish"), "sugera" (en: "suggest"), which are found in the original avl for the english language. range. the range measure allows for selecting words that only occur in multiple disciplines, and filtering out discipline-specific words. @xcite proposed that a word should have at least 20% of the expected frequency in 78% of the sub-corpora (i.e. 7 out of 9 domains). for computing the expected frequency, we first calculated each word's frequency in relation to the corpus by dividing the word count by the total number of words in expres. afterwards, the frequency in relation to the corpus is multiplied by the number of words in a given sub-corpora to get the expected frequency in each sub-corpora. in our case, expres has only four domains, and we chose words that had at least 20% of the expected frequency in at least three out of four fields, corresponding to 75% of sub-corpora. dispersion. the measure used for dispersion is julliand's d @xcite , which shows how evenly a word appears in a corpus. the formula is as follows: the values of dispersion range from 0.01 (corresponding to words that appear in a small part of the corpus) to 1.00 (meaning that a word is spread evenly in the corpus). unlike the range measure, which estimates if a word has the expected frequency in the four domains, the dispersion measure ensures that a given word is distributed uniformly in the four sub-corpora. gardner and davies (2014) chose 0.80 dispersion, while, in other works, the dispersion measure varies between 0.30 to 0.60 @xcite @xcite . we decided to use a dispersion value of 0.50 in our work. discipline measure. this measure is used for filtering out words with a very high frequency in a given domain, which may be technical disciplinespecific words. @xcite proposed that a word cannot have more than three times the expected frequency in any domain. following a similar approach, we remove words with more than three times the expected frequency in any of the four domains. as an additional measure, we excluded words with low frequency in the academic corpus, because the metrics mentioned above do not filter them out. inspired by @xcite and @xcite , we remove from the final academic list the words that have a minimum frequency of 28.57 per million words, corresponding to the minimum frequency originally proposed by @xcite of 100 times in the 3.5 million words corpus they used in their work. we also performed a manual analysis of the academic word list and removed the noise, such as proper nouns (e.g., "bucures , ti", en: "bucharest"), some numerals, and some words that were not academic and that were not filtered out by the measures mentioned above. ## results the final romanian academic word list consists of 673 lemmas with their corresponding part of speech tags. the list comprises 332 nouns, 167 adjectives, 157 verbs, and 17 adverbs. we automatically translated into romanian the words from the avl developed for english @xcite that contains 3015 words. we found that 381 words in our list are in the original avl. there are some cases of academic words found in our romanian academic list and in the avl for english for which the automatic translation fails to provide the correct match. for example, the noun "adoption" from avl was translated as "adopt , ie", which is not in the ro-awl, but the word "adoptare" is an academic word from ro-awl with the same meaning. the fact that we found more than half of the ro-awl in the original avl, even though in some cases the translation fails to capture the correct meaning of the words, makes us confident that the measures used are reliable for building a romanian academic word list. in line with previous works @xcite @xcite , to demonstrate the viability of the newly developed academic word list, we measured the coverage of the ro-awl in two corpora: the academic corpus expres and in the general corpora rombac. the academic words from our list cover 15.25% of the expres corpus and 6.73% of rombac. in line with the english avl results, ro-awl has a higher coverage in the academic corpus and a lower coverage in the general corpus. regarding the coverage in expres, we show the coverage of academic words categorised by their part-of-speech tags in ## discussion a first observation concerns the different coverages of ro-awl in the expres corpus (see although in english language academic writing "the dichotomy of soft and hard sciences is rather fluid and as such insignificant" @xcite , discipline-specific peer-review practice in the ro-manian setting seems to influence the academic writing style. this is particularly visible in the ex-pres subset of political sciences and linguistics. romanian academic writing in ssh seems rather unfocused, descriptive and rich in rhetorical structures. in contrast, research articles in economics and information technology contain many statistics, tables, and formulas, making the writing in the discipline less descriptive. secondly, although our extraction measures were successful in filtering most of the technical vocabulary, small amount of technical language remains in the ro-awl (terms such as "dauna", en: "damage" -in contexts related to insurances; "institutional", "security", "electronic" etc.). nevertheless, the majority of the ro-awl components are discipline neutral, thus contributing to academic discourse cohesion and coherence. thirdly, a technical challenge regarding the functionality and accuracy of the romanian pos tagger should be mentioned. an overview of the assigned tags revealed the difficulty of the tagger to distinguish between adjectives and adverbs (for instance: "important", "social", "european" were assigned as adverbs, but the contexts prove their prevalent use as adjectives). it also confused past participles ending with "-t" (e.g. "accentuat", en: "emphasised". this technical difficulty can be observed in a technical advantage of the romanian pos tagger, however, is its capacity to recognise nouns with a definite article while being a part of prepositional phrases ("în pofida", en: "despite", "în jurul", en: "around"). this also explains the increased percentage levels of nouns, adverbs and verbs and the lower values for adjectives (see figure 1 ). despite some of the technical challenges, the extraction of the romanian awl using the ex-pres corpus resulted in successfully identifying the recurrent discourse conventions used by romanian researchers. during the process and alongside the extraction procedure per se, translating the academic vocabulary list (avl) @xcite was a helpful procedure, as it is well accepted that academic writing, irrespective of the language, contains a large number of words of greek and latin origin (see e.g., @xcite ; @xcite ). ## conclusions and future work this study reports the extraction of the first romanian academic word list (ro-awl), which can be used to check the degree of academic vocabulary coverage in discipline-specific and general language samples. ro-awl consists of 673 lemmas, distributed among the main part-of-speech categories (nouns, verbs, adverbs, adjectives). our methodology adopted measures used for the academic vocabulary list for the english language, such as ratio, range, dispersion and discipline measures. the percentages calculated by testing ro-awl on the disciplinary datasets in the expres corpus @xcite , indicate a lower coverage for linguistics (11.82%) and political sciences (13.17%) and a higher coverage for information technology (17.03%) and economics (17.75%). also, the academic vocabulary coverage in rombac, a general language reference corpus, is 6.73%, while the coverage is much higher (15.25%) in expres, an expert academic writing corpus. this aligns with previous research, since ro-awl coverage is similar to thresholds for academic vocabulary @xcite . despite several computation constraints (e.g. romanian pos tagger not being able to distinguish between adjectives and adverbs), our study provides important insights into the academic writing vocabulary in romanian by proposing a validated romanian academic word list. our findings also have pedagogical implications, as the list can be used to support academic writing teaching activities and nlp tasks focusing on romanian. for example, the ro-awl can be paired up with the freely available expres corpus platform to develop corpus-assisted learning activities commonly known as data-driven learning (ddl) (see e.g., @xcite ). however, even if the coverage test results in the expres are encouraging, further research is needed to test the validity of the ro-awl on corpora containing academic writing from more disciplines. future work can be conducted in at least two directions: refining the lists from a contrastive perspective, by developing a disciplinespecific awl, or, on the contrary, by searching for highly frequent academic words present in an extended corpus containing more disciplines.