Guidelines for Fine-grained Sentence-level A rabic Readability Annotation (2025)

Abstract: This paper presents the annotation guidelines of the Balanced Arabic Readability Evaluation Corpus (BAREC), a large-scale resource for fine-grained sentence-level readability assessment in Arabic. BAREC includes 69,441 sentences (1M+ words) labeled across 19 levels, from kindergarten to postgraduate. Based on the Taha/Arabi21 framework, the guidelines were refined through iterative training with native Arabic-speaking educators. We highlight key linguistic, pedagogical, and cognitive factors in determining readability and report high inter-annotator agreement: Quadratic Weighted Kappa 81.8% (substantial/excellent agreement) in the last annotation phase. We also benchmark automatic readability models across multiple classification granularities (19-, 7-, 5-, and 3-level). The corpus and guidelines are publicly available: http://barec.camel-lab.com.

## introduction text readability plays a crucial role in comprehension, retention, reading speed, and engagement @xcite . when texts exceed a reader's ability, they can lead to frustration and disengagement @xcite . readability is shaped by both the content and presentation @xcite . in educational settings, readability leveling is widely used to align texts with students' reading abilities, promoting independent and more effective learning @xcite . fine-grained readability systems, like fountas and pinnell's 27-level scale in english @xcite , and taha's 19-level arabic system @xcite , guide progression from early readers to adult fluency. these levels support instructional goals and can be mapped to broader categories for practical use in nlp. ## 8th next, §2 reviews related work, §3 outlines the annotation framework, §4 covers data selection, and §5 discusses evaluation results. authors project metric levels unit size content al-khalifa and al-ajlan (2010) arability readability 3 document 150 school textbooks forsyth (2014) dli corpus ilr 5 (3) document 179 l2 learner kilgarriff et al. (2014) kelly cefr 6 word 9,000 most frequent taha-thomure (2017) taha/arabi21 readability 19 document 9,000 children's books al khalil et al. (2020) samer lexicon readability 5 word 40,000 general vocab habash and palfreyman (2022) zaebuc cefr 6 document 214 prompted essays naous et al. (2024) readme++ cefr 6 sentence 1,945 multi-domain soliman and familiar (2024) arabic vocab profile cefr 2 word 1,200 l2 learner (a1, a2) el-haj et al. (2024) dares grade level 12 sentence 13,335 school textbooks alhafni et al. (2024) samer corpus readability 3 word 159,265 literature bashendy et al. (2024) qaes aes 7×5 document 195 argumentative essays our work barec readability 19 (7-5-3) sentence 69,441 multi-domain ## conclusions and future work this paper presented the annotation guidelines of the balanced arabic readability evaluation corpus (barec), a large-scale, finely annotated dataset for assessing arabic text readability across 19 levels. with over 69k sentences and 1 million words, it is, to our knowledge, the largest arabic readability corpus, covering diverse genres, topics, and audiences. we report high inter-annotator agreement (qwk 81.8% in phase 2) that ensures reliable annotations. benchmark results across multiple classification granularities (19, 7, 5, and 3 levels) demonstrate both the difficulty and feasibility of automated arabic readability prediction. looking ahead, we plan to expand the corpus by increasing its size and diversity to include more genres and topics. we also aim to add annotations for vocabulary leveling and syntactic treebanks to study the effect of vocabulary and syntax on readability. future work will analyze readability variations across genres and topics. additionally, we intend to integrate our tools into a system that assists children's story writers in targeting specific reading levels. the barec dataset, its annotation guidelines, and benchmark results, are publicly available to support future research and educational applications in arabic readability assessment. university year 1-2 superior high • scientific and heritage vocabulary not in use today, but familiar to a novice specialist university year 3-4 distinguished • scientific and heritage vocabulary not in use today, but familiar to a specialist specialist distinguished • scientific and heritage vocabulary not in use today, but familiar to the advanced researcher specialist this tag is used when there is difficulty in assessing the level. it is preferred to use this tag so that the team can find a solution (for example, by adjusting the criteria or adding explanatory details). generally, we use this tag for sentences containing: