Automatic H adith Segmentation using PPM Compression (2020)

Abstract: In this paper we explore the use of Prediction by partial matching (PPM) compression based to segment Hadith into its two main components (Isnad and Matan). The experiments utilized the PPMD variant of the PPM, showing that PPMD is effective in Hadith segmentation. It was also tested on Hadith corpora of different structures. In the first experiment we used the non- authentic Hadith (NAH) corpus for train- ing models and testing, and in the second experiment we used the NAH corpus for training models and the Leeds University and King Saud University (LK) Hadith cor- pus for testing PPMD segmenter. PPMD of order 7 achieved an accuracy of 92.76% and 90.10% in the first and second experiments, respectively.

## introduction automated text segmentation is the task of building a tool that can automatically identify sentence boundaries in a given text and divide them into their components. the need to convert unstructured text into a structured format is especially important when dealing with unstructured text such as web text or old documents. one of the most important types of old holy islamic texts in the arabic language is hadith. hadith-the second source of islam-refers to any action, saying, order, or silent approval of the holy prophet muhammad that was delivered through a chain of narrators. each hadith has an isnad-the chain of narrators-and a matan-the act of the prophet muhammad. figure 1 shows an example of hadith. while most ordinances of islam are mentioned in the quran in general terms, detailed and vivid explanations are often provided in the hadith. this gives the hadith importance among muslims. for example, prayer, ' ‫ا‬ ‫ل‬ ‫ص‬ ‫ل‬ ‫ا‬ ‫ة‬ ', is mentioned in the quran, while the hadith specifies what muslims should do and say; the hadith explains the time for each prayer and what muslims should do before and after the prayer. in contrast to the quran, some hadiths, which have been handed down over centuries, have been corrupted by incompetent narrators who transferred them incorrectly. hadith scholars have classified these as non-authentic hadiths. automatic hadith segmentation of isnad and matan can help hadith researchers, some of whom focus on an isnad with the aim of studying narrators' reliability, the links between them, or how a specific hadith has been transferred through the ages, sometimes generating a graphical visualization to represent this @xcite . other research concentrates on matan to classify hadiths into topics @xcite . teahan (2000) used prediction by partial matching (ppm) to solve several nlp problems, such as text classification and segmentation. @xcite and @xcite pointed out that us-ing a character-based compression scheme for tasks such as detecting code-switching and gender/authorship categorization is more effective than word-based machine learning approaches. many current hadith studies use a word-based method to segment hadith from the six canonical hadith books, but the method of this paper uses a character-based ppm compression method to automatically segment the isnad and matan. our goals are to evaluate ppm segmenter on (1) unstructured hadith text from lesser-known hadith books and (2) wellstructured hadith text from the six canonical hadith books. this paper explains the data sets chosen for our experiments and outlines the experiments performed on arabic hadith text to evaluate the ppm compression method. finally, we draw conclusions and suggest future work based on this study. ## related work there have been relatively few studies on the segmentation of hadith into isnad and matan. one study was carried out by @xcite , who developed a finite state transducers-based system to detect the different parts of a hadith, such as title-bab, num hadith, sanad 'isnad', and matn 'matan'. the disadvantage of this system is that it was built to depend on the hadith structure in sahih al-bukhari book (the most trusted hadith book), which cannot be used for other hadith books. figure 2 shows the hadith structure in the sahih al-bukhari book. this system achieved a precision of 0.44 for isnad extraction and 0.61 for matan extraction. '. these books contain a mixture of authentic and non-authentic hadiths and do not have a clear structure, which makes the segmentation task more complex. also, characterbased text compression methods have not been used in previous hadith segmentation studies. our work seeks to fill these gaps in research. ## data collection for this study, we selected a non-authentic hadith (nah) corpus built by @xcite as a training and testing set. the main feature of this corpus is that it contains 452,624 words from different lesser-known hadith books. it also included several annotated hadith books, which help to determine the switch points between the isnad and the matan, and thus provide a ground truth. these books were downloaded from hadith websites such as islamweb.net and almeshkat.net as word files and converted to csv files. some of these books have both hadiths (authentic and nah), while others only contain nah. the annotating process was done to determine eight primary features for each hadith in this corpus. these are no., full hadith, the isnad, the matan, the authors comments, the hadith type, authenticity and topic. a description of the nah corpus features is shown in ## ppm compression-based segmenter the ppm text compression algorithm is a character-based model that predicts an upcoming symbol by using the previous symbols with a fixed context. every possible upcoming symbol is assigned a probability based on the frequency of previous occurrences. if a symbol has not been seen before in a particular context, the method will 'escape' to another lower-order context to predict the symbol. this is called the escape method and is used to combine the predictions of all character contexts @xcite . different variants of ppm have been produced in order to give better compression results, such as ppmc @xcite and ppmd @xcite . howard (1993), who invented ppmd, showed that ppmd gives better results for text compression than ppmc. equation 1 defines how ppmd estimates the probability p for the next symbol φ: ## evaluation experiments two experiments were performed as part of the evaluation of the compression-based method (provided by tawa) (teahan, 2018) to automatically separate hadith into two components, isnad and matan. in the first experiment we used the nah corpus for training models and testing, and in the second experiment we used the nah corpus for training models and the leeds university and king saud university (lk) hadith corpus, built by @xcite , for testing ppm segmenter. ## conclusion and future work in this paper, we evaluated ppm compressionbased method for automatic segmentation of arabic hadith. the experiments showed that ppmd is effective in segmenting hadith into its two main components (isnad and matan), having been tested on hadith corpora (nah and lk) that have different structures. the main innovation in these experiments is their use of a character-based text compression method to segment hadith. for training isnad and matan models we used the first book in the nah corpus. in the first experiment, we used the third book in the nah corpus, which lacks a clear structure, as a testing set. we found that ppmd of order 7 obtained a higher accuracy (92.76%) than other orders. in the second experiment, we aimed to evaluate ppmd segmentation on a different hadith corpus so we used the sahih al-bukhari book of the lk hadith corpus for testing purposes, which produced an accuracy of 90.10%. the first experiment showed that the hadith's type is not in the same place between the training and testing set, which leads to some confusion between isnad and matan. possible ways to reduce this confusion that could be undertaken in future work may be to ( 1