Establishing a Baseline for A rabic Patents Classification: A Comparison of Twelve Approaches (2022)

Abstract: Nowadays, the number of patent applications is constantly growing and there is an economical interest on developing accurate and fast models to automate their classification task. In this paper, we introduce the first public Arabic patent dataset called ArPatent and experiment with twelve classification approaches to develop a baseline for Arabic patents classification. To achieve the goal of finding the best baseline for classifying Arabic patents, different machine learning, pre-trained language models as well as ensemble approaches were conducted. From the obtained results, we can observe that the best performing model for classifying Arabic patents was ARBERT with F1 of 66.53%, while the ensemble approach of the best three performing language models, namely: ARBERT, CAMeL-MSA, and QARiB, achieved the second best F1 score, i.e., 64.52%.

## introduction over the past few years, there has been an increased focus on improving patent classification systems. this is due to the growing recognition of the importance of classification in improving the efficiency of patent examination and in providing better access to information for users of patent databases. currently, patent examiners manually classify patents, which is a time-consuming process. if a method could be developed for automatically classifying patents, it would greatly reduce the amount of time needed to examine a patent application. this is an important area of research because it has the potential to significantly improve the efficiency of patent examination. current research efforts in patent classification are focusing on improving the efficiency and accuracy of the classification process. one area of research is exploring the use of machine learning algorithms to automatically classify patents @xcite . another area of research is looking at ways to improve the use of prior art information in the classification process @xcite . while classifying patent text is applied widely for some languages, such as, english. the arabic version of the problem requires the availability of arabic patent annotated corpus with considerable size as well as experimenting with different classification models. therefore, the contributions of this paper include: 1-constructing the first arabic patent dataset (called arpatent) labeled with the international patent classification (ipc). 2-evaluating twelve classification approaches in order to achieve a baseline for arabic patents classification. the rest of the paper is organized as follows: sections 2 and 3 provide background and related work on patents and their classification; section 4 presents the data collection and preprocessing process. sections 5 and 6 discuss the used methods and the obtained results. finally, section 7 concludes the paper with limitation and research outlook. ## background a patent is a form of intellectual property that gives its owner the legal right to exclude others from making, using, or selling an invention for a limited period of time. ## related work there are many studies in the literature that tackled automated patent classification. the earliest studies employed classical natural language processing (nlp) and machine learning (ml) approaches with feature engineering. for instance, @xcite did many experiments in two levels, classes and subclasses. they compared the precision of using many classifiers, naïve bayes (nb), support vector machine (svm), spars network of windows (snow) and k-nearest neighbor (knn) classifiers on their self-collected wipo 2 dataset named wipo-alpha, with performing stop words removal, stemming, and term selection using information gain as a preprocessing step. the best result in classes-level experiments was obtained with nb classifier (79%). while in the subclasses-level the best result was with knn (62%). similarly, @xcite used stemming, dimensionality reduction, stop word removal and removal of rare terms with a neural network called hitec which was evaluated on wipo-alpha corpus and espace a/b 3 corpora. there results outperform other state-of-the art classifiers significantly (by 6.5~14.5%). additionally, @xcite created a list of stop words specifically for the patent domain and used a tf-idf weighing system to choose their feature set. the patent document classification was examined using a multi-label model and 564,793 registered korean patents at the ipc subclass level. they achieved a precision rate of 87.2% when using titles, abstracts, claims, technical fields, and backgrounds. around 2017, the focus of automated patent classification research changed to deep learning (dl) approaches. @xcite trained espace a/b patent dataset with word2vec and fed it to lstm classifier. at the level of subclasses, they achieved an accuracy rate of 63%. likewise, @xcite applied similar approach (word2vec and lstm) with their self-collected domain-specific patent datasets for security. their approach achieved 93.48% of accuracy. on the other hand, @xcite employed bidirectional grus (another type of rnn) to improve classification performance compared to word embeddings trained with word2vec on wiki pages and the fasttext embedding that was trained on different datasets (wipo-alpha ,uspto foot_3 -2m and uspto-5m ). their approach increased the average precision for patent classification by 17 percent compared to state-of-the-art approaches. moreover, @xcite created a self-trained word embedding that was trained on a million patents collected from multiple patents datasets (european, german, japanese and chinese patents), and then classified them using lstm section (class) topic a human necessities b performing operations, transporting c chemistry, metallurgy d textiles, paper e fixed constructions f mechanical engineering, lighting, heating, weapons g physics h electricity their results outperformed traditional rnn. however, using cnns alone has gained the best results in patent classification task. in @xcite , they achieved an accuracy of 52.02% at the subclass level with the wipo-alpha dataset. ensemble techniques were also among the used approaches in patent classifications. @xcite experimented with clef-ip 2011 test collection to compare the accuracy of classifying english patents using different individuals' classifiers (cnn, bi-lstm, bi-gru, lstm, and gru) and using ensemble approach with three classifiers. their highest accuracy (64.85%) was gained by using ensemble approach that combined three of their best performing classifiers. from the previous studies we noticed that there is a huge interest in the community for patent classification in languages such as english and korean. furthermore, the best results were obtained with fine-tuning bert language models and ensemble approach. therefore, in this paper we created the first arabic patent dataset and applied the best classification approaches mentioned previously in the literature. ## methodology to achieve the goal of finding the best baseline for classifying arabic patents, different machine learning approaches were considered. first, from the traditional approaches, svm were implemented with different word embeddings techniques; namely: tf-idf, and skip-gram word2vec. for word2vec, we used the pre-trained embeddings aravec @xcite , and we also trained our own word2vec version, named arpatent-word2vec 8 , on the entire preprocessed arabic patent text. moreover, since bert-based models have shown state-of-the-art performance at language understanding, we fine-tuned multiple arabic bert-based models on the unprocessed titles and abstracts of arpatent for the task of patent classification. the used fine-tuning hyperparameters were: learning rate of 2e-5 using adam's optimizer, a dropout layer of 0.1 at the feed-forward classifier, a batch size of 32 and 3 epochs. moreover, the max length of tokens was set to 350 tokens for the tokenizer, truncating any input beyond that length. furthermore, stratified split was used for splitting the data into 10/10/80 for testing, validation, and training respectively. the evaluation metrics used are accuracy, macroprecision, macro-recall and macro-f1 score, having the last as the primary metric since the dataset is highly imbalanced. the following subsections give a summary of the used pre-trained models. ## results and discussions although arpatent-word2vec embeddings was trained on the arpatent text, yet it did not improve the classification task, this might be attributed to the models being trained on the title and abstract of a patent rather than the complete patent dataset. on the other hand, the accuracy of ensemble-1 results outperformed all other models with 70.49%. ensemble-1 consists of the three best performing models namely: arbert, camel-msa and qarib, with weighted sum of 1, 0.5 and 0.5, giving arbert the priority in voting. we also combined the svm models in ensemble-2 with the same weighted sum, giving the priority to svm-tf-idf. the last ensemble, ensemble-3, was a combination of all svms and arbert, giving arbert and tf-idf the twice weight of other svms. from the results, we can observe that the best performing model in terms of f1 score was arbert. although we tried to ensemble different models; but due to little diversity in base-models' predictions; the performance could not get improved, for example, refer to the confusion matrices in figure 2 , figure 3 and figure 4 for the models participating in ensemble-1 and notice the similarity in predictions. moreover, due to the little number of samples for some classes, the overall performance was low, especially for class d where most of its instances were not classified correctly even with arbert. ## conclusion classification of patents is a crucial part of the patent system, as it allows for the efficient and effective management of patent information. in this paper, we constructed the first arabic patent dataset called arpatent and experimented with twelve different classification approaches to develop a baseline for arabic patent classification. our results show that the arbert model had the best performance in classifying patents. we can say that our results are comparable to those found in the literature, even with our small sized dataset. one limitation of this work resides in the imbalanced dataset, this affected the performance of patent classification. also, using only the patent's title and abstract for the purpose of classification did not yield good results. therefore, as a future plan we intend to repeat the experiments while considering the whole text for classification g also we need to increase the size of the dataset to obtain more successful results. finally, we believe that our paper has produced some preliminary knowledge and useful results that will help support the task of arabic patent classification.