Paper: HHD-Ethiopic: A Historical Handwritten Dataset for Ethiopic OCR with Baseline Models and Human-level Performance (ICLR-2024)

Reviewer comment (Novelty):
This paper is a dataset paper, without new technical methods.

Author response:
[**This paper is a dataset paper, without new technical methods**]:  Yes, the  paper is a datasets paper and our goal is to  create a historical handwritten dataset for a low-resourced script and provide benchmark results with SOTA OCR  models. In addition we compare the human-machine performance based on evaluation results from several human annotators and the baseline models and we believe this contribution is a good fit for “**dataset and benchmark**" themes (https://iclr.cc/Conferences/2024/CallForPapers) of ICLR-2024.

Reviewer comment (Novelty):
The academic challenges of this dataset are not representative. The impact of this new benchmark seems limited.

Author response:
[**The academic challenges of this dataset are not representative**]: The experimental evaluation results of this dataset, involving several SOTA OCR models and human performance, indicate limited recognition performance. Therefore, this dataset presents challenges and can be valuable for further investigation of new methods to address such challenges. If you have a particular aspect you are curious about, let us know.

Reviewer comment (Experiments):
full pages are not provides, which prevents the evaluation of full page models, which are currently the best performing models, compared with cascade models (line detection + HTR), which accumulate errors

Author response:
[**full page are not provided to evaluate line detection+HTR**]: line detection + recognition is one way of OCRing. in literature, the cascaded detection and recognition task  is most common in scene text recognition. Our focus was to create segmented text-line images  for the recognition task and the annotation was done for lines images. However, we will keep updating the dataset for detection and recognition tasks in the future.

Reviewer comment (Novelty):
no reference to https://journals.openedition.org/jtei/4109

Author response:
[**include reference paper done for Ethiopic**]: we included the paper in our reference list.

Reviewer comment (Experiments):
no reference or comparison to the models available in Transkribus https://readcoop.eu/model/ethiopic-classical-ethiopic-scripts-from-ethiopia-and-eritrea/

Author response:
[**no reference or comparison with the already available Ethiopic_Eritrea  HTR model in Transkribus**] This model is released in the form of webApp, and cannot be retrained. Furthermore, the authors do not provide an API to easily upload samples and download results. Therefore, we limited ourselves to evaluating its performance manually by uploading a few with a samples manuscript image from our test dataset and comparing the results with those of our model. The Ethiopic_Eritrea  HTR model from the Transkribus model archives a CER of~65.2%  while our Hopt-attn-CTC model archives 17.82 for this sample specific test manuscript. Link for demo:https://github.com/ethopic/hhd-ethiopic-I/blob/main/Hopt_attn_CTC_vs_Transkribu.ipynb

Reviewer comment (Experiments):
tesseract supports Ethiopic but is neither mentioned nor tested. As the script is not cursive, a test of tesseract would be possible.

Author response:
[**Tesseract model**], Tesseract is an OCR framework designed for hundreds of scripts including Ethiopic. We evaluated the model proposed for Amharic on a sample test set. The results being so poor, this model cannot be considered state of the art. For this reason we did not include it as part of our baselines.

Reviewer comment (Experiments):
the models tested have never been evaluated on standard handwriting recognition bases. How do they compare with standard libraries such as Transkribus, pylaia and trOCR?

Author response:
[**the models tested have never been evaluated on standard handwriting recognition bases. How do they compare with standard libraries such as Transkribus, pylaia and trOCR?**] 
We have assessed eight methods, including SOTA OCR techniques validated on Latin, Chinese, and Amharic scripts as documented in the literature, under the same computational constraints and resources. Our focus is on models that are comparatively more lightweight than those requiring extensive computing resources or subscriptions. Additionally, we have incorporated the evaluation results of TrOCR (refer to Table 3 in the revised version). TrOCR, being a model with a large number of parameters, is fine-tuned for fewer epochs. This approach is not chosen because it requires more time for training and inference, leading us to prioritize smaller, more efficient networks. Similarly, the Ethiopic_Eritrea Handwritten Text Recognition (HTR) model in Transkribus, which you recommended and we have evaluated, is developed using PyLaia. It's important to note that models in Transkribus are not free and require a subscription for extensive HTR tasks. They also require a stable internet connection. Furthermore, the need to upload manuscripts for the Transkribus models raises ethical concerns.

Reviewer comment (Evaluation):
no measure of inter-annotator agreement: it seems that annotation is difficult, judging by the poor performance of humans. How correct is the annotation? What is the variance of the annotation?

Author response:
[**inter-annotator agreement of annotators**]  As discussed in section 3.1 and Appendix B, each text-line image in the training set is annotated individually. In contrast, the test sets undergo expert review, and we evaluate the ground-truth text differences before and after the review in CER (see section 4.1.)

Reviewer comment (Experiments):
no details are given about the manuscripts: where do they come from, how were they chosen, how many are there?

Author response:
[**details about the manuscripts**] As discussed in appendix B, the text line images are extracted from manuscripts between the 18th and 20th century; collected from ENALA covering 7 different books of cultural and religious contents (~ 1300 pages), while the remaining is from the internet.

Reviewer comment (Presentation):
The structure of Table 2 and its description in the bottom of page 6 is not match: According to the Table, the Test-set-I is the IID data, which should be Annot-V with 26.56% CER and 24.56% NED; the Test-set-II is the Annot-VI. And it is better to add a row displaying the average performance.

Author response:
[**Correction on structure of table 2 and add a row for average performance**]: we have corrected it (the middle horizontal line is moved four steps down, and  inserted one additional row for average performance of each group. Furthermore, we have updated the corresponding values of CER and NED in the text.

Reviewer comment (Evaluation):
Because of the error in Section 4.1, the Figure 5 and the conclusion inferred from it are also wrong (i.e. HPopt-Attn-CTC cannot surpass human performance on Test-set-II).

Author response:
[**Mismatch form section 4.1, the Figure 5 and the conclusion inferred from it are also wrong**]: After fixing the table structure in Section 4.2 (Table 2), the remaining inferences drawn from this table are now accurate.

Reviewer comment (Experiments):
According to the previous two reasons, I think the experiments of the paper are insufficient.

Author response:
[** According to the previous two reasons, I think the experiments of the paper are insufficient**]: Although we have previously shared the source code, trained models, and dataset for reproducibility, an “\hline” mistake in the LaTeX table code caused confusion in a section of the paper derived from this table. We have now corrected this error and the experiments are corrected.

Reviewer comment (Novelty):
This work did not propose a new approach to recognize with the specific Historical handwritten Ethiopic script.

Author response:
[**New approach for historical handwritten Ethiopic script**]  as mentioned in the introduction section, there is lack of historical handwritten dataset for Ethiopic, the main goal of this paper is to create the dataset and evaluate it with SOTA OCR models. In addition, we evaluate human-level performance as a benchmark.

Reviewer comment (Evaluation):
Researchers often compare the performance of recognition methods for other languages using the recognition accuracy. Why not include this metric?

Author response:
[**Why not include accuracy as a metric?**] We are using standard metrics (e.g  e.g https://arxiv.org/pdf/1909.07741.pdf) for sequence to sequence comparisons, where 3 types of errors occur at the character level: insertions, substitutions, and deletions. The 1-CER (CER=Character Error Rate) is as close as it gets to an accuracy (taking into account the 3 types of possible errors). At the sentence level, the accuracy is usually zero because there are usually a few character-level mistakes. Of course, if a language model or lexicon would be integrated in our system to mitigate some of the mistakes, the sentence-level error would go down, but this is no in the scope of this research.

Reviewer comment (Writing):
There are maybe sometimes too many details in the text, or repeated statements, which could be shortened in favor of either more analysis of the data, a more detailed description of the challenges of Ethiopic scripts (e.g. some parts of App. A are very interesting and could fit in the main part of the paper).

Author response:
[**Content rearrangement**] we reorganized contents in appendix A, B moving a portion to section 1 and 2 of the main paper (updates are highlighted )

Reviewer comment (Presentation):
The fact that some datasets exist for Ethiopic script (described in App. B) should appear in Section 2, where the reader is led to think that no such dataset exists.

Author response:
[**Content rearrangement**] we reorganized contents in appendix A, B moving a portion to section 1 and 2 of the main paper (updates are highlighted )

Reviewer comment (Experiments):
It would be interesting to see how the models presented in the experiment section would perform on these other datasets, or to include the models used in these other papers in the baseline, as they might address the challenges of Ethiopic script (for example, Abdurhaman et al. report a CER of less than 2% on their dataset).

Author response:
[**Evaluate models proposed in baseline papers on other Ethiopic datasets**] we evaluate our HHD-ethiopic dataset using models proposed for Ethiopic script by Dikubab et al. (2022) and Belay et al. (2019a) and present the result in Table 3. However, neither the source code nor pretrained models are  provided by Abdurahman et al. (2021) and it is difficult to reproduce the results and use the model to evaluate our dataset. In addition, the dataset of Abduraman is a word level dataset extracted from just 1574 lines of modern Amharic handwritten texts, which follow different writing structures (Example in modern ethiopic writing space is a separator while both space and  “two dots” are used as a separator in historical scripts–see Figure 11 in appendix section).  To add more, we evaluated our dataset using a similar CNN-RNN-CTC architecture proposed by Abdurahman et al. (2021), although we haven't accessed the implementation details provided by Abdurahman (See Table 3 in the main paper).

Reviewer comment (Experiments):
How would the human performance be with access to reference material?

Author response:
>**How would the human performance be with access to reference material?** Those individuals participated in the human-level performance have no access for  references. The only individuals having access to the reference are people participating in Ground-truth annotation and their supervisors ( see  section 3.1 and appendix B, phase I).

Reviewer comment (Evaluation):
Do the annotators make the same errors or different ones?

Author response:
>**Do the annotators make the same errors or different ones?**  Annotators make different errors in the case of tests, as each text line is annotated by more than one person. However, in the training set, due to the cost associated with annotating each line by many annotators, each text-line image was annotated by an individual.

Reviewer comment (Evaluation):
Do the models make the same mistakes as the human evaluation?

Author response:
>**Do the model make the same mistakes as the human evaluation?**  Based on our observations during the experiment and the sample predictions shown in Figure 7, the mistakes differ. Both models and humans made substitution errors, but the model tends to have a higher number of insertions and deletions.

Reviewer comment (Experiments):
For the training set labeling, is there a measure of, for example, inter-annotator agreement?

Author response:
>**For the training set labeling, is there a measure of, for example, inter-annotator agreement?**  For the training set, since each text-line is annotated by an individual annotator we haven’t measured inter-annotator agreement for the training set. However we measure the agreement between the supervisors and the expert for the test set (see the revised version of the main paper, section 4.1 highlighted in yellow).

Reviewer comment (Evaluation):
Why are CRNN, ASTER, SVTR, etc. models worse than the CTC ones despite being larger?

Author response:
[**Why are CRNN, ASTER, SVTR, etc. models worse than the CTC ones despite being larger? In appendix C we learn that they were not trained to the end due to the lack of resources, but that makes the result confusing, if not misleading if they are understood as baselines**].  As discussed in Appendix C.2, we conducted runs for all models with a limited wall time, less than 24 hours, regardless of their complexity. The initial CTC-based models, with smaller parameters, underwent more epochs, whereas models with larger parameters could not. Furthermore, we implemented the CTC-based model using optimized hyperparameter values through Bayesian optimization (refer to section 3.2 of the main paper), resulting in improved outcomes.

Reviewer comment (Experiments):
In appendix C we learn that CRNN, ASTER, SVTR, etc. were not trained to the end due to the lack of resources, but that makes the result confusing, if not misleading if they are understood as baselines.

Author response:
[**Why are CRNN, ASTER, SVTR, etc. models worse than the CTC ones despite being larger? In appendix C we learn that they were not trained to the end due to the lack of resources, but that makes the result confusing, if not misleading if they are understood as baselines**].  As discussed in Appendix C.2, we conducted runs for all models with a limited wall time, less than 24 hours, regardless of their complexity. The initial CTC-based models, with smaller parameters, underwent more epochs, whereas models with larger parameters could not. Furthermore, we implemented the CTC-based model using optimized hyperparameter values through Bayesian optimization (refer to section 3.2 of the main paper), resulting in improved outcomes.

Reviewer comment (Experiments):
In the first test set, did you make sure that the same writer or document do not appear in the training and test set?

Author response:
[**In the first test set, did you make sure that the same writer or document does not appear in the training and test set?**].  Text-line image samples in the IID test sets are randomly selected from the distribution using sklearn train-test split. For OOD, only 18th-century documents are chosen without mixing the same documents.

Reviewer comment (Presentation):
The format of citations needs to be fixed.

Author response:
[**Minor remarks**] We corrected and highlighted the changes.

Reviewer comment (Presentation):
The "remark" column in Table I is not necessary.

Author response:
[**Minor remarks**] We corrected and highlighted the changes.

Reviewer comment (Writing):
On p5, CTC, attention, and transformers are put in parallel when they correspond to different things (loss, mechanism, model).

Author response:
[**Minor remarks**] We corrected and highlighted the changes.