Paper: Nougat: Neural Optical Understanding for Academic Documents (ICLR-2024)

Reviewer comment (Presentation):
Figure 5 and section 5.4 on repetitions during inference being difficult to follow. It indicates that the explanation or presentation of this aspect of the model may not be clear or well-explained, making it challenging for readers to grasp the concept.

Author response:
Thank you for your feedback. We will rework this section to make it easier to follow.

Reviewer comment (Experiments):
What does “out-of-domain documents” refer to on page 8, and what is the objective of conducting experiments on repetition data?

Author response:
“out-of-domain documents” are in our case academic documents found on the internet including scanned or digital books (for examples, see in Appendix B: Fig B.1 and Fig B.2). We realize this was not clear in the original text and updated it. Thank you for the pointer.

Reviewer comment (Writing):
In Eq. X, does “a” or “b” denote a paragraph index?

Author response:
Yes, in Eq. X, “a” or “b” denote a paragraph index.

Reviewer comment (Experiments):
Why were Table, Plain text, and Math treated as separate modalities for experimentation, and how does this differ from conventional methods?

Author response:
In our study, we chose to categorize Tables, Math, and Plain Text as distinct text modalities, only for our evaluation process. This distinction was made to gain more detailed insights into the potentially varying results, acknowledging their inherent differences. For instance, the way Tables might yield different outcomes compared to Plain Text is noteworthy. It might be more apt to use the term "category" rather than "modality". Importantly, this differentiation was applied only in the evaluation phase, not during training. As such, the reference to 'conventional methods' isn't directly applicable here, since our approach didn't alter the training methodology, but rather focused on a more nuanced evaluation strategy.

Reviewer comment (Presentation):
Would it be better to move the first paragraph in Section 3 Model from page 2 to the related work section?

Author response:
Yes, that would be better, thank you for the suggestion!

Reviewer comment (Evaluation):
Are the scores presented in % for the editdistance metric? This metric is referred to as CER (Character Error Rate) in the document understanding community.

Author response:
Thank you for the pointer. The scores are not presented in %. We will rename the metric in the paper.

Reviewer comment (Writing):
This sentence is not clear, as Fig. 2 shows that LaTeX sources are recompiled. In Overleaf you can match paragraphs in the LaTeX source and their localization in the PDF - can this feature be used to simplify this step?  
"Since we are not recompiling the LaTeX sources for each paper, we must heuristically split the source file into parts, which correspond to different pages. To achieve that we are using the embedded text on the PDF page and match it to source text."

Author response:
Recompiling LaTeX sources using SyncTeX could indeed streamline our data generation process, but it comes at the cost of significantly increased computational requirements.  
To mitigate this, we have opted for a more general approach by utilizing the original PDF files supplied by the authors. This choice allows us to accommodate various data sources, including cases such as PMC documents where no LaTeX source files are provided.  
Fig. 2 was only supposed to show the relationship between the tex and pdf files, not to imply that we are recompiling the LaTeX sources for each paper. The image caption does clarify the relationship: “d) the PDF file provided by the authors”

Reviewer comment (Experiments):
The label noise strategy is not clear. Is noise added to 10% of labels?  
"This process continues until the newly sampled number is greater than a specified threshold (in this case, 10%)"

Author response:
Thank you for the feedback. The strategy is as follows:  
With a chance of 10% we change one token in the ground truth text to another randomly selected token. The likelihood of changing two tokens simultaneously is 1%, calculated as (10%)^2. Similarly, the probability diminishes exponentially with each additional token substitution, such that the chance of changing three tokens is 0.1%, and so forth.

Reviewer comment (Evaluation):
Text vs equations vs tables all have different semantics. Could you comment on which metrics are the most relevant for each text category? It would be interesting to use specific metrics for equation recognition (see the CHROME competition https://www.cs.rit.edu/~rlaz/files/CROHME+TFD%E2%80%932019.pdf).

Author response:
In my opinion, the CER is the most important metric for all of the modalities, since we pre-process all equations and tables to reduce ambiguity. However, using a symbolic graph could help give a clearer signal. Thank you for the pointer, we will investigate it in the future.

Reviewer comment (Experiments):
The limitation of 4096 tokens seems low for a full page. Did you encounter pages with a larger number of tokens to predict, and how did you handle them? How would this limitation affect document-level recognition (sliding window/patch?).

Author response:
On average a page has 1400 tokens. The 4k context size is more than enough for most of the pages. The only problem we noticed are with large tables, because they are token intensive since the GALACTICA tokenizer was not trained on LaTeX tables.  
We will leave long context exploration to future work.

Reviewer comment (Reproducibility):
Is any post-processing strategy used to normalize section/subsection formatting (example: `##` (section) predicted instead of `###` (subsection))?

Author response:
No, we don’t have a post-processing strategy for section titles.

Reviewer comment (Presentation):
Do you handle references to other parts of the page? (Ref to tables, figures in the text?). It might not be crucial when dealing with pages, but it will be useful at document-level.

Author response:
All references (to e.g. citations, formulas, tables, figures) are replaced with the corresponding label in the LatexML pre-processing step. So you would need to pair them up again during post-processing.

Reviewer comment (Experiments):
To go beyond recognition, how could Nougat be adapted to a multitask setting with e.g. Visual Question Answering / Neural Machine Translation / Summarization? It would certainly be useful to allow users to ask questions about the article: ask for an explanation, ask for a reference, ask to translate or summarize.

Author response:
This is a good idea, but we did not look into it yet. One could possibly use the attention maps of the vision transformer to highlight the most important image sections for a given output token.  
We agree, adapting Nougat to Visual Question Answering would be an interesting direction. Please refer to our general response 3.

Reviewer comment (Experiments):
One of the central topics in the results is about the repetition of the same sentence again, an artifact of the greedy sampling, and authors approach this problem by doing a data augmentation to help reduce those repetitions. Is there a particular reason why the authors chose to stick with greedy sampling instead of using Top-K / Top-p sampling, which is a standard approach to reduce the overconfidence during autoregressive decoding? (See ex. "The curious case of neural text degeneration")

Author response:
Thank you for your thoughtful feedback.

In our OCR problem domain, the task is to predict the next token, and there is only one correct token to be predicted at each step. Given this nature of the task, we opted for greedy decoding as it ensures determinism in our predictions. Nucleus sampling introduces a level of randomness that is unnecessary for our specific use case, as there is a single correct token at each decoding step.

During the course of our experimentation, we did explore other sampling techniques such as top-k and top-p. However, we observed that these methods were also prone to introducing repetitions in the generated sequences. In the next version of our paper, we will report those additional observations.