CATALPA _ E du NLP at P rag T ag-2023 (2023)

Abstract: This paper describes our contribution to the PragTag-2023 Shared Task. We describe and compare different approaches based on sentence classification, sentence similarity, and sequence tagging. We find that a BERT-based sentence labeling approach integrating positional information outperforms both sequence tagging and SBERT-based sentence classification. We further provide analyses highlighting the potential of combining different approaches.

## introduction this paper describes the catalpa_edunlp entry to the first shared task on pragmatic tagging of peer reviews @xcite . in this task, sentences within peer-reviews for academic articles from various domains are assigned a label expressing the pragmatic function of that sentence, namely recap, strength, weakness, todo, structure or other @xcite . we experiment with various approaches presented in section 3 and 4. as there is no clear winner among them (see results in section 5), we further focus on comparing them to see under which conditions each setting works best (section 6). ## datasets we participated in all three evaluation setups of the shared task, which provided different amounts of training data. in the full-data setting, 117 reviews with 2326 sentences are provided, from which we split ten reviews to serve as our internal validation data. in the low-data setting, 33 reviews with 739 sentences are used (we take five of these reviews as our internal testing data or perform four-fold cross-validation). in the no-data setting, we use our internal test data from the full-data setting for evaluation. the shared task provides two additional data sets: f1000raw contains unlabeled data (7423 reviews from the same domains as the training data. to use this data, we first extract the domain for each article via a lookup of the respective gateway on https://f1000research.com . since a large number of articles cannot be assigned any domain, we only use articles for which we can assign a domain, yielding 269 additional iscb, 144 rpkg, 445 diso, 525 case and 227 scip reviews. the arr-22 dataset consists of 684 labeled reviews coming from a different domain and using a different annotation scheme @xcite . while some of the mappings are straightforward (paper summary to recap, summary of strengths to strength, summary of weaknesses to weakness), we mapped comments, suggestions and typos to todo and found no correspondences for structure and other. ## approaches with training data we explore three complementary approaches, following similar tasks of identifying sections in scientific articles or abstracts that cast the problem as one of sentence classification @xcite or sequence labeling @xcite : a bert-based sentence classification model @xcite , a longformerbased sequence tagging model @xcite , and a sbert-based model @xcite to compute semantic similarity between sentences. the total training and inference time was about 22 hours on a single gpu. ## zero-shot approaches this section describes our no-data approaches. ## results following the evaluation scheme in the shared task, we report macro-averaged f1-scores per domain for our own data split and only an overall f1-score for the challenge test set. in the low-data setting, our sbert-based method performs better than the bert-based methods, which consists of the results observed on the validation set. the no-data performance of our sbert-based method is slightly better on the test set than the average on our validation splits. (following the competition rules regarding reproducibility, we did not submit our gpt results since the model requires a paid api.) ## analysis the different approaches produce results in the same ballpark so that one may wonder if they can be used interchangeably. to investigate this we compare the results by checking four conditions: the percentage of sentences that all four models judge correctly, the percentage that none of the models classified correctly, which proportion is classified correctly in a majority setting and the percentage of correctly classified sentences that could be reached in an oracle condition if we knew to which model a sentence should be passed, i.e. the percentage of sentences judged correctly by at least one model. for this analysis, we use the respective bestperforming model variant on our internal split of the data provided for the full data setting. we analyze all four approaches we took: sentence classification using roberta, similarity-based classification with sbert, sequence tagging using the longformer architecture, and zero-shot application of gpt. figure 2 gives an overview of how many sentences are correctly classified by which method. the oracle condition sums up to 94% of test instances being assigned the correct label, meaning that the remaining six percent are classified correctly by none of the methods. about a third (36%) of the data is correctly solved by all four models, and a majority voting over their predictions comes up to 83% accuracy, which is 1% lower than what roberta achieves on its own. overall, gpt seems the most distinct from the other methods: it has the highest number of 11 sentences that none of the other methods can classify correctly. such sentences often have the label other, for example "dear authors". however, there are 66 sentences for which all other methods except gpt predict the correct label. gpt rarely labeled instances of "recap" correctly and often mislabeled "structure" as "other", such as "reviewer response for version 1". figure 1 breaks down performance for the individual labels, revealing gpt to be much worse in both precision and recall when it comes to structure, and showing especially low recall for recap. all methods have the most difficulty with sentences labeled other, with our se-quence tagging approach having both precision and recall of zero. the overall best-performing roberta method especially shows superiority in terms of high and balanced precision and recall values for the labels strength, todo, and weakness. ## conclusion we have presented experiments using a variety of very different approaches. the comparison shows that they behave quite differently and that a sensible combination of approaches yields further improvements. future work therefore has to determine which approach is most suitable for a given item to be classified.