Transformers Go for the LOL s: Generating (Humourous) Titles from Scientific Abstracts End-to-End (2023)

Abstract: We consider the end-to-end abstract-to-title generation problem, exploring seven recent transformer based models (including ChatGPT) fine-tuned on more than 30k abstract-title pairs from NLP and machine learning (ML) venues. As an extension, we also consider the harder problem of generating humorous paper titles. For the latter, we compile the first large-scale humor annotated dataset for scientific papers in the NLP/ML domains, comprising 2.6k titles. We evaluate all models using human and automatic metrics. Our human evaluation suggests that our best end-to-end system per-forms similarly to human authors (but arguably slightly worse). Generating funny titles is more difficult, however, and our automatic systems clearly underperform relative to humans and often learn dataset artefacts of humor. Finally, ChatGPT, without any fine-tuning, performs on the level of our best fine-tuned system.

## introduction computer-assisted writing is an important and longstanding use case of nlp and natural language generation (nlg) @xcite , e.g., via and beyond tools such as spell checkers or grammatical error correction. the recent success of large-scale language models (llms), such as the gpt generation of nlg models, has made the goal even more realistic and promises full-scale automatic text generation, without any human intervention. in this work, we concern ourselves with automatic text generation in the scientific domain. sample scenarios in this general context involve (semi-)automatically generating reviews for scientific papers @xcite , e.g., as a response to high reviewing load in the face of exploding submission 1 our paper title is a (modified) merge of a funny and unfunny title suggested by @xcite . our paper logo is drawn by dall-e ( https://openai. com/dall-e-2/ ). data+code: https://github.com/cyr19/a2t numbers; and generating captions for tables that require reasoning capabilities @xcite . our goal is much more modest: we ask whether language models can generate adequate titles given a human authored abstract as input; we refer to this task as a2t (abstract-to-title generation). title generation is important as titles are the first access points to papers; a good title may attract more readers and consequently increase paper impact, e.g., in terms of citation numbers @xcite . besides generating titles per-se, we also aim for generating humorous titles, an inherently difficult problem due to small sample size and the vagueness of humor. generating funny titles may be relevant as a funny title may attract more readers: for example, @xcite find that funny titles have significantly higher citation rates. we approach the problem as a standard sequenceto-sequence text generation problem, where we fine-tune llms on more than 30k abstract-title pairs from ml and nlp. our contributions: we also leverage the relationship to summarization by considering pre-trained models fine-tuned on summarization datasets. in contrast to @xcite and @xcite , we only consider end-to-end models that do not involve pipelines. while refinement steps could be further helpful (but also error-prone), they additionally require potentially undesirable human intervention @xcite . related to the task of title generation is the task of headline generation e.g. for news. @xcite use a coarse-to-fine approach which first identifies important sentences and then converts them into a headline. in this way, the model is not confused by 'too much' irrelevant information. in a2t, the first summarization step may not be necessary, as the abstract is already a summary of the scientific paper. how titles should be (and are) structured has been researched for a long time, e.g., @xcite . hartley @xcite gives a typology of title types, distinguishing 13 title classes, e.g., those that state results vs. methods. beyond title generation, related fields of text generation for science are related work generation @xcite , more general automatic paper section writing assistance @xcite , and automatically generating reviews for scientific articles @xcite . more broadly relating to science, meta has in 2022 released an llm for the scientific domain called galactica @xcite , but they mostly explore it for scientific classification tasks rather than generation. ## data we use the dataset released by @xcite , which contains title-abstract pairs and corresponding meta-information such as the publication year and venue. @xcite extracted the data from two sources: acl anthology (from 1984 to 2021) and machine learning conferences (from 1989 to 2021); we refer to the datasets from these two sources as nlp and ml, respectively. after filtering (described in appendix a), 32,952 abstracttitle pairs remain in our dataset. ## title generation we first explore whether existing state-of-the-art seq2seq models manage to generate human-level titles from abstracts. hence, we do not include humor constraints. we use an 8:2 ratio to divide the data into train and test sets, and randomly select 1,000 instances from the train set for the dev set. ## humorous title generation to generate humorous titles, we first need a dataset of humor annotated titles in our domain (nlp and ml papers). we cannot resort to the data of @xcite ; @xcite as those leverage papers from other scientific fields. as a consequence, we build our own dataset. when constructing the dataset, we ask annotators to rely on their intuition of humor rather than issuing guidelines of what they should find funny. this can be justified as humor is often subjective and culture-and even gender-specific @xcite . there is also a multitude of theories around humor, indicating the ambiguity of the concept. 4 humor annotation + classification we train humor classifiers on human annotated data to automatically label titles as funny, funny med , and ¬funny (examples see stage 1: the two annotators initially annotated 1,730 titles: 1,603 titles as ¬funny, 106 as funny med , and 21 as funny (kappa 0.65 on 300 common instances). to combat this severe data imbalance, we resort to ensembling with each classifier trained on more balanced splits: we randomly generate 11 different data splits, where the train set of each split consists of 100 funny or medium funny titles and 200 not funny titles (all randomly drawn). on those splits, we train 11 classifiers to construct an ensemble classifier. to evaluate the classifier performance, the two annotators annotated another 315 titles jointly, obtaining 0.639 kappa. our best ensemble classifier leverages the sum of the label values assigned by the 11 individual classifiers to predict humorousness, yielding 4.8% macro f1 improvement compared to the individual classifiers (62.4% vs. 57.6%). details are in appendix g. stage 2: to find more funny title candidates to annotate, the two annotators annotated the funniest 396 titles in the original dataset from beese et al. (2023), predicted by the stage 1 ensemble classifier; 75.8% (300 titles) were judged as funny or funny med , which is substantially higher than the proportion of funny titles in the annotated data of stage 1 (7.3%). thus, the annotated data expands to 2,441 titles (= 1, 730 + 315 + 396), where 1,893 are labeled as ¬funny, 492 as funny med and 56 as funny. subsequently, we re-train 11 classifiers on newly generated 11 data splits from the expanded data of 2,441 titles; now the train set of each split has 400 (medium) funny titles and 800 not funny titles. as before, we ensemble the 11 classifiers as in stage 1. we test the classifiers from both stages on a heldout test set containing 197 titles annotated by the two annotators (0.649 kappa). the macro f1 scores of those classifiers are presented in final dataset we use our humor classifier to automatically label the rest of the data. considering the difficulty of three-way classification for both humans and classifiers, we only consider two humor levels in further experiments: (1) funny (for funny and medium funny titles) and ( 2 ) ¬funny (for not funny titles). thus, we collect 31,541 instances (>95%) with ¬funny and 1,411 with funny titles. we split the resulting data to train, dev, and test sets, ensuring that (1) the data with human-annotated titles remains in the train set, as the humor classifier trained and evaluated on it will be used as an automatic humor evaluator; (2) 80% of the data in dev/test is from nlp and 20% from ml because our annotators are more knowledgable for nlp papers, and (3) the ratio of funny data to ¬funny data in dev/test set is 1:2. 5 as funny data is only a small portion of the whole data, we only keep 600 instances in the dev/test sets, the remaining data serves as the train data. appendix h summarizes the statistics of the final dataset. ## comparison with chatgpt we compare our fine-tuned bart xsum (without training on pseudo data) with the recent popular chatgpt model. 7 firstly, we use the two models to generate funny and not funny titles for 100 abstracts from the emnlp 2022 handbook which chatgpt could not have seen in its training data. our prompt for chatgpt is "i want a funny title and a not-funny title for the following abstract: chatgpt generates funnier but lower-quality titles compared to bart xsum but chatgpt is almost on par for non-funny titles. hence, we conclude that chatgpt without any fine-tuning may already perform similarly to our fine-tuned bart xsum . after our experiments, chatgpt has been updated several times. to inspect whether the new version performs better, we conduct a second experiment using the latest model "gpt-3.5-turbo-0613" with the official api, utilizing the default hyperparameters. details are given in appendix l. overall, our evaluation suggests that the newer chatgpt does not perform better: in 25 out of 40 cases, the previous titles were selected as the better ones. in fact, the new version performs much worse for generating funny titles: it loses to the previous version on 18 out of 20 instances. ## discussion & analysis are automatic titles really superior? overall, our results in §5 and §6 seem to indicate that automatically generated titles outperform human titles. however, looking at the distribution of best/worst titles, we see again a high frequency of worst human titles as annotated by our human annotators; in fact, human titles are most frequently selected as worst titles except when the automatic systems use the humor constraint. as before, the likely reason is a lower lexical overlap between human titles and abstracts. indeed, we find that human titles have lower lexical overlap with abstracts when compared to automatically generated titles from chatgpt and bart xsum , e.g., 57-61% of content words in human titles appear in the abstract, while the number is 64-67% for bart xsum and chatgpt. very negatively evaluated human titles have even lower lexical overlap. in contrast, human titles were again most frequently selected as best titles except when including chatgpt. overall, our findings implicate that automatically generated titles can be competitive but are presumably still slightly worse than author choices. to verify this hypothesis, we suggest a more costly evaluation scheme in the form of a user study involving the authors of papers instead of paper external annotators in future studies. is training on extra parts besides abstract beneficial? we argued that human titles may not only be based on abstracts, but (to some extent) the full papers. to inspect whether training title generation systems on more than abstracts alone leads to better systems, we train bartxsum and the popular longformer-encoder-decoder (led) @xcite , which can deal with longer input sequences, in two settings: (1) only abstracts and (2) abstracts, introductions, and conclusions; we denote the corresponding models as "[model]+a" and "[model]+x", respectively. we use the data from @xcite , which contains the sentences of all papers from acl anthology until 2019. technical details are given in appendix m. we randomly select 29 instances from the test sets for human evaluation: 14 for bartxsum and 15 for led. two evaluators were asked to select the better one among the two titles generated by "[model]+a" and "[model]+x" with the same underlying model, given the abstract, introduction and conclusion. on the jointly assessed 10 instances, they obtained 0.474 kappa. our evaluation results show that: bartxsum seems to benefit from training on more parts (bartxsum+x wins 8 out of 14 instances); for led, it is not the case (led+a wins 11 out of 15 cases). on introspection, we do find that the models trained on more than abstracts can indeed leverage some relevant keywords not in the abstracts, which makes their titles sometimes better. on the other hand, they are tasked with identifying relevant titles given more 'background noise' (longer texts) which causes them to hallucinate more and be more vague. we show examples in appendix n. evaluation with more than abstracts alone is also considerably more costly for humans. overall, these experiments thus indicate that training (and evaluating) on highly specific and condensed abstracts is advantageous. humor constraints on introspection, we find that the funny titles generated by chatgpt do not conform to a style of humor used in scientific papers. this indicates that chatgpt lacks fine-tuning on humor in science. for bart xsum , its problem seems to be that it overfits to data artefacts learned from the data indicating that it does not properly learn a generalizable notion of humor. additionally, both models often do not match the content of the abstract/title to the humor framing (examples see ## conclusion we considered the abstract-to-title generation problem using end-to-end models. to do so, we trained six recent text-to-text generation systems on more than 30k nlp and ml papers. we evaluated the systems using an array of state-of-the-art automatic metrics as well as human evaluation. our evaluation indicates that some current text generation models can generate titles with similar quality as humans, but human authors are apparently still superior. we also considered the humorous title generation problem as an extension, compiling the first dataset in the nlp/ml domain in this context, comprising over 2.6k titles annotated by two annotators with acceptable agreement. we find that our systems struggle with generating humorous titles and instead overfit to frequent patterns in the data, indicating much scope for future research.