What’s in a (dataset’s) name? The case of B ig P atent (2022)

Abstract: Sharing datasets and benchmarks has been crucial for rapidly improving Natural Language Processing models and systems. Documenting datasets’ characteristics (and any modification introduced over time) is equally important to avoid confusion and make comparisons reliable. Here, we describe the case of BigPatent, a dataset for patent summarization that exists in at least two rather different versions under the same name. While previous literature has not clearly distinguished among versions, their differences do not only lay on a surface level but also modify the dataset’s core nature and, thus, the complexity of the summarization task. While this paper describes a specific case, we aim to shed light on new challenges that might emerge in resource sharing and advocate for comprehensive documentation of datasets and models.

## introduction sharing models and datasets is essential for natural language processing (nlp). with the rise of transfer learning in the last few years, releasing large pre-trained models has become standard practice. consequently, several libraries have provided apis to access and work with those models efficiently. datasets have followed a similar trend: they are often shared by their authors and stored in hubs that expose apis. two notable examples of this trend are the tensorflow datasets collection foot_0 and the hugging face dataset library 2 @xcite . these libraries allow accessing published data, often with just a few lines of code. they drastically ease the experimentation loop, and allow users to download, experiment with, and probe existing resources. there is, however, another side to the coin: the dataset documentation is sometimes insufficient, which might lead to inconsistencies when performing experiments and comparing results to previous work. this paper analyzes a somewhat extreme case: the bigpatent dataset @xcite . bigpatent is a dataset for patent summarization, first published in 2019. patents have many peculiar characteristics that might be challenging for standard nlp systems: they span multiple pages, have very long sentences, contain a mix of legal and technical vocabulary, and are built out of noun phrases instead of clauses, with a long lexical chain @xcite . thus, the dataset has also become popular as a general benchmark for summarization. we show that the two popular tensorflow and hugging face dataset hubs expose different versions of bigpatent. these differences are not only superficial (e.g., casing, tokenization) but regard the very content of the source documents. we first briefly describe this difference and its impact on the dataset features (section 2); then, we examine previous work and show it hardly ever clarifies the version of the dataset used in experiments (section 3); finally, we show how the difference substantially impacts models' performance (section 4). while strongly advocating for resource sharing and infrastructure that make them easier to use, we hope that the discussion of this extreme case can shed light on the importance of careful resource documentation. ## the bigpatent dataset bigpatent is a dataset for the automatic summarization of patent documents. patents award inventors the exclusive right to use, make, and sell their inventions for a specific time and geographical area. patents are structured legal documents containing several sections. the de-scription section reports the technical characteristics of the invention and its preferred embodiments so that a person skilled in the art can understand and reproduce it. the description can be further divided into subsections (e.g., background, field of the invention, summary of the invention, detailed description, description of the drawings, etc.). the patent document also contains a humanwritten abstract. it is thus somewhat natural to construct a summarization dataset using the descriptions (or part of them) as the source texts and the abstracts as the gold-standard summaries. the dataset is not only interesting for a niche of patent mining researchers: in fact, patent documents show several interesting linguistic characteristics worth investigating (e.g., long sentences, unusual vocabulary, specific syntactic structure). moreover, since many popular large-scale summarization datasets are in the news domain @xcite @xcite , gathering data from different sources opens new challenges for nlp systems. for example, patent documents are very long, and their abstract is not very extractive with respect to the detailed description, as the original dataset shows @xcite . in its original version, published by bigpatent's authors and accessible on github foot_2 , only part of the description (typically the detailed description) is included in the input document, and the source does not contain any of the other subsections. the published dataset is also cased and tokenized. the hugging face dataset library exposes this version of the dataset (described in the related paper) 4 . with the advent of sequence-to-sequence transformer models for summarization (e.g., bart @xcite or pegasus @xcite al., 2020) )), however, using a strongly preprocessed dataset is not ideal. it is common practice to process the raw text with a model-specific tokenizer. this is likely why the tensorflow datasets collection contains a different version of the dataset that is cased and untokenized, with limited preprocessing over the original raw text foot_4 . however, a deeper look at the data reveals another difference: the tensorflow source documents contain a superset of the text contained in the original version. all subsections in the patent description are included. thus, the input not only contains the detailed description but often also the background, the field of the invention, etc., and, interestingly, a summary of the invention 6,7 . in the following, we compute some statistics on the two dataset versions (we call the original version bigp atent original and the subsequent modified cased version bigp atent n ew ) and their different characteristics. the dataset is divided into several subsets, following the cooperative patent classification (cpc) codes. due to the large dataset size (over 1.3 million examples), we restrict our analysis to its g (physics) subset: it includes patents of information systems devices and processes, for which the authors of this paper might be considered skilled in the art. however, our considerations are general. ## experiments to understand if the version of the dataset impacts models' performance, we fine-tuned a pre-trained bart @xcite base model on the two versions of the dataset. we train using the hugging face library with early stopping on the evaluation loss (patience: 5) and the following hyperparameters: max source length: 1000; max target length: 150; number of beams: 5; eval steps: 10k; max steps: 500m. we leave all other parameters to their default values. to corroborate the idea that the summary of the invention in the input improves the performance on bigp atent n ew , we trained a model using, as input, only the text in the summary of the invention subsection. in the few cases in which the patent did not include the summary subsection, we used the detailed description or the description of the embodiments. as described in section 2, we resorted to the raw data to extract the text in the summary of the invention subsection. this setting further improves the performance, with an increase of almost 16 and almost 5 points of rouge-l with respect to the original and the new version; note, however, that since bigp atent n ew does not contain the subsection headers, it is not directly possible to train models using the summary of the invention only as input. ## conclusions we have discussed the case of bigpatent, a dataset that exists in two very different versions. we have shown that the updated version of the dataset lacks some of the original characteristics (e.g., the high level of abstraction in the reference summaries and their high percentage of novel n-grams) and leads to much higher results with a simple transformer. to our best knowledge, this difference is not reported elsewhere, either in published research or in the dataset's online documentation. in fact, previous work tends to ignore the difference between the original and the new version, making it virtually impossible to understand experimental results, reproduce, and compare them. we believe bigpatent is an extreme case in which the lack of clear documentation has led to confusion -with two datasets so distant in their characteristics that they might be considered two different ones, used interchangeably. we always advise reporting the dataset version and characteristics when using bigpatent (and being aware of the possible problems with the comparison with previous work). we hope that the analysis of this case underlines the importance of clearly documenting datasets' characteristics and any possible modifications introduced over time.