The G utenberg Dialogue Dataset (2021)

Abstract: Large datasets are essential for neural modeling of many NLP tasks. Current publicly available open-domain dialogue datasets offer a trade-off between quality (e.g., DailyDialog) and size (e.g., Opensubtitles). We narrow this gap by building a high-quality dataset of 14.8M utterances in English, and smaller datasets in German, Dutch, Spanish, Portuguese, Italian, and Hungarian. We extract and process dialogues from public-domain books made available by Project Gutenberg. We describe our dialogue extraction pipeline, analyze the effects of the various heuristics used, and present an error analysis of extracted dialogues. Finally, we conduct experiments showing that better response quality can be achieved in zero-shot and finetuning settings by training on our data than on the larger but much noisier Opensubtitles dataset. Our open-source pipeline ( https://github.com/ricsinaruto/gutenberg-dialog ) can be extended to further languages with little additional effort. Researchers can also build their versions of existing datasets by adjusting various trade-off parameters.

## introduction current open-domain dialogue datasets offer tradeoffs between quality and size. high-quality datasets are usually too small to represent the multitude of topics required for a conversational agent. large datasets often lack good turn-segmentation and are generally noisy, models trained on such datasets generate low-quality or generic output. in section 2 we analyze publicly available dialogue corpora and the trade-offs they offer. to address the need for large, high-quality datasets we build a corpus of 14.8m utterances in english using publicly available books from project gutenberg. we also build datasets for german, dutch, spanish, portuguese, italian, and hungarian, with utterance counts in the 20k-200k range. we call this dataset ensemble the gutenberg dialogue dataset. we wish to make it explicit that we are not aiming to create a gold dataset. our goal is to create a dataset which offers a better size-quality trade-off than other dialogue corpora. the gutenberg dataset is both larger than dailydialog @xcite and has better quality than opensubtitles (tiedemann, 2012), and we think it benefits researchers by filling a need in the landscape of dialogue corpora. the gutenberg dialogue dataset and the code used to build it can be accessed through this repository: https://github.com/ricsinaruto/ gutenberg-dialog . the repository also contains all trained models presented in this paper and all data and training scripts used to produce the results. we also built a web demo interface for interacting with the trained models 3 . in section 3 we offer a detailed quantitative analysis of our heuristics to better understand their effects on data quality. section 4 presents our error analysis of the english dataset both at the utterance and dialogue level. using our mit licensed pipeline, researchers can easily build various dataset versions by adjusting a small number of parameters that control multiple dimensions of the size-quality trade-off. in section 5 we evaluate our dataset in a generative multi-turn and single-turn setting using the gpt-2 @xcite and transformer @xcite architectures, respectively. for each of the 7 languages, we compare models trained on gutenberg and opensubtitles. for english, we also compare zero-shot and finetuning performance of gutenberg and opensubtitles on two smaller datasets. potential improvements and future work is discussed in section 6. extension to additional languages is ongoing, we welcome all contributions from the community: our modular code requires only a limited amount of language-specific effort for each new language. books, especially fiction, have so far seen little use as a source of dialogue data. in dailydialog @xcite , 90 000 high-quality utterances are extracted from online resources for english language learners, extraction steps are not detailed. the quality of these dialogues and the lack of a large book-based dataset motivates our work. dialogues extracted from books, like movie subtitles, lack context, but their usefulness is evidenced by the cornell corpus @xcite and dailydialog. as argued by @xcite and @xcite , artificial dialogues in movies and books generally resemble natural conversations. such dialogues are also called written dialogues as opposed to spoken corpora like the switchboard corpus @xcite . though our corpus contains written dialogues we also perform evaluation on persona-chat, which can be considered as a spoken dialogue corpus, and show gutenberg's effectiveness in this setting as well. unfortunately, the cornell corpus is relatively small, while the opensubtitles corpus suffers from the fact that the original dataset lacks both dialogue and turn segmentation: subtitle lines are treated as turns and dialogue history consists of the previous n lines, with little to no additional post-processing used to extract dialogues instead of using the raw data @xcite removes the shortest and longest utterances to improve quality). these issues lead to trained models outputting generic responses, e.g., to the input "yes i believe there are green teas black teas and scented teas. any others?" a model trained on opensubtitles outputs "sure.". in addition, the types and ratio of errors in these datasets have not been explicitly analyzed. for the gutenberg dataset, we build a multi-step extraction pipeline and analyze both the performance of each heuristic and the ratio of each error type in a sample of the final corpus. unfortunately, most of the tools developed here are specific to the book domain, and use textual patterns which are not available in opensubtitles. in order to increase the quality of opensubtitles, subtitle-specific methods need to be developed, like taking into account the elapsed time between two subtitle lines. the size of our corpus facilitates effective training of large transformer-based models @xcite . recently, pretraining and finetuning large language models on specific tasks (including dialogue modeling) has gained popularity @xcite . transformer-based models and specifically gpt-2 have gained state-of-the-art status in the dialogue domain @xcite @xcite . through these models the community has gradually shifted from single-turn to multi-turn scenarios. since we wish to demonstrate our dataset's quality on the dialogue-level, we conduct experiments primarily with gpt-2. we report some single-turn trainings using transformer for comparison. we show gutenberg's effectiveness for multi-turn pretraining in section 5, comparing it to opensubtitles pre-training, which is popular in the literature @xcite @xcite . ## extraction pipeline most of project gutenberg's 60 000 online books are in english (47 300 books; 3 billion words). french, finnish, and german, the next most common languages, contain 3000, 2000, 1750 books, and 194m, 74m, 82m words, respectively. dutch, spanish, italian, portuguese, and chinese are all above 10m words, followed by a long tail of various languages. we used the gutenberg python package foot_7 to download books and query their license, language, and author metadata. further gutenberg statistics can be found in appendix a.2. this section describes heuristics and methods used to extract dialogues from books and remove noise. the main challenges are identifying changes between speakers within a dialogue and separating sets of utterances that do not belong to the same dialogue. to separate dialogues, changes in location, time, speaker, etc. would have to be identified directly, but we develop simple heuristics (e.g., distance between utterances) that can extract relatively high-quality conversations at scale. tunable parameters of our system offer trade-offs between data quality and size. using our open-source system researchers can build custom datasets that best suit their applications. our dialogue extraction pipeline includes three main steps: 1. conversational and narrative text is separated. 2. dialogic text is split into separate dialogues. 3. dialogues are segmented into separate turns (utterances). in most books, conver-sational text is highlighted; e.g., placed between single/double quotation marks in english or started by an em-dash in hungarian. naturally, these delimiters have other uses as well, but such cases are rare (about 5% of utterances, see section 4). we can only extract dialogues from books which clearly delimit both the start and end of conversations. in some languages/books, the start of an utterance is given, but the end is not, and narrative text can get mixed in (e.g., si vous arrivez avant nous, cria luigi au messager, annoncez à la nourrice que nous vous suivons. 'if you arrive before us, shouted luigi to the messenger, tell the nurse that we are following you.'). this is why we could not build a french dataset, and have relatively smaller datasets in dutch, italian, portuguese, and hungarian. figure 1 shows a sample dialogue highlighting our heuristics. in the following paragraphs, we offer a parameter-based description of our pipeline. method parameter filtered what pre-filter 2 (kl-div) 2090 books (4.42%) old books and noise delimiter filter 150 delimiters / 10 000 words 20 500 books (43.3%) books with no dialogues long utterances 100 words 610 000 utterances (3.95%) non-conversational utterances post-filter 20% rare words 20 478 dialogues (0.8%) dialogues containing many rare words in the english dataset, 2090 books were removed (4.42%). by analyzing 100 filtered and 100 nonfiltered books randomly, we found 8 false positives (books that should not have been removed), and 9 false negatives. delimiter filter before dialogue extraction, books with less than 150 delimiters per 10 000 words are removed. we assume that under a certain threshold the probability of delimiters used for non-conversational purposes is increased. we empirically set this ratio by increasing it until the assumption starts failing. since many books do not contain dialogues, almost half were removed (20 500) in the english pipeline. sampling 100 filtered and 100 non-filtered books, we found 8 false positives (books that should not have been removed), and 22 false negatives. in a sample of the final dataset, less than 5% of utterances were non-conversational (cf. section 4). dialogue gap if two dialogue segments highlighted by delimiters are far apart, i.e. there @xmath0, they will not be considered part of the same dialogue. this heuristic, the dialogue gap, will always offer a false positive/negative trade-off since the amount of text between dialogues varies considerably. we tuned this trade-off by reasoning that shorter dialogues are less problematic than incoherent dialogues: our setting yields 3.5 times fewer false negatives, as shown in section 4. our turn segmentation heuristic will also always treat separate paragraphs as separate utterances. in a sample of the final dataset, this assumption fails for roughly 4% of utterance pairs (cf. section 4). long utterances and rare words during dialogue extraction utterances with more than 100 words are removed to ensure that remaining ut-terances are truly conversational and to facilitate neural model training @xcite . as all other parameters in the pipeline, this is adjustable to the needs of the user or task. finally, we remove dialogues with more than 20% rare words (not in the top 100 000), removing noise and facilitating neural model training. dialogues are split randomly into train (90%), validation (5%), and test (5%) datasets, dialogues from the same book are placed in the same split. languages differ only in the dialogue extraction step. the modular pipeline can be easily extended to new languages by specifying conversational delimiters and a minimal implementation of dialogue and turn segmentation, generally adaptable from english. in practice, adapting the english pipeline to other languages ranged between 0-50 lines of python code. optionally further analysis might be needed to check the output of the pipeline and refine the extracting process if needed. delimiters and parameters for other languages were not analyzed as profoundly as for english, leaving room for improvements in future work. we aim to show that good dialogue datasets can be constructed with minimal effort, as a first step towards a high-quality multi-language dataset ensemble. in total, the four filtering steps removed about 12.5% of utterances from the english dataset, detailed in ## error analysis utterance-level to assess the single-turn quality of the english dataset we manually analyzed 100 random utterance pairs with book context. 89 pairs did not contain any errors. remaining utterance pairs contained 1 error type each, out of 2 major and 2 minor types, minor errors occurring in only 1 case each. the extracted text is not conversational in 5 utterance pairs, a consequence of the delimiter threshold and other sources of noise (figure 2 ). utterances of a single speaker were falsely treated as multiple turns in 4 cases, most often because of our assumption that paragraph breaks signal dialogue turns (figure 3 ). and he was singing, too, as he went on with his task; sometimes-"play on, minstrèl, play on, minstrèl, my lady is mine only girl;" in his progress he passed the door of the dormitory of his victim-he paused a moment, and listened attentively. then in a voice of deep anguish he said,-"she can sleep-she can sleep-no ghostly vision scares slumber from her eyes-while-" he shuddered, and passed a step or two on, then pausing again, he said,-"oh, if she, the young and innocent..." dialogue-level errors in whole dialogues exhibit a much greater variety. based on a manual analysis of 50 dialogues in the english dataset we identified 7 error categories (figure 4 ). the following numbers are always out of the 50 analyzed dialogues. 16 dialogues contained 0 errors, 21 contained 1 error type, 11 contained 2 types, remaining dialogues containing 3. we detail the number of dialogues affected by each error type below. we note that this does not constitute a proper statistical analysis. utterances from the same conversation frequently end up in different dialogues (17 cases, example in figure 5 ) because of the dialogue gap threshold. the inverse, a dialogue containing utterances from multiple conversations, occurred in richard curbed an impatient rejoinder, and said quietly, "william durgin had an accomplice." mr. taggett flushed, as if richard had read his secret thought. durgin's flight, if he really had fled, had suggested a fresh possibility to mr. taggett. what if durgin were merely the pliant instrument of the cleverer man who was now using him as a shield? this reflection was precisely in mr. taggett's line. in absconding durgin had not only secured his own personal safety, but had exonerated his accomplice. it was a desperate step to take, but it was a skillful one. "he had an accomplice?" repeated mr. taggett, after a moment. "who was it? figure 5 : a single conversation cut up because of the long paragraph between the two utterances. figure 6: first three and last two utterances are not part of the same conversation, but they were merged because of the dialogue gap threshold. books often contain dialogues between more than two speakers, our second most frequent source of error (14 dialogues). however, such conversations are still coherent and provide useful data for model training. in contrast, the same speaker uttering at least two consecutive turns breaks coherence in 7 dialogues. tackling these issues would have to involve speaker identification (cf. section 6). as in the utterance-level analysis, there were some dialogues (4) in which non-conversational text got mixed in. the remaining errors, delimiter missing and different speakers in same paragraph occurred in only 1 dialogue out of 50. ## conclusion we presented the gutenberg dialogue dataset consisting of 14.8m utterances in english and smaller in future work, we wish to improve heuristics and dataset quality. a classifier could be trained to decide whether two consecutive utterances are part of the same dialogue (looking at non-conversational context). positive and negative examples could be generated by a very low/high dialogue gap, or by manual annotation. speaker-related errors could be addressed using speaker identification. we also hope to extend our dataset to more languages. this involves delimitation analysis, implementation of heuristics, and error analysis. we welcome contributions from the community, as our open-source modular pipeline minimizes the effort required for adding new languages. input trf gpt2 gut zs open ft gut ft stay for dinner with us . eou i 'm afraid i ca n't . i have a dinner party at seven tonight . you see i am going to england the day after tomorrow . eou well i wo n't keep you then . it was very nice of you to come to see me . you 're welcome . thank you . i will . i knew you would n't mind . yes . i hope you feel better . it was my pleasure . sure . take a seat . . . what would you like to know ? eou first i ve heard that your latest book is based on a true story . eou that s correct . it s a murder mystery based on actual murders that book place in florida several years ago . the main characterthe police invesigatoris based on the man who investigated the case . i see . i thought it was a great idea . i could do that . oh i see . thanks a lot . i ll have to check it out . well ? i ve never been involved in an actual murder . what are your chances of catching the killer ? so what happened to the man who investigated the case ? well i do n't eat any meat but i do eat fish and eggs . eou do you eat a lot of vegetables ? eou oh yes and fruit i love fruit . what kind of food do you like ? do you like fruits and vegetables ? and do you eat a good many things ? do you like fruits ? what do you do with your vegetables ? that 's terrific . i ca n't wait to check it out . eou hang on . are you sure you want to keep a nontraditional pet ? eou why not ? it 's so cool . i do n't know . i do n't know what to do . the pets come with a 2 week guarantee . i 've been in the market for two years . i 'm really glad that you came . then let 's go . that 's my boy ! katherine curtis created this activity in 1920 . and in 1984 it was authorized as one activity in olympic games . eou wow mom is more knowledgeable than dad . i must learn from you . eou my little boy you should learn more from me . i 'm almost thirty years older than you . i 'm not sure . i 'm a bit nervous . dad i 'm not a child anymore . i 'm a grownup . you 're a nice old lady mom . okay . i will begin learning when i grow up . what 's the matter with your boy ? it will cost you 10 . eou by the way what 's the time ? eou you can see the time on the meter . i have radio time you know . i know . i do n't know what to do . that 's fine . i 'll see it tomorrow . you had a good time in new york ? you can watch the news . i see . hello . is bob there please ? hello . this is mr . green . yes he is . how 's your business going ? yes my dear but is n't he coming ? yes . he 's gone out to the market . speaking . it seldom rains this summer . i think it 's a bit expensive . why ? you are always on the watch for rain i am sure it does . what is the weather like today ? i think i may have seen it on the kitchen table . i m afraid i have a bit of a small town . it 's great . i 've enjoyed it . i 'm sure you did no i didn t . i didn t see it . what did you see ? thanks . and could i have his email just in case i ca n't get him by phone ? eou sure . his cell phone is 09112223 33 . and his email is lower case t smiththat 's one word at c c w dot com dot t w . eou thank you so much . does he read his emails daily ? yes . he was in the middle of the city . not every day . he just sends email to his friends . sure . he has n't been here since the night of the murder . yes he does . he reads a lot of letters per day . yes he does . he 's a very good student . i can read and write and he keeps in touch with my friends .