P aper P ersi C hat: Scientific Paper Discussion Chatbot using Transformers and Discourse Flow Management (2023)

Abstract: The rate of scientific publications is increasing exponentially, necessitating a significant investment of time in order to read and comprehend the most important articles. While ancillary services exist to facilitate this process, they are typically closed-model and paid services or have limited capabilities. In this paper, we present PaperPersiChat , an open chatbot-system designed for the discussion of scientific papers. This system supports summarization and question-answering modes within a single end-to-end chatbot pipeline, which is guided by discourse analysis. To expedite the development of similar systems, we also release the gathered dataset, which has no publicly available analogues.

## introduction scientific papers are a crucial part of academic research and are used to disseminate new findings, theories and knowledge to the wider community. at the same time, rapid scientific progress makes it challenging to keep up with new technologies without spending a lot of time reading papers. while traditional summarizing services like elicit foot_0 and scholarcy foot_1 can be helpful, they often unable to explain sophisticated and complex concepts. more advanced solutions, such as explainthepaper foot_2 , have emerged to address this limitation as they can elucidate user-highlighted text, but also require the user to read the article beforehand. dialogue systems are an alternative capable of combining extractive and generative approaches. grounding-based approaches were suggested to eliminate issues associated with the hallucinations of lms @xcite . the release of chatgpt foot_3 has propelled chatbots to the forefront of text data processing. the chatgpt api and proprietary solutions have enabled the creation of communication services like chatpdf foot_4 and xmagic foot_5 . however, these are services with a closed architecture and paid for. interestingly, there are no publicly available open systems that do not use the api of llms. one of the reasons, is the lack of open-source datasets for dialogue on scientific grounding. to bridge this gap, we present paperpersichat foot_6 , a chatbot pipeline designed for the scientific paper domain. it capable of communicating on the basis of a userselected paper by providing summaries and answering clarifying questions. our second contribution is the training dataset that can be used to develop solutions for similar tasks. our code is available at https://github.com/ai-forever/paper_persi_chat . ## related work the incorporation of external information, referred to as grounding, has been shown to enhance the quality of the generation by improving the factual component. several approaches utilize knowledge bases or web mining @xcite , while others focus on extracting information from individual documents. @xcite proposed a transformer-based model which retains context semantics while sacrificing text details due to the use of averaging word embeddings. unigdd @xcite and dialki @xcite systems also consider document-grounded generation but are limited by context length or investigated for task formulations different from ours. the main limitation of such systems is the lack of training datasets. cmu dog @xcite was proposed for grounding-based movie conversations but contains few documents which complicates generalization. larger datasets, such as the wizard of wikipedia @xcite , are labeled more roughly and are not tailored to the scientific domain either. at the same time, recent approaches employ synthetic training datasets collected via chatgpt @xcite . we follow this idea and propose a dataset collection process, which we used to train our pipeline later. ## system overview figure 1 shows the general architecture of the pa-perpersichat system. the chatbot discusses paper segments step by step, with each segment containing one or several sections of the paper. the dialogue ends when all segments have been discussed or too much negative feedback has been received. for each segment, the chatbot firstly suggests discussing it. if the suggestion is accepted, it provides a short summary and proceeds to the questionanswering session. otherwise, the chatbot moves to the discussion of the succeeding segment. for each question query, the qa module attempts to extract grounding from the current segment. however, if this fails, it continues to look over all other segments. in case when qa module can't find an answer in the entire paper, it informs the user about that. if the user's query is not a question, the system moves to the discussion of the following segment. further details are described in section 5. ## data there is a lack of publicly available datasets for training the dialogue systems with scientific text grounding. since manual markup requires significant resources, we constructed the dataset automatically. as the source, we used 63,321 computer science papers from the semantic scholar open research corpus published at top science conferences between 2000 and 2021. we utilized its subset to collect our dataset, which consists of two parts: instances collected via openai's davinci or chat-gpt foot_7 . the davinci model processed complex instructions and tried to produce the part of the dialogue related to the whole segment discussion part (see figure 1 ). in this way, we collected 3,588 raw outputs and each of them was processed further into a summary and dialogue turns. all these summaries were used to train the summarization submodule. further filtering was done to remove unparsed outputs, short dialogues and dialogues with inconsistent structure (including incorrect speaker order). this yielded a set of 2,817 dialogues that were used to train the models from the qa session module. to construct qualitative dialogues for qa, and also to manage the inputs of the dialogue participants, we used two chatgpt models talking to each other. the resulting dataset totals 2,817 dialogues produced by davinci and 8,787 dialogues produced by chatgpt, with an average of four turns per dialogue. we have made this dataset publicly available via https://huggingface.co/datasets/ai-forever/ paper_persi_chat . ## submodule details this section provides details about submodules of the paperpersichat pipeline. dialogue discourse flow management (dm) this component is employed to classify the user's reaction and navigate to the pertinent pipeline steps. it is composed of two models: a dialogue discourse parser and an agreement classifier. to acquire the discourse parser, we trained the parser proposed by @xcite from scratch on cdsc @xcite . to classify the last relation in the dialogue, the pipeline passes the last ten utterances of the dialogue history as the parser input. in this pipeline, we consider only the following dialogues acts: agreement, disagreement, question and negative reaction. since the discourse parser has broad classes and can not distinguish well "yes" and "no" cases, an auxiliary agreement classifier based on the sbert model @xcite was trained using the open-source cdsc and iac @xcite corpora. summarization this component briefly summarizes the current segment text. here, the bart model @xcite was selected for generation due to its state-of-the-art performance in similar tasks. we trained it using the 3,204 samples and started from distilbart 9 . evaluation of the model on 356 test samples from our dataset yielded rouge-1 score of 51.43, rouge-2 score of 30.80, and rouge-l score of 40.97. ## user interface figure 2 depicts a screenshot of a sample dialogue between the user and the proposed chatbot. here, the bot suggests discussing the related work section; the user agrees and the system moves to the qa session. if during the session the bot cannot find a grounding for a question, it informs the user that there is not enough information in the paper. the qa session continues until the user ceases asking questions, after which the dialogue advances to the next section. during the dialogue, the user enters his message in the corresponding field and then the dialogue history above the input field is updated with the addition of the last user's query and the bot's response. then the process repeats. the auxiliary menu to the left, illustrated in figure 3 ), assists the user in selecting a paper for discussion, switching to another segment (via radio buttons), or clearing the dialogue history. additionally, the menu provides short instructions to facilitate communication with the bot. there are several options to select a paper for discussion: ## conclusion we have presented paperpersichat, chatbot based only on open-source models and capable of engaging in conversations about scientific papers. for each paper segment, the bot offers the user an opportunity to get a summary and moves to the qa session mode in the case of agreement. the dialogue flow is controlled by a discourse analyzer. we also presented a novel dataset to facilitate the development of similar systems. future work includes refining individual submodules and dialogue management to promote greater flexibility.