IITR at B io L ay S umm Task 1:Lay Summarization of B io M edical articles using Transformers (2023)

Abstract: Initially, we analyzed the datasets in a statistical way so as to learn about various sections’ contributions to the final summary in both the pros and life datasets. We found that both the datasets have an Introduction and Abstract along with some initial parts of the results contributing to the summary. We considered only these sections in the next stage of analysis. We found the optimal length or no of sentences of each of the Introduction, abstract, and result which contributes best to the summary. After this statistical analysis, we took the pre-trained model Facebook/bart-base and fine-tuned it with both the datasets PLOS and eLife. While fine-tuning and testing the results we have used chunking because the text lengths are huge. So to not lose information due to the number of token constraints of the model, we used chunking. Finally, we saw the eLife model giving more accurate results than PLOS in terms of readability aspect, probably because the PLOS summary is closer to its abstract, we have considered the eLife model as our final model and tuned the hyperparameters. We are ranked 7th overall and 1st in readability

## introduction scientific paper's growth has increased a lot in recent times. papers in various domains are used to share research data. so it is important to summarize these papers so as to decrease the workload of the researchers and also to reduce the gap between the public and researchers. technical summary still is a difficult text to understand for non-research people and can lead to misinterpretation of information. in the context of bio-medical articles the need for lay summarization is higher because of the more dynamic and difficult terminology and also information misinterpretation having a direct impact on human lives. the shared task of bionlp lay summarization of biomedical research articles aims to improve the tools used for lay summarization for training models which give realistic lay summaries. it has tasks in which data is to be trained on two datasets elife and plos which are two large biomedical article datasets, both varying in sizes of summary and readability of summaries @xcite . our approach tries to build a model to keep the lay summary relevant to the original model and improve its readability. ## related work the models which were existing earlier are in general extractive models which include sentence selection or hybrid models selecting sentences and then summarizing them . @xcite was the first paper describing a model abstractive summarization. laysumm subtask of the cl-scisumm 2020 @xcite shared task series also had submissions that use abstractive summarization. the data insufficiency was found in the task as the sufficient number of annotated lay summaries were not present at the time of the task which will help in getting a more relevant and better lay summary. many other models that are trained for lay summarization also faced the same problems due to a lack of data or data summaries being too large and not suitable for lay summary training. dimsum lay summarization @xcite ) used the cl-scisumm dataset and a bart @xcite baseline along with sentence labels as external supervision signals, data augmentation and got better results of relevance and readability. corpora for lay summarization @xcite introduced the dataset with the articles of elife and plos each with annotated summaries to improve the existing literature of bio-medical articles available to train a lay summarization model. ## datasets there were two datasets in the task elife and plos. plos is the larger of the two datasets, containing 24, 773 instances for training and 1, 376 for validation (the same for each subtask). elife contains 4, 346 instances for training and 241 for validation. each dataset contains train, validation, and test datasets in the form of json files. each sample contains articles with the following structure: @xcite • id-denoting the id of the article. ## data pre-processing datasets as mentioned were in the json format with various headings as keys of json dictionary objects. these datasets are converted into a csv file with each heading as a column. the irrelevant information columns like title, id, year are removed from the csv file. also as per a new text column that contains a fraction(k) of the introduction and its concatenation with the abstract is made which is helpful to train the model with introduction+abstract as the input text. k is a hyper-parameter and it was found out by experimentation that the optimal value was around 0.6. the text is truncated to a max length of 1024 tokens due to the capacity of the bart model. ## experiments we did experiments with different parameters as follows: for the hyper-parameters, we have considered a batch size of 4 due to the memory constraints of gpu. we have performed 15 epochs and chose a model with the least validation loss. we have taken the implementation of bart from hugging-face transformers. we have calculated the rouge1, rouge2, and rougel values for all the experiments. we can see that bart baseline without any fine-tuning gives rouge1 value of around 27% showing that without fine-tuning the summarization might be more technical and fine-tuning is required for summarization to be lay. we can see from the table that using the data of the sections of introduction and abstract combined with bart baseline gives better results than using any of them separately probably due to introduction not having relevant key points of the article and abstract size in elife being very small compared to the summary. also, t5 results are a bit less accurate than the bart model showing bart model performance when fine-tuned with proper domain-related datasets gives very good results also, the plos dataset tested with the elife model has almost the same rouge values as the plos model so considering the plos model has less readability as compared to the elife model we considered the elife model for both the datasets for the final submission. ## conclusion our results show that the bart model fine-tuned gives the best results as compared with other models. also, the plos dataset has an abstract near to its summary so using the elife model for summarization gives more readability.