Read between the lines - Functionality Extraction From README s (2024)

Abstract: While text summarization is a well-known NLP task, in this paper, we introduce a novel and useful variant of it called functionality extraction from Git README files. Though this task is a text2text generation at an abstract level, it involves its own peculiarities and challenges making existing text2text generation systems not very useful. The motivation behind this task stems from a recent surge in research and development activities around the use of large language models for code-related tasks, such as code refactoring, code summarization, etc. We also release a human-annotated dataset called FuncRead, and develop a battery of models for the task. Our exhaustive experimentation shows that small size fine-tuned models beat any baseline models that can be designed using popular black-box or white-box large language models (LLMs) such as ChatGPT and Bard. Our best fine-tuned 7 Billion CodeLlama model exhibit 70% and 20% gain on the F1 score against ChatGPT and Bard respectively.

## introduction large language models (llms) are known to perform really well on many text2text @xcite generation tasks such as summarization @xcite ), translation @xcite , etc. because of this success, there is a growing research interest in applying llms in novel task settings such as explaining complex codes, generating new recipes, simplifying contents, etc foot_0 . in this paper, we introduce another novel task called functionality extraction from git readme files -a variant of text summarization task @xcite ) that detects all the functionalities supported by the corresponding application software. this task can also be seen as a variation of a question-answering (qa) @xcite task where the question like list all functionalities is fixed. the motivation to introduce automatic functionality extraction from git readme files stems from the requirement of application code refactoring to decompose a monolith application into functional microservices. here each microservice is a collection of closely connected application artifacts (programs, tables etc.) supporting a common functionality @xcite @xcite . current microservice recommendation systems rely a lot on subject matter experts (smes) and falls short to correctly group artefacts since they do not have reference list of functionalities. but many application git readme files tend to contain capture different functionalities foot_1 of the underlying software code base foot_2 along with other implementation details like what it does, how others can use it, licensing, etc., @xcite . as an example, the readme file of the daytrader application foot_3 discusses the application overview, the technology used, licensing terms, etc., and in between discusses four functionalities as highlighted in figure 1(a) . recently, @xcite focused on leveraging llm to generate sections of readme.md like "about" section (brief 1-2 line summary of repo) but they do not aim to list all the functionalities. extraction of the application functionalities from such readme files is not straightforward. the functionalities may not be always structured and might spread across multiple paragraphs and lines. therefore, there is a need for an intelligent system that can parse the text, understand functionality expressions, de-duplicate, and list them. to tackle this first-of-its-kind task, we also introduce and release a new dataset called funcread that will help the community to benchmark their functionality understanding module and refactor monolith applications into discovered functional microservices. the key contributions of this paper are as follows. the funcread dataset is a first-of-its-kind dataset that consists of functionalities described in the readme files. these functionalities were handcurated by human annotators after carefully reading the file. for each readme file, the functionalities are annotated in two formats -extractive and abstractive. extractive functionalities are segments of the text or span from the readme file; whereas abstractive functionalities are the self-explained versions of the corresponding extractive functionalities, written in the annotator's own words. each of these format outputs are presented in the form of a list. the dataset consists of unique 2101 human annotated github readme files. ## task modelling for modeling purposes, one can view the functionality extraction as a generation task. in the generation mode, the goal is to generate a list of functionalities from a given readme file. as ours is the first-of-its-kind dataset, we used chatgpt and bard models known to perform really well on most nlp and code tasks even in zero-shot setting as a baseline for our task. among many prompts, the following prompt "list all the features from above text. each features should be in individual line without headings. each features should be in individual line without headings. do not include features related to license" provided the best results. the actual list of prompts tried on chatgpt and bard can be found in section 6.5. we wanted to study if task specific small sized models can provide competitive results. for this we considered mix of nl and code model variants like 1b and 7b starcoderbase, 2.7b phi-2 and 7b llama-2 and codellama. for fine-tuning, we pre-processed the readme data through the steps listed in section 2.1. next, we append it with "\n##features##\n" as the task designator prompt followed by the hu-man annotated list of functionalities corresponding to that readme file. for inference, we simply appended the task designator prompt to the readme text and then allowed the model to complete sequence to generate list of functionalities. ## experiments and results for our experiments, we divided the funcread dataset into train, validation, and test sets comprising 1801, 100, and 200 samples respectively. it also has ranking functions based on peewee, utility function to add fts5 auxiliary functions, and an fts5 aux function implementation. the module has a sample tokenizer for fts3,4 and fts5. one-to-one matching many-to-one matching ## conclusion we introduced a novel task functionality extraction from git readme files and studied on a new dataset curated from public repositories to demonstrate reliability of small sized fine-tuned llms. we organize the appendix to cover the following :