Automated and Context-Aware Code Documentation Leveraging Advanced LLM s (2025)

Abstract: Code documentation is essential to improve software maintainability and comprehension. The tedious nature of manual code documentation has led to much research on automated documentation generation. Existing automated approaches primarily focused on code summarization, leaving a gap in template-based documentation generation (e.g., Javadoc), particularly with publicly available Large Language Models (LLMs). Furthermore, progress in this area has been hindered by the lack of a Javadoc-specific dataset that incorporates modern language features, provides broad framework/library coverage, and includes necessary contextual information. This study aims to address these gaps by developing a tailored dataset and assessing the capabilities of publicly available LLMs for context-aware, template-based Javadoc generation. In this work, we present a novel, context-aware dataset for Javadoc generation that includes critical structural and semantic information from modern Java codebases. We evaluate five open-source LLMs (including LLaMA-3.1, Gemma-2, Phi-3, Mistral, Qwen-2.5) using zero-shot, few-shot, and fine-tuned setups and provide a comparative analysis of their performance. Our results demonstrate that LLaMA 3.1 performs consistently well and is a reliable candidate for practical, automated Javadoc generation, offering a viable alternative to proprietary systems.

## related works numerous studies have been conducted on code documentation, starting with conventional rulebased techniques and advancing to pre-llm ai models like lstm and early transformer-based methods. ahmad et al. @xcite evaluated the transformer model, which learns code representation for summarization through a selfattention mechanism. to summarize c# code snippets, code-nn, an lstm-based model with an attention mechanism, was proposed by iyer et al. @xcite ). an early neural attention model for code summarization was presented by allamanis et al. @xcite . it incorporates a dual attention mechanism and convolutional features into a recurrent encoder-decoder architecture. however, they were constrained by low flexibility, poor generalization, limited memory, and an insufficient understanding of the content of the code. modern llms are increasingly applied to automated code documentation, yet existing studies reveal critical limitations. for instance, @xcite used codex for multi-language documentation, achieving a modest bleu score of 20.6, while diggs et al. @xcite developed specific prompting strategies and evaluation rubrics for generating comments in legacy systems. similarly, geng et al. @xcite focused on satisfying developer goals by pre-training models with code-comment pairs. despite these efforts, performance remains a key issue, with kneidinger et al. @xcite demonstrating that even a powerful proprietary model like gpt-4 produces unsatisfactory results for class-level documentation. furthermore, a major gap persists: existing research has almost exclusively used proprietary models, overlooking the application of open-source llms specifically for javadoc generation. this leaves developers who require flexible and transparent solutions without a viable alternative to paid, closed-source apis. agent-based approaches have also shown promise in this area. for instance, repoagent is an open-source system that excels at repositorylevel documentation, though it is limited to python projects and lacks template support @xcite . similarly, commercial agents like github copilot have demonstrated significant efficiency gains, saving up to 50% of the time developers spend on documentation tasks @xcite . although llm-powered agents have strong capabilities, there are still significant obstacles to overcome before they can be used in the real world. data security is still a top priority, particularly when managing private or confidential data without the right protections. another challenge is customization, since many llms find it difficult to adjust to domain-specific requirements without prompt engineering or fine-tuning. furthermore, delay can impact usability, especially in interactive environments where users anticipate prompt responses. deploying dependable and safe ai-driven bots requires addressing these constraints. several existing datasets are relevant to code documentation generation, but possess limitations for our specific focus on template-based javadoc. for instance, hasan et al. introduced codesc (hui et al., 2024), a large dataset containing over 4.2 million java methods paired with natural language descriptions. despite its size, codesc suffers from noise and inconsistencies, lacks necessary contextual information for javadoc generation, does not provide template-based documentation, and offers limited coverage of modern java constructs. similarly, codexglue-concode (iyer et al., 2018) provides java code snippets and natural language descriptions but shares identical drawbacks when considering template-based documentation needs. codesearchnet (husain et al., 2020), introduced by husain et al., covers multiple programming languages but also exhibits issues like noise, potential duplicate entries, a lack of modern java features, and a primary focus on function-level summaries rather than structured documentation. while the stack (kocetkov et al., 2022) represents a massive collection (over 6tb) of permissively licensed source code across many languages, it is a general code corpus and is not specifically curated or structured for the task of documentation generation. ## experiment this section presents our experimental setup and the corresponding model results. ## conclusion we introduced a new dataset consisting of reference documentation for methods, lambdas, packages, and class references, designed to provide richer context for fine-tuning publicly available models for javadoc-style generation. additionally, we trained models such as llama-3.1-8b, gemma-2-9b, phi-3.5-mini-instruct, mistral-7b-v0.3, and qwen-2.5-coder-3b on our dataset. furthermore, we assessed the performance of each model across four different evaluation stages (zeroshot, one-shot, few-shot, and fine-tuned) and measured their effectiveness using bleu and rouge evaluation metrics. finally, we provided a comprehensive analysis of their performance, highlighting how pre-training influences initial capabilities and how fine-tuning on a targeted documentation generation dataset affects their performance, with llama-3.1-8b showing consistently strong results after fine-tuning.