Comparing CodiumAI PR-Agent to Copilot for Pull Request
Tags: ai, pullrequest, productivity, codequality
Published: 2023-12-16 02:18:02

## The Power of Collaboration in Code and the Role of Pull Requests in Collaboration

Collaboration lies at the heart of successful software development. The power of collaboration is particularly evident in the collective creation of code. In the dynamic realm of programming – where diverse skill sets converge – the synergy of collaborative efforts enhances the quality, efficiency, reliability, and ingenuity of code production. At the forefront of this collaborative landscape are [pull requests](https://dev.to/karaluton/a-guide-to-perfecting-pull-requests-2b66) (PRs). In this era of interconnected development, understanding the significance of collaboration and appreciating the role of pull requests is essential for building robust and scalable software solutions.

At best, PRs are pivotal mechanisms that facilitate the seamless integration of individual contributions, not only serving as gateways for incorporating changes into a codebase, but also playing a crucial role in fostering communication, code review, and continuous improvement. At worst – and, unfortunately, often in practice – PRs are a tedious purgatory of nits, miscommunication, misalignment, and requests for missing tests.

<figure>
![A comic about how opening a pull request can destroy your confidence](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rfq1grme8g1wgemt8nx3.png)
<figcaption>Comic via <a href="https://www.monkeyuser.com/2021/confidence/">MonkeyUser</a></figcaption>
</figure>

## Using AI to Make Pull Requests Better

What if we could find a way to alleviate some of the painful dysfunctions that arise during the process of opening and reviewing pull requests? Fortunately, we are engineers, and solving problems is what we do best!

Artificial Intelligence (AI) has the potential to offer us new avenues of efficiency in the realm of collaborative coding. With machine learning algorithms and natural language processing capabilities, AI tools can autonomously analyze code changes, identify potential nits or coding style issues, and even suggest improvements, thereby expediting the review process. Moreover, AI-driven systems can assist in mitigating miscommunications by providing contextual understanding, bridging the gap between different coding styles, and ensuring a harmonized codebase. By automating routine checks and flagging missing tests, AI not only streamlines the pull request journey but also allows developers to focus on the more intricate aspects of the code, fostering a collaborative environment that thrives on innovation and problem-solving. In other words, AI can help engineers spend more time doing what they love.

In response to the promise of AI – particularly when it comes to pull requests – GitHub Copilot released [Copilot for Pull Requests](https://githubnext.com/projects/copilot-for-pull-requests/) and CodiumAI released [PR-Agent](https://www.codium.ai/products/git-plugin/). Both GitHub Copilot and CodiumAI are artificial intelligence-powered tools for developers intended to make code generation more efficient. While these tools offer similar features, they are fundamentally different and in this post, we will consider those differences and take a quick tour of each tool.

## Proprietary vs. Open-source

[GitHub Copilot](https://github.com/features/copilot) is a cloud-based artificial intelligence tool developed by GitHub and OpenAI to assist users of popular integrated development environments (IDEs) by autocompleting code. It was launched in October 2021 and – [according to GitHub](https://github.blog/2023-10-10-research-quantifying-github-copilots-impact-on-code-quality/#:~:text=85%25%20of%20developers%20felt%20more,faster%20with%20GitHub%20Copilot%20Chat.) – “85% of developers felt more confident in their code quality when authoring code with GitHub Copilot and GitHub Copilot Chat.” This is probably good news since GitHub Copilot users have to pay to use it. Copilot is only available via subscription at $10/month for individuals.

In March 2023, GitHub launched [Copilot for Pull Requests](https://githubnext.com/projects/copilot-for-pull-requests/) to limited release. (There is currently a waitlist.) Owned by Microsoft, GitHub Copilot does not support any other Git-hosting platform.

[CodiumAI](https://www.codium.ai/products/ide-plugin/) uses generative AI to help developers [build code logic tests automatically](https://medium.com/@sampath.katari/lets-try-testgpt-the-test-cases-generator-codium-ai-bd1b6b108fd1). CodiumAI focuses on code integrity by generating tests that help decipher how code behaves, discover edge cases and suspicious behaviors, and ultimately produce more robust code. Like GitHub Copilot, CodiumAI is available as an extension for Visual Code Studio and JetBrains. Unlike GitHub Copilot, CodiumAI is free for individuals.

CodiumAI also offers an AI solution for pull requests called [PR-Agent](https://www.codium.ai/products/git-plugin/). PR-Agent is open-source, supports other popular Git-hosting platforms besides GitHub, and is available without a waitlist.

Like a lot of engineers I know, I’ve had pull requests languish in review across multiple sprints. It’s often a demoralizing rite of passage for a developer, so I was curious to see how Copilot for Pull Requests and PR-Agent could help expedite reviews and get sprints – and spirits – back on track.

## Creating an Example Pull Request

For this example, I am using a pull request I created in my `testing-strategies-for-python` [repo](https://github.com/liz-acosta/testing-strategies-for-python). I want to add a database to my Build-a-Pug application and I have created a (branch)[https://github.com/liz-acosta/testing-strategies-for-python/tree/add-database-ai-pr-example] and pull request to do so. You can view the pull request [here](https://github.com/liz-acosta/testing-strategies-for-python/pull/4).

## GitHub Copilot for Pull Requests

GitHub Copilot for Pull Requests operates within the GitHub UI for accounts subscribed to Copilot. Copilot for Pull Requests is triggered by using specified Markers within the comments of a PR.

The current available Markers are:

* `copilot:all` all the content, in one go!
* `copilot:summary` a one-paragraph summary of the changes in the pull request.
* `copilot:walkthrough` a detailed list of changes, including links to the relevant pieces of code.
* `copilot:poem` a poem about the changes in the pull request.

Unfortunately, at the time of this writing, I am still on the waitlist for Copilot for Pull Requests, so I can’t test it out with my own repo, but here’s a[ demonstration provided by GitHub](https://githubnext.com/projects/copilot-for-pull-requests#suggestions-for-your-pull-request-description).

<figure>
![Gif of Copilot for Pull Requests in action](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hhwu1obccsw6xzpg9c4j.gif)
<figcaption>Demonstration of Copilot for Pull Requests via <a href="https://githubnext.com/projects/copilot-for-pull-requests/">GitHub</a></figcaption>
</figure>

## CodiumAI PR-Agent

CodiumAI’s PR-Agent [supports functionalities across the most popular Git-hosting platforms](https://github.com/Codium-ai/pr-agent?tab=readme-ov-file#overview) in addition to GitHub. PR-Agent offers flexibility in its implementation: It can be installed and executed locally via a CLI, run using a Docker image, or integrated with a specific Git platform workflow. Or you can get started using it right away with the PR-Agent Git plugin – no waitlist necessary.

PR-Agent supports the following commands:

* `/describe` automatically generates a PR description with a title, type, summary, code walkthrough and labels.
* `/review` provides adjustable feedback about the PR’s main theme, type, relevant tests, security issues, score, and various suggestions for the PR content.
* `/ask` answers free-text questions about the PR.
* `/improve` provides committable code suggestions for improving the PR.
* `/update_changelog` automatically updates the CHANGELOG.md file with the PR changes.
* `/similar_issue` automatically retrieves and presents similar issues.
* `/add_docs` automatically adds documentation to un-documented functions and classes in the PR.
* `/generate_labels` automatically suggests custom labels based on the PR code changes.

To use the PR-Agent Git plugin, all you need to do is comment on a PR with one of the commands and preface it with `@CodiumAI-Agent`.

For example, commenting with `@CodiumAI-Agent /review` on my example PR first elicits an eyes emoji response from CodiumAI-Agent (which indicates acknowledgment of the command) and then a new comment with the following:

> ## PR Analysis

> - 🎯 **Main theme:** Adding database functionality to the application
- 📝 **PR summary:** This PR introduces a database to the application by using SQLAlchemy. It also defines a new model, Pug, and configures the application to use a SQLite database. Additionally, it removes a GitHub action and adds a new dependency to the Pipfile.
- 📌 **Type of PR:** Enhancement
- 🧪 **Relevant tests added:** No
- ⏱️ **Estimated effort to review [1-5]:** 3, because the PR introduces a significant new feature (a database), but the changes are relatively straightforward and well-structured. However, the lack of tests for the new functionality increases the review effort.
- 🔒 **Security concerns:** Yes, because the SECRET_KEY is hardcoded in the code. This could potentially expose sensitive information. It's recommended to use environment variables for such data.
## PR Feedback

>- 💡 **General suggestions:** It's great to see that you've added a database to the application. However, it would be beneficial to include tests for the new functionality. This would help ensure that the database and the new Pug model are working as expected. Also, consider adding error handling for potential database errors.

See the rest of the review [here](https://github.com/liz-acosta/testing-strategies-for-python/pull/4#issuecomment-1855005965).

And because I believe that the more documentation the better, I decided to use `@CodiumAI-Agent /add_docs` and received a new comment with the following:

>**Suggestion:** Proposed documentation
>```suggestion
"""
This is a class representing a Pug in the database.
It has the following attributes:
- id: The unique identifier for the Pug.
- name: The name of the Pug.
- age: The age of the Pug.
- home: The home of the Pug.
- puppy_dinner: The time for the Pug's dinner.
- created_at: The time when the Pug's record was created.
- image: The image of the Pug.
"""
class Pug(db.Model):
```

See the rest of the proposed documentation [here](https://github.com/liz-acosta/testing-strategies-for-python/pull/4#discussion_r1426099685).

But most impressive was the `/ask` command: `@CodiumAI-Agent /ask What would be the best way to refactor the Pug class into the Pug database model?`

>Question: What would be the best way to refactor the Pug class into the Pug database model?

>Answer:
The Pug class has been refactored into a database model quite well in this PR. Here are some suggestions to further improve it:

>1. **Separation of Concerns**: The Pug class is currently defined in the `app.py` file. It would be better to move this class to a separate file, say `models.py`. This way, the application setup and the database models are separated, making the code more organized and easier to maintain.

>2. **Adding Relationships**: If there are other tables in the database that have a relationship with the Pug table, these relationships should be defined in the Pug model. For example, if there is a `Owner` table, and each Pug has an owner, then a relationship should be defined in the Pug model like so: `owner = db.relationship('Owner', backref='pugs')`.

>3. **Adding Class Methods**: It could be useful to add class methods for common database operations. For example, a method to add a new Pug to the database, or to find a Pug by its name. This would encapsulate these operations within the Pug class, and make the rest of the code cleaner and more readable.

>4. **Adding Validation**: You could add validation to the Pug model to ensure that only valid data is saved to the database. For example, you could add a check to ensure that the `age` is a positive integer, or that the `name` is not empty.

See the rest of the answer [here](https://github.com/liz-acosta/testing-strategies-for-python/pull/4#issuecomment-1855015537).

While this may not be the best answer or the answer I was looking for, it’s a good answer to start working from, which is what pair programming is all about and would help me improve this pull request.

## Final Thoughts on Copilot for Pull Requests vs. PR-Agent  

At the moment, CodiumAI’s PR-Agent is not only more accessible and flexible, it also offers more commands and all for free for individual developers. In contrast, GitHub Copilot for Pull Requests is currently waitlist only, supports no other Git-hosting platform but GitHub, does not offer as many commands, and costs $10/month even for individual developers. 

Until I get access to GitHub Copilot for Pull Request, a complete and accurate assessment may not be possible, but I _can_ say this: as an open-source project readily accessible, CodiumAI’s PR-Agent is much more aligned with the spirit of collaborative innovation that pull requests represent.