Amazon CodeGuru Reviewer: already time for retirement?
Tags: aws, cloud, machinelearning, security
Published: 2023-08-01 07:22:52

**Or why you might not have heard about CodeGuru Reviewer and that's okay**

_Written by @samudurand_ 

# Introduction

[Amazon CodeGuru Reviewer](https://press.aboutamazon.com/2020/6/aws-announces-general-availability-of-amazon-codeguru) is a relatively recent AWS service (2020) which promises to improve __code quality__ and find __security vulnerabilities__. Just like its partner released around the same time, [CodeGuru Profiler](https://docs.aws.amazon.com/codeguru/latest/profiler-ug/what-is-codeguru-profiler.html), CodeGuru Reviewer is based on __Machine Learning models__ built using AWS' many years of coding experience.

The recent release of a new service in the same series, [CodeGuru Security](https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-codeguru-security-available-preview/), brought the two older services back to my attention. I decided it was time to look into what they offered, and whether it would be worth including in the deployment services we offer to our clients. I was also wondering why it seems to have completely disappeared less than a year after its release, at least from most community blog posts, despite regular feature releases and a new language addition (Javascript).

This article focuses on exploring and demonstrating with real examples __CodeGuru Reviewer__ features, as well as determining its cost and potential usefulness. While reading this article, you should quickly realize that the features are not that great in its current state, and why I think that this service might altogether disappear or be merged into, or replaced by __CodeGuru security__.

![CodeGuruLogo](//images.ctfassets.net/hqu2g0tau160/3i59aQBRDECYW530bmfGbl/b09cecd785aefe71dabc2bc3142e2cf3/CodeGuruLogo.png)

# What is CodeGuru Reviewer?

CodeGuru Reviewer is a __"static" analysis tool__ that will connect to __Git repositories__, and analyze the code of the main branch to figure out if there are critical or hard-to-find bugs or issues, which could have an impact on:

- __performance__: deep loops, heavy logging, inefficient code...
- __maintainability__: bad practices, redundant code...
- __concurrency__: thread-safety, race conditions...
- __security__: hard coded secrets, un-sanitized inputs...

The analysis is performed using a set of [detectors](https://docs.aws.amazon.com/codeguru/detector-library/index.html), built based on machine learning, and classified by:

- __language__: Java, Javascript, Python
- __tags__: like access-control  or resource-leak
- __severity level__: 5 levels from Critical (example: [Hardcoded Credentials](https://docs.aws.amazon.com/codeguru/detector-library/python/hardcoded-credentials/)) down to Info (example: [Stack Trace Exposure](https://docs.aws.amazon.com/codeguru/detector-library/python/stack-trace-exposure/)))
- __Category__: either [Security](https://docs.aws.amazon.com/codeguru/detector-library/python/categories/security/) or [Code Quality](https://docs.aws.amazon.com/codeguru/detector-library/python/categories/code-quality/)

Once it finds such issues, it will highlight and describe them in a human-friendly way in one of the following places:

- in the __CodeGuru AWS console__ (always visible there for all analysis)
- as comments in the Pull Request analyzed (for incremental analysis)
- in the Github Security tab (when integrating via CI/CD)

![2](//images.ctfassets.net/hqu2g0tau160/5TAlVJVwasei23Qf2WUiwh/e2b18432b5312b44fc1f01122b5fda6c/2.png)

## Demo

I have experimented with CodeGuru Reviewer on a couple of my personal repositories. I cannot give access to my AWS account, but if you want to see what PR reviews look like with this tool, have a look at [this PR on my fork of Sl4j](https://github.com/samudurand/slf4j/pull/1), which I created just for this purpose.

## Supported languages and repositories

CodeGuru Reviewer supports the following languages:

- __Java__ (8?)
- __Python__ (3.8?)
- __JavaScript__ (ECMAScript 2021?)

Note: the __versions are educated guesses__, based on the experiences of myself and others with the tool, since the docs don't mention any language version (and doesn't answer [community related questions](https://repost.aws/questions/QUxWrHalCoTHmLfv6nXO66eA/when-will-codeguru-support-java-17)). 

As for Git hosting services, it supports AWS CodeCommit, Bitbucket, GitHub and Amazon S3 (not Gitlab, cloud or on-premise). 

# How to use it?

CodeGuru Reviewer can be used in one of two ways:

- directly as a reviewer on your repository
- as part of your CI/CD pipeline

## As a reviewer

Using that approach, the configuration is simple:

1. configure CodeGuru as __a reviewer__ in your repository
2. let CodeGuru automatically run a __full repository scan__ (which can also be triggered manually) on the main branch
3. check the findings in the AWS console

![3](//images.ctfassets.net/hqu2g0tau160/7j08qPLJWB71G4GXU4EPkt/37b471cd1ede8d3ad4d8e87287fb1d1e/3.png)

![6](//images.ctfassets.net/hqu2g0tau160/2WDc1GqXR1WN7CqoNF78Sb/72b1ef6ed32bb3403496a1041675b1a8/6.png)

Once CodeGuru has access to your repo, any Pull Request will trigger an __incremental__ review, focused on the code changes you made. The findings will be shown as:

- PR comments and
- browsable items in the __AWS console__ (just like full scans)

![4](//images.ctfassets.net/hqu2g0tau160/6Qhol02enwZDgv3vwCbTrl/83948baea4ccb0aad53ac3de42078932/4.png)

## As part of the CI/CD

As an alternative, CodeGuru Reviewer can run reviews [triggered by Github Actions](https://aws.amazon.com/blogs/devops/integrating-with-github-actions-amazon-codeguru-in-your-devsecops-pipeline/). The findings will then be available in the Github "Security" tab.

![5](//images.ctfassets.net/hqu2g0tau160/4M0myekrSUIAyULHNJJsgQ/37cda164a311f90fd1ff2a5746e24b35/5.png)

You might want to use this approach if:

- you want to __control__ when PR reviews should happen (you prefer to avoid new comments at every commits) 
- you want to __integrate__ CodeGuru as part of a pipeline

However this approach has a few drawbacks:

- you can only use it on Github, or on other Github actions supporting pipelines (AWS recently made GH actions [usable on CodeCommit](https://aws.amazon.com/about-aws/whats-new/2023/07/aws-codebuild-github-actions/))
- the code will be downloaded to an __S3 bucket__ that you have to __cleanup yourself__, which can end up being pricey if the codebase is very large

# Pricing

Now that we know what we can do with CodeGuru Reviewer, let's see how much it would cost us to use the pricing described in the [docs](https://aws.amazon.com/codeguru/pricing/).

The pricing model is based almost solely on the size of the codebase. The only other criteria is the number of __full scan run__, which shouldn't be more than twice a month. The number of __incremental scan__ (in PRs) does not affect the price.

Let's take an example, and say that we have a project with:

- 3 repositories containing each 150k lines of code in their largest branch
- 3 full scans performed per month on 1 of the repositories instead of the free maximum of 2 

![7](//images.ctfassets.net/hqu2g0tau160/PpAL8qJAzlMki91azw2pa/210817b8319eda9aac5e9a8f48f96df5/7.png)

The result is a monthly cost of __$150__. A good way to see if this price is expensive or not is to compare it with other static analysis tools offering similar code quality and security checks.

The most popular competitor among those is probably [SonarQube](https://www.sonarsource.com/plans-and-pricing/#sonarcloud), which offers for its cloud hosting option and 500k lines of code, a similar price of $150. It is worth noting that SonarQube offers a __much larger set of features and languages supported__, so the comparison only by price is not exactly fair.

# Where it falls short

## Documentation

The [documentation](https://docs.aws.amazon.com/codeguru/latest/reviewer-ug/welcome.html) is not consistent and often out-of-date:

- [some parts](https://aws.amazon.com/codeguru/faqs/#:~:text=Which%20programming%20languages%20are%20supported,%2C%20Python%2C%20and%20JavaScript%20code.) mention Javascript while the rest of the docs only mention [Java and Python](https://docs.aws.amazon.com/codeguru/latest/reviewer-ug/welcome.html)
- no language version is mentioned anywhere
- on the main page of CodeGuru, CodeGuru Reviewer is not listed directly anymore, suggesting changes in progress behind the scenes

## Languages

Only 3 languages are supported, and not their latest versions. For Java this becomes quickly a blocker, since the supported version is only 8 (while the most recent LTS is 17), and unsupported code breaks the analysis.

Another point worth mentioning is that CodeGuru [can only review one language](https://docs.aws.amazon.com/codeguru/latest/reviewer-ug/create-code-reviews.html) at a time. If your project has multiple languages, __you will not be able to select__ which you want to scan. It will be determined automatically by the number of files. From the documentation:

> If a repository contains Java and Python files, then CodeGuru Reviewer generates recommendations for the language for which there are more files. [...] If the number of Java and Python files is the same, then only Java recommendations are generated.

## Findings

While the __security__ findings can be pretty elaborate and helpful, the __code quality and performance__ focused findings are not that impressive and can often be detected by more basic or powerful tools like [SonarQube](https://www.sonarsource.com/products/sonarqube/) (paying) or [PMD](https://pmd.github.io/) (free). To see what I mean you can have a look at the list of [Java code quality detectors](https://docs.aws.amazon.com/codeguru/detector-library/java/categories/code-quality/), which is pretty short and contains a lot of simple findings like:

- [general syntax error detection](https://docs.aws.amazon.com/codeguru/detector-library/java/file-syntax-error/)
- [constructor deprecation in favour of builders](https://docs.aws.amazon.com/codeguru/detector-library/java/client-constructor-deprecated-rule/)
- [simple SQS misconfigurations](https://docs.aws.amazon.com/codeguru/detector-library/java/amazon-sqs-enable-long-polling/)

There is also a few detectors capable of finding [more elusive bugs](https://docs.aws.amazon.com/codeguru/detector-library/java/aws-custom-retries/) but they are not many.

# Conclusion

CodeGuru Reviewer offers several interesting features with useful recommendations, especially in the [security domain](https://docs.aws.amazon.com/codeguru/detector-library/java/categories/security/). The findings it provides for that area are not easy to identify with standard static analysis. The fact that it can integrate as a normal reviewer or via CI/CD is also convenient and makes it easy to quickly generate results.

However, when it comes to code quality, there are many alternative tools that provide similar insights, for more languages and at a lower cost, or even for free (if you are willing to plug and configure together multiple open source tools). Furthermore, the poor quality of the documentation in some areas (which is not common for most AWS services), and the recent documentation updates removing mentions of CodeGuru Reviewer, give the impression that this __tool is not the focus of the AWS team.__

The final hint that something will probably happen soon was the announcement of the [CodeGuru Security service](https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-codeguru-security-available-preview/), and the modification of the [main CodeGuru page](https://aws.amazon.com/codeguru/) to point instead towards this new service. CodeGuru Security at first glance seems to be a modified version of the Reviewer, with a focus on security. This is pure speculation at this point, but I suspect that CodeGuru Reviewer will soon be __either dropped or maybe merged__ into the newer and more focused __CodeGuru Security__.

So as a final advice, I would recommend not investing your time in CodeGuru Reviewer since its current state does not make it a very attractive tool, and even suggest that the time invested might be wasted if the tool is not maintained or discreetly disappears. However, I would keep an eye out for the future evolutions of [CodeGuru Security](https://aws.amazon.com/codeguru/), which seem much more promising.