Improving Compose snapshot tests with Paparazzi
Tags: android, compose, testing, snapshottesting
Published: 2022-07-31 13:10:00

##Update:
Since writing this article I've moved on to yet another improved way of doing this. Instead of generating code with KotlinPoet, it automatically runs Paparazzi tests for each Compose Preview. Check it out in this article:
https://proandroiddev.com/no-emulator-needed-for-screenshot-tests-of-compose-previews-on-ci-d292fa0eb26e

## Background
I previously wrote about snapshot testing Compose with [Shot](https://github.com/pedrovgs/Shot) here: 
https://medium.com/proandroiddev/oh-snap-966bd786b7a4

One problem with these tests is they take too long to run. Starting an actual emulator and running the tests on this emulator in our CI environment takes around 30 minutes. This is too long to include it in our continuous integration pipeline which runs on every pull request. Instead we have a nightly job that runs the snapshot tests and reports to our Teams channel if any of them fail.

This approach has some major drawbacks
- You are no longer "forced" to fix your failing snapshot tests before you merge
- You might discover failing tests too late - even after the changed code has been released
- Someone has to take the time to check the Teams channel every morning and fix the tests

## Paparazzi to the rescue
[Paparazzi](https://github.com/cashapp/paparazzi) is a library which can render your application screens without a physical device or emulator. Not having to deal with the emulator is a massive time saver. 

Back when we initially implemented our snapshot tests, Paparazzi was not an option because it did not support Compose. This all changed recently in the 1.0.0 release.

Paparazzi, like Shot, is quite easy to set up. A Paparazzi test looks something like this: 
```kotlin
class AlertViewPaparazziTest : PaparazziTest() {
    
    @get:Rule
    val paparazziRule: Paparazzi = Paparazzi(
        theme = "android:Theme.MaterialComponents.Light.NoActionBar",
        deviceConfig = DeviceConfig.NEXUS_5.copy(softButtons = false, screenHeight = 1),
        renderingMode = SessionParams.RenderingMode.V_SCROLL
    )

    @Test
    fun alert() {
        paparazziRule.snapshot { AlertPreview() }
    }
}
```

Again like Shot, Paparazzi has one gradle task for recording: 
`$ ./gradlew sample:recordPaparazziDebug`
And one for verifying:
`$ ./gradlew sample:verifyPaparazziDebug`

Unfortunately Paparazzi does not (yet) support taking a screenshot with the exact size of the Composable: https://github.com/cashapp/paparazzi/issues/383
The `screenHeight = 1` and `renderingMode = V_SCROLL` is a trick to get the minimum height needed to take a screenshot of the Composable. Thanks https://github.com/hrach!
There is no similar trick for the width (that I know of, please let me know if you know one!), so we always get the full width of the device which leads to some screenshots looking like this (if you click it you see the full device width):
![Plus button screenshot with full device width](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6w6n92p33rbge58jdfyq.png)
Though it would be nicer without the full width, this is not a big problem. The main point is to compare the current situation to the previously recorded snapshot and see that there are no unintended changes. This works just as well with a full width.

## Rewrite all the tests
To really compare Paparazzi with Shot and see if it solved our problems, we had to rewrite all our snapshot tests. This would be quite boring and repetitive, so we decided to try to generate them instead of writing them manually. 

We always write one snapshot test per @Preview annotation. Because of this, all our snapshot tests look pretty much the same, 
Our idea was something like this:

1. Find every file that has at least one @Preview annotation
2. Create a snapshot test file for it
3. Create one test per @Preview in that file

To do this we created a Kotlin program with a main function and used another library, also from Square, KotlinPoet (https://github.com/square/kotlinpoet) to generate the code.

![The kotlin program with its main function in Android Studio](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ghfimc7pl9hnuucethpv.png)

_Disclaimer: This code is still quite rough, but you get the idea and it does what we needed. I am sure it could be improved further._

```kotlin
/**
 * Finds all files in the components module which have Compose previews
 * and generates Paparazzi screenshot tests for them.
 *
 * The generated tests can then be used to record screenshots with
 * ./gradlew components:recordPaparazziInternalDebug
 *
 * To verify that the current implementation matches the recorded screenshots
 * ./gradlew components:verifyPaparazziInternalDebug
 */
fun main() {
    val path = System.getProperty("user.dir") ?: error("Can't get user dir")

    // Paparazzi does not currently work in the app module: https://github.com/cashapp/paparazzi/issues/107
    // For now this is hardcoded to only check files in the components module.
    // If we pull our compose files out of the app module to a separate module this code has to be updated.
    File(path).walk().filter { it.path.contains("/components/src/main/java") && it.extension == "kt" }.forEach {
        if (it.readText().contains("@Preview")) {
            processFileWithPreviews(it)
        }
    }
}

/**
 * Reads the given file, finds the names of all the functions annotated with @Preview
 * and uses them to generate a Paparazzi test file with one test for each preview.
 */
private fun processFileWithPreviews(file: File) {
    val lines = file.readLines()
    val previewNames = mutableListOf<String>()
    var saveNextFunctionName = false
    var packageName = ""
    lines.forEachIndexed { i, line ->
        if (i == 0) {
            packageName = line.split(" ").last()
        }
        if (line.contains("@Preview")) {
            saveNextFunctionName = true
        }
        if (saveNextFunctionName && line.startsWith("fun ")) {
            previewNames += line.split(" ")[1].removeSuffix("()")
            saveNextFunctionName = false
        }
    }
    val pathString = file.path.replace("src/main", "src/test").split("java").first() + "java"
    val testFilePath = pathString.toPath()
    generatePaparazziTest(packageName, file.nameWithoutExtension + "PaparazziTest", testFilePath, previewNames)
}

fun generatePaparazziTest(packageName: String, fileName: String, path: Path, previewNames: List<String>) {
    val classBuilder = TypeSpec.classBuilder(fileName)
        .superclass(PaparazziTest::class)
        .addAnnotation(
            AnnotationSpec.builder(Suppress::class)
                // KotlinPoet does not let us remove redundant public modifiers or Unit return types for the functions,
                // but we don't mind for generated code as long as the tests work
                .addMember("\"RedundantVisibilityModifier\", \"RedundantUnitReturnType\"")
                .build()
        )
    previewNames.forEach {
        classBuilder.addFunction(
            FunSpec.builder(it.removeSuffix("Preview").usLocaleDecapitalize())
                .addStatement("paparazziRule.snapshot { $it() }")
                .addAnnotation(Test::class)
                .build()
        )
    }
    val testFile = FileSpec.builder(packageName, fileName)
        .addType(classBuilder.build())
        .addFileComment("AUTO-GENERATED FILE by generate_paparazzi_tests.kt\nDO NOT MODIFY")
        .build()
    val nioPath = path.toNioPath()
    testFile.writeTo(nioPath)
}
```

Since we always use the same device specs and Paparazzi rule, we created a parent class and the program above generates sub-classes for that:

```
open class PaparazziTest {

    @get:Rule
    val paparazziRule: Paparazzi = Paparazzi(
        theme = "android:Theme.MaterialComponents.Light.NoActionBar",
        deviceConfig = DeviceConfig.NEXUS_5.copy(softButtons = false, screenHeight = 1),
        renderingMode = SessionParams.RenderingMode.V_SCROLL
    )
}
```

As you can see from one of the comments, Paparazzi does not yet support running tests in the app module: https://github.com/cashapp/paparazzi/issues/107

In our project the composables live in two modules: `:components` for reusable components and `:app` for the screens and single use views. For now, our program only generates paparazzi tests for the `:components` module. We would have to move all our composables out of the app module which can turn into a bigger refactor. This recent issue gives some hope that it could be fixed soon and we might not need to do that: https://github.com/cashapp/paparazzi/issues/524

## Add it to continuous integration
After all the tests had been generated, we recorded the snapshots (60 of them in our `:components` module) and pushed them to our source control. We then added a new step to our CI pipeline which runs on every pull request. Just a simple gradle step that runs the `verifyPaparazzi` task. It is super fast:
![Image showing the verify paparazzi step running in less than 30 seconds](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m5axq44yemsgh2hylnld.png)

## Final thoughts

1. Paparazzi makes snapshot test verification fast enough to run in our CI pipeline on every pull request
2. Recording screenshots locally when adding new UI or changing existing UI is much faster without an emulator
3. Not having to deal with an emulator seems to solve a problem we had in the past where screenshots would be slightly different if they were recorded on a Mac with Apple Silicon or one with Intel
4. It is unfortunate that we can't use Paparazzi in the app module (yet)

We have decided to move forward with Paparazzi and will wait for the app module issue to be fixed or move our composables out of the app module.
 
Please let me know if you have any feedback or questions in the comments or on [Twitter](https://twitter.com/andersullnass).

I leave you with an image of some of our components recorded by Paparazzi: 
![Paparazzi HTML component showing images of some of our components](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/agb2nfnk3ew4clqxb9n9.png)