Test failing only in pipeline when mocking filesystem
Asked by Ezequiel:

We have a few tests that use the Javascript mockFs module to mock out the filesystem in node.

For the longest time these tests worked fine. Recently however they started failing only when run inside bitbucket-pipelines.

Here's a minimal reproduction <https://bitbucket.org/ezequiel-wejugo/mockfs-bb-test/pipelines/results/1>

What might be happening?

I imagine it could be to do with some security upgrade or other recent change on Bitbucket's side.


Answers:
author: Theodora Boudale
body: Hi Eze,

Just to give you some context, Bitbucket Pipelines builds run in Docker containers. For every step of the pipeline, a Docker container starts (the build container) using the image you have specified in your bitbucket-pipelines.yml file. The repo is cloned in that container and then the commands of that step's script are executed. When the step is finished, the container gets destroyed.

We have a [guide](https://confluence.atlassian.com/bbkb/debug-pipelines-locally-with-docker-1167698072.html) for debugging Pipelines builds locally. Since you provided a public repo to reproduce, I cloned it and followed this guide.

In Step 2, I used the image **node:20-alpine** in the Dockerfile (the same one as your repo's). In Step 3, I used **--entrypoint=/bin/sh** in the command that starts the container (instead of **--entrypoint=/bin/bash** ). The error that you see in the pipeline also occurs when debugging this locally, when I run the **npm test** command. So, it doesn't look like this is an issue with Bitbucket Pipelines infrastructure.

I would suggest checking the following guide, section "**Scenario 1: Builds were successful earlier, but started failing recently**" for possible causes and troubleshooting steps:

* <https://confluence.atlassian.com/bbkb/troubleshooting-bitbucket-pipelines-1141505226.html>

Please feel free to reach out if you have any questions!

Kind regards,  
Theodora

comments: author: Ezequiel
body: Hi Theodora, thanks for the detailed response.

I run your steps locally and in my case the test passes.

In addition I run the container inside of a GKE kubernetes cluster to eliminate the possibility that running docker on Windows might be changing the result. Here too the test passes.

Assuming npm should detect a version/integrity discrepancy through the lock file, I cannot reconcile your run failing and mine succeeding.  

Here's an attempt at custom integrity checking of my node_modules which is stable across installs:  

```
$ find node_modules/ -type f | sort | xargs cat | sha512sum
1ebdcf6f06c9fd01e2213a8da35b75425e94357db18a1cdd7693eb600edf452a59a1d90ebdd72c4a4bfe8b09d5abe00b7df01fee2b8af988867398cdbf23fdad -
```

Do you get the same checksum?


author: Ezequiel
body: Actually, I just checked this myself by adding a step to my bitbucket-pipelines.yml file and indeed the node_modules in the pipeline and in my local container are the same, yet the test passes one and fails in the other.


author: Theodora Boudale
body: Hi Ezequiel,

I got the same checksum as well.

Can you please confirm if you followed the same steps I am sharing below to debug this locally? This is just to confirm that you haven't done anything differently than me.
> **1.** Create a directory for testing
>
> ```
> mkdir testdocker
> ```
>
> **2.** Clone the repo in this new directory
>
> ```
> cd testdocker
> git clone git@bitbucket.org:ezequiel-wejugo/mockfs-bb-test.git localDebugRepo
> ```
>
> **3.** Reset the HEAD of the repository to the same commit as the failed Pipelines build
>
> ```
> cd localDebugRepo
> git reset --hard 48d74b4
> ```
>
> **4.** Create a Dockerfile outside the clone dir
>
> ```
> cd ..
> touch my.dockerfile
> ```
>
> **5.** Edit the Dockerfile and add this content:
>
> ```
> FROM node@sha256:37750e51d61bef92165b2e29a77da4277ba0777258446b7a9c99511f119db096
> WORKDIR /localDebugRepo
> COPY ./localDebugRepo /localDebugRepo
> ```
>
> I am pinning the Docker image to the same sha that the Pipelines build is using.
>
> **6.** Build the Docker image
>
> ```
> docker build --memory=1g --memory-swap=1g -t myimage/test -f my.dockerfile .
> ```
>
> **7.** Start a container using the image built in the previous step:
>
> ```
> docker run -it --memory=4g --memory-swap=4g --memory-swappiness=0 --cpus=4 --entrypoint=/bin/sh myimage/test
> ```
>
> **8.** Inside the container, execute the commands of the repo's yml file

If you rerun the last successful build in the private repo, does it still succeed or does it fail?

Is the build image SHA the same or different in the last successful build and the first build that started failing (in your private repo)? You can find the image SHA if you expand the **Build setup** section in the Pipelines build log.

Kind regards,  
Theodora


author: Ezequiel
body: Thanks,

That now reproduces the failing test locally. It's to do with some change in the base image vs the version of the same image tag I have in my local cache.


author: Theodora Boudale
body: You are very welcome, Ezequiel!

If you want, you can pin an image by digest in your yml file to ensure that no changes can be introduced.

You can find an example here:

* <https://support.atlassian.com/bitbucket-cloud/docs/use-docker-images-as-build-environments/#Pin-images-by-digest>

Kind regards,  
Theodora


Tags: bitbucket-cloud

bitbucket-pipelines

cloud

javascript