Avoid downloading the docker image atlassian/google-gke-kubectl-run each time it's used as a pipe
Asked by Mossroy:

The atlassian/google-gke-kubectl-run docker image is used by the GKE pipe of Atlassian

When you use it as a pipe in a pipeline (see <https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/google-gke-kubectl-run>), it's downloaded the first time.

If it's used again in the same step, it's kept from local cache.

But if it's in another step (or another run), it's downloaded again (which slows down the pipeline)

I thought adding a cache would be enough:

```
  caches:
    - docker
```

but it did not seem to help


Answers:
author: Tommo
body: Personally I played around with the caching but it was faster to simply download the fresh image. According to <https://hub.docker.com/r/bitbucketpipelines/google-gke-kubectl-run/tags> it's doubled in size from 2.1.0 354.54MB to its current 3.0.0 753.52MB. Maybe the bigger size is better to cache to a local hard drive, but if Dockerhub serves the image from SSD in global data warehouses it takes something like 12s, and Bitbucket serves cache from SSD and it takes around 16s, then what's the difference, really? Just get the clean image again if it's a fairly popular one. I assume that caching would make sense if you have a customised image, but I'm not seeing benefits for a public one.


author: Theodora Boudale
body: Hi [@Mossroy](/t5/user/viewprofilepage/user-id/899952),

Are you building a docker image in any of the steps that have a docker cache definition?

If I understand correctly, you are using the same pipe in different steps and pipelines in your bitbucket-pipelines.yml file, is that correct?

If it's the same pipe, do you use the same version of the pipe in all these different places?

Do you have a definition for the docker cache in every step using the pipe?

Kind regards,  
Theodora

comments: author: Mossroy
body: Here is an excerpt of the pipeline:
> definitions:  
> steps:  
> - step: \&build  
> name: 'Build'  
> services:  
> - docker  
> caches:  
> - docker  
> artifacts:  
> - (...)  
> script:  
> - (...)  
> - docker build (...)
>
> - step: \&push-docker-image  
> name: 'Push docker image to registry'  
> (...)
>
> - step: \&deploy  
> name: 'Deploy'  
> deployment: XXX  
> artifacts:  
> download: false  
> script:  
> (...)  
> - pipe: atlassian/google-gke-kubectl-run:2.2.0  
> variables:  
> (...)  
> pipelines:  
> default:  
> - step: \*build  
> - step: \*push-docker-image  
> - step:  
> \<\<: \*deploy  
> name: "Deploy to Dev"  
> deployment: dev  
> - step:  
> \<\<: \*deploy  
> name: "Deploy to Staging"  
> deployment: staging  
> trigger: 'manual'

So we use the same pipe (same version) several times in the pipeline, and we run this pipeline many times a day.

Currently, it downloads the docker image atlassian/google-gke-kubectl-run:2.2.0 every time, which looks not optimal.

I initially added the cache "docker" on step "deploy" but it did not help.

As I explained in my other answer, I suspect it's simply because the cache was already made by the "build" step, and is never updated.

So I probably simply need to use a custom cache, with a different name


author: Theodora Boudale
body: Hi [@Mossroy](/t5/user/viewprofilepage/user-id/899952),

Thank you for the details. You are right, if a docker cache is saved from the docker image you are building, then no docker cache will be saved for the pipe.

You can define a second cache in this case.

I am posting below an example yml file with **(1)** a definition of a custom cache for the image you are building and the commands you need to use to generate and load the cache and **(2)** a docker cache definition for the pipe.

I used your excerpt but removed some definitions like artifacts and some steps for simplicity:

```
definitions:
  steps:
    - step: &build
        name: 'Build'
        services:
          - docker
        caches:
          - ourdockerimage
        script:
          - docker load -i docker-image-cache/* || echo "No cache" 
          - docker build .
          - mkdir -p docker-image-cache && docker save $(docker images -aq) -o docker-image-cache/cache.tar
          - docker image ls -aq
    - step: &deploy
        name: 'Deploy'
        services:
          - docker
        caches:
          - docker
        script:
          - pipe: atlassian/google-gke-kubectl-run:2.2.0
            variables:
              (...)
  caches:
    ourdockerimage: docker-image-cache

pipelines:
  default:
    - step: *build
    - step: *deploy
```

<br />

Make sure to use the following for every step that is using **pipe: atlassian/google-gke-kubectl-run:2.2.0**

```
services:
  - docker
caches:
  - docker
```

Remove also any saved caches when you make the changes I suggested, so that the two caches will get populated for the image you build and for the pipe.

I hope this helps, if you have any questions, please feel free to let me know.

Kind regards,  
Theodora


author: Mossroy
body: Thinking again about that, it might be because I already had a populated docker cache.

From what I understood, a bitbucket pipeline cache is never updated. It's only cleared after one week.

It might be the reason why it was not efficient here.

Maybe I should create a custom cache, with a different name, based on <https://support.atlassian.com/bitbucket-cloud/docs/cache-dependencies/#Custom-caches-for-other-build-tools-and-directories> ?


Tags: bitbucket-cloud

bitbucket-pipelines

cloud

google

kubernetes