BItbucket pipelines suddenly becoming very slow today
Asked by Emmanuel Chavane:

I have been using pipelines for several weeks with builds that usually last around 6 minutes. Today they all jumped to at least 20 minutes, and I just killed one that was not completed after 45 minutes.  

Looking at the logs, it seems that the "Build setup" phase is the culprit, used most of the time. Caches are apparently downloaded at normal speed (the only timing information I can read in the phase log).

I see no incident reported on the Bitbucket status page.

So I am not sure where to go from here. It is the first time we have had such an incident, but it's kinda blocking for the team (and our time budget).

Copying Build phase log below in case someone sees something interesting/meaningful.

Thanks

Build setup14m 55s  
+ umask 000  
+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="features/mia-algolia-sync" --depth 50 <https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git> $BUILD_DIR  
Cloning into '/opt/atlassian/pipelines/agent/build'...  
+ git reset --hard 362cec8d48b956ba40c020fc0046b9c31b2c386d  
HEAD is now at 362cec8 [MIA] - ...  
+ git config [user.name](http://user.name/) bitbucket-pipelines  
+ git config user.email [commits-noreply@bitbucket.org](mailto:commits-noreply@bitbucket.org)  
+ git config push.default current  
+ git config http.${BITBUCKET_GIT_HTTP_ORIGIN}.proxy <http://localhost:29418/>  
+ git remote set-url origin <http://bitbucket.org/$BITBUCKET_REPO_FULL_NAME>  
+ git reflog expire --expire=all --all  
+ echo ".bitbucket/pipelines/generated" >> .git/info/exclude  
+ chmod 777 $BUILD_DIR  
Cache "maven": Downloading  
Cache "maven": Downloaded 214.3 MiB in 6 seconds  
Cache "maven": Extracting  
Cache "maven": Extracted in 1 seconds  
Images used:  
build : [eu.gcr.io/janngo-studio-shared/build-machine@sha256:b5692d9eb0362a80a63309cc9933b97432749f24b82310bfbbf995f5c5063fdd](http://eu.gcr.io/janngo-studio-shared/build-machine@sha256:b5692d9eb0362a80a63309cc9933b97432749f24b82310bfbbf995f5c5063fdd)  
./mvnw clean test4m 9s  
Build teardown


Answers:
author: Patrik S
body: Hi [@Emmanuel Chavane](/t5/user/viewprofilepage/user-id/3060418) ,

Thank you for reporting this issue to us.

Based on our investigation, we've identified that images hosted in Google Container Registry (gcr.io) are slow to pull, whilst images hosted on [Artifact Registry](https://cloud.google.com/artifact-registry) (pkg.dev) are not impacted.

Our engineering team is already in contact with Google as we suspect that Pipelines is possibly being rate-limited/throtled by Google which impacts users who are using GCR images on Pipelines.

You can refer to our status page for further updates. - <https://bitbucket.status.atlassian.com/incidents/4gfrv6xyym2t>

Also, since Artifact registry is recommended over Container registry by Google ([See this link](https://cloud.google.com/container-registry/quotas)), we recommend [transitioning to the Artifact registry](https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr) as a workaround while we're working with Google on this issue.

Thank you [@Emmanuel Chavane](/t5/user/viewprofilepage/user-id/3060418) .

Kind regards,

Patrik S

comments: author: Emmanuel Chavane
body: Thanks for the reply [@Patrik S](/t5/user/viewprofilepage/user-id/4603309) . Very reassuring to see you guys are monitoring this forum and are able to provide detailed explanations / workarounds :)


author: Emmanuel Chavane
body: The problem is still here today: absolutely random durations for the build setup phase (from 1 minute to 50 minutes).

Found this thread <https://community.atlassian.com/t5/Bitbucket-questions/Pipelines-Build-Setup-taking-too-long/qaq-p/857558> that describes the same problem and references a ticket, with apparently no solution.

Not sure if someone from Atlassian is reading this, but as a paying customer, watching the numbers of minutes that are used for a build phase (on which I have no control) multiplied by 10, with no way to understand if this is temporary or not, is just not acceptable (in addition to the impact on the dev team productivity).

I have been happy with pipelines for the few weeks we have used it until yesterday, but this issue will be a killer if there is no explanation/resolution very quickly. Will need to move to another CI solution.

comments: author: David Golden
body: FWIW We've been using Bitbucket for our pipelines for a couple years and this has never happened to us before to my knowledge.


author: David Golden
body: Also seeing this. A pipeline that's always been 16-18mins just took 41min.


author: Theodora Boudale
body: Hi [@Nelu Donelly Akejellu](/t5/user/viewprofilepage/user-id/5002192), [@David Golden](/t5/user/viewprofilepage/user-id/1784020), and [@Emmanuel Chavane](/t5/user/viewprofilepage/user-id/3060418),

If you're still experiencing issues, please create a ticket with the support team so we can look into your respective issues.

You can create a ticket via <https://support.atlassian.com/contact/#/>, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

Please make sure to share in the ticket  
- the URL of a build that is taking now more time than previously  
- the URL of a previous build that was taking less time

This way we can check the build logs and extra logs from our side in order to further investigate.

The support ticket will be visible only to the reporter and Atlassian staff, so any build URLs and other private info you post there will not be publicly visible.

Kind regards,  
Theodora


author: Nelu Donelly Akejellu
body: Hello [@Patrik S](/t5/user/viewprofilepage/user-id/4603309) I also have the same issue. A build that takes less than 5 minutes now takes over 120 minutes and it times out. How can this issue be resolved ?


Tags: bitbucket-cloud

cloud