Is information sharing between pipelines possible?
Asked by Dewald van Hoven:

Hi Atlassian Community

My team and myself have a use case where we want to share information between different pipelines while they are running. The successful execution of our integration tests is dependant on this information sharing.

We went quite far with how we wanted to implement the information sharing. The mechanism we thought would work is by sharing information between pipelines by using the BITBUCKET_PIPE_SHARED_STORAGE_DIR described here: <https://support.atlassian.com/bitbucket-cloud/docs/advanced-techniques-for-writing-pipes/>. We moved the integration tests to a custom pipe, described here: <https://support.atlassian.com/bitbucket-cloud/docs/write-a-pipe-for-bitbucket-pipelines/>, in order to gain access to the BITBUCKET_PIPE_SHARED_STORAGE_DIR.

We were naive as to how this machanism would work. We discovered that the information sharing can only happen between two pipes running within the same pipeline. Not pipes running in different pipelines.  
Our current best way forward is to set up an external service where we can post to and read from to share information between pipelines.

Is there a better way that we can do this from a mechanism in BitBucket?

Cheers


Answers:
author: Theodora Boudale
body: Hi [@Dewald van Hoven](/t5/user/viewprofilepage/user-id/4221517),

What is possible is to share files that are produced by a step with a following step of the same pipeline, if you define certain files as artifacts:

* <https://support.atlassian.com/bitbucket-cloud/docs/use-artifacts-in-steps/>

If you do that for files generated in one step, these files will become available in the following steps.

Can you combine these different pipelines into one, with multiple steps? Or is that not possible (e.g. do you want to share info between a pipelines build running on develop branch and another one running on master)?

Kind regards,  
Theodora

comments: author: Dewald van Hoven
body: Hi [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086)   

Thanks for your suggestion. Unfortunately it that will not work.

The key for us is to for completely different pipelines to be able to communicate with one another.

We are building marketplace extensions for Confluence. In order to perform our integration tests, an app needs to register with a Confluence instance. Only one instance of an app can register to an instance at a time.

Out of a pool of Confluence instances, a pipeline should connect an instance another app is not connected to for a given run of the integration tests.

We can't find any mechanism to do this communication in the pipelines. The only way we can think of at the moment is to run a small external service which acts as a broker of instances a pipeline can connect to.

Kind regards

Dewald


author: Theodora Boudale
body: Hi Dewald,

Thank you for the info.

I'm afraid there is no way to have info shared between different pipelines builds.

If there is no way to get this info while a build is running and without communicating with other running pipelines:
> a pipeline should connect an instance another app is not connected to for a given run of the integration tests

then the only way would be with a mechanism like the one you suggested.

Kind regards,  
Theodora


author: Dewald van Hoven
body: Thanks Theodora :)


author: Theodora Boudale
body: You are welcome Dewald, please feel free to reach out if you ever need anything else!


Tags: bitbucket-cloud

cloud