Access multiple Bitbucket repositories from a single Pipeline
Asked by Michael Atkinson:

We currently have two repositories: one that contains the source code for our application, and another that contains scripts to build and deploy that application to the cloud (among other things). I want to set up two Pipelines: one in the application repo that simply triggers a Pipeline in the build repo upon commit to a specific branch, and another in the build repo that is triggered from the application repo and simply grabs the source from the application repo and executes the build process as normal. This is not a problem on our local machines, but it seems the only solution for Pipelines it to write a custom pipe that uploads the source to S3 (for example) in the application repo, and then pulls that same source down in the build repo. Is there anyway to use something like an app password to grant the build repo access to the application repo? Or is Pipelines just not the tool for this job?


Answers:
author: Theodora Boudale
body: Hi Michael,

It is possible to clone another repo in Pipelines. Instead of using app passwords, you can use SSH. The following blog post explains how you can set this up:

* <https://bitbucket.org/blog/cloning-another-bitbucket-repository-in-bitbucket-pipelines>

We also have a pipe that can trigger a pipeline for a different repo:

* <https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/trigger-pipeline>

You can use this in the yml file of the application repo in order to trigger a pipeline for the build repo. There is also a variable WAIT for the pipe that you can set to 'true' or 'false', depending on whether you want the pipeline in the application repo to wait for the pipeline in the build repo to complete or not.

I'm not sure if you're already aware of this pipe and if you are using it already, I thought I'd mention it just in case.

Please feel free to let me know if you have any questions.

Kind regards,  
Theodora

comments: author: Michael Atkinson
body: This is exactly what I was looking for, thank you Theodora!


author: Theodora Boudale
body: You are very welcome Michael, I'm glad to have helped!


author: Sameeksha Chepe
body: This is useful for the CI/CD I am setting up too. Thanks for this answer.

However, I need to trigger a pipeline from another repo and use the build created by this pipeline.

Will I be able to use 'artifacts' from the nested pipeline into the current pipeline which triggers the nested one?


author: Theodora Boudale
body: Hi [@Sameeksha Chepe](/t5/user/viewprofilepage/user-id/4481443),

Artifacts can only be used by subsequent steps of the same pipeline.

I'm afraid that it is not possible to transfer files from one Pipelines build to another one with the pipe.

One thing you could do would be upload these files to another storage server during the Pipelines build of the second/nested repo, and then when the pipe finishes, download them in the build that triggered the nested one.

Kind regards,  
Theodora


author: Sameeksha Chepe
body: Thanks for the quick reply, [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086)

Another variation of the question - I understand now that it's not possible using the pipes.

However, we can save artifacts which are preserved for 14 days. Is it possible to use a command like cURL or something to copy the artifact from the client to the server repo during the pipeline run of of the server repo

So the steps could be

\* A pipeline runs on the client repo for a new commit on a specific branch. The build is stored as an artifact on the bitbucket artifacts storage.

\* Whenever the pipeline for the server repo triggers, it copies the artifact from the latest build on client repo and creates a unified build of the server side and the client side code.


author: Theodora Boudale
body: Hi [@Sameeksha Chepe](/t5/user/viewprofilepage/user-id/4481443) ,

I'm afraid that it is not possible to access artifacts via API.

Another way to work around this would be to upload the file to the **Downloads** section of the repo and then download it from there. My colleague Mark posted more details on the other question you have created here: <https://community.atlassian.com/t5/Bitbucket-questions/How-to-use-build-artifact-from-one-pipeline-into-another/qaq-p/2025317>

We have a feature request in our issue tracker for accessing artifacts via API: <https://jira.atlassian.com/browse/BCLOUD-19330>  
It has been closed due to inactivity, as there was not a lot of demand for this feature. However, you can add your feedback and use case there if you'd be interested in this feature, as our product managers continue to monitor even closed requests.

Kind regards,  
Theodora


author: brenden_kehren
body: [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086) I've referenced the article you mentioned here:

* <https://bitbucket.org/blog/cloning-another-bitbucket-repository-in-bitbucket-pipelines>

however, I still receive errors even after adding the SSH private and public keys in the Bitbucket UI. Is there a full YML with the exact steps out there in order to clone another repository in the same workspace? Seems there may be some other configuration or steps in my YML I need to perform to make sure the SSH process works or finds the keys I added.

My process is:

1. Checkout the current repo that was triggered in the pipeline,
2. Clone the secondary repo,
3. Perform a build

```
- step:
name: Get Web project & Library Code
script:
- mkdir web.project
- cd web.project
- git init
- git fetch
- git checkout -b development
- cd ..
- mkdir web.library
- cd web.library
- git init
- git clone git@bitbucket.org:<organization>/web.library.git
- step:
name: build
....
```

The error I receive is "**error: cannot run ssh: No such file or directory**"

I should mention, this is all on the Cloud version of Bitbucket (not self-hosted).


author: James Merkle
body: It's possible this error is due to your "git config" definitions.

**Configure local git user (Windows)**   
git config --global user.name "Bob Builder"  
git config --global user.email "bob@builderbob.com"  
git config --global core.editor "C:\\Program Files (x86)\\Notepad++\\notepad++.exe"  
git config --global core.sshcommand "C:/Windows/System32/OpenSSH/ssh.exe"  
git config --global url."git@bitbucket.org:".insteadOf "<https://bitbucket.org/>"

**Configure local git user (Linux)**   
git config --global user.name "Bob Builder"  
git config --global user.email "bob@builderbob.com"  
git config --global core.editor "/usr/bin/gedit"  
git config --global core.sshcommand "/usr/bin/ssh"  
git config --global url."git@bitbucket.org:".insteadOf "<https://bitbucket.org/>"

**View your git configuration:**

git config --list

**View the source for each configuration item:**

git config --list --show-origin

When git attempts to run an SSH command, it will use the method you defined in one of the definitions above. If ssh isn't on the system path, or incorrectly defined, you will get an error.

To get this correct, you need to find where your ssh is located, and make sure it's accessible. Running "ssh --version" is a simple command to test access and determine what version of ssh you are running.


author: brenden_kehren
body: [@James Merkle](/t5/user/viewprofilepage/user-id/5290598) isn't the point of using SSH so you don't need a user info?

The image I'm using is

```
mcr.microsoft.com/dotnet/sdk:6.0
```

And I'm not running this on a local server or machine, I'm running it under Bitbucket Pipelines in the Bitbucket Cloud.

Frustrating that something so simple in Azure Dev Ops is like rocket science in Bitbucket.

I did run those config commands on my local GIT install and still no change when the Bitbucket Cloud Pipeline runs.


author: Theodora Boudale
body: Hi Brenden,

The error means that there is no SSH client in the Docker image that you use as a build container (the image mcr.microsoft.com/dotnet/sdk:6.0).

This Docker image is based on Debian 11, so you can install the OpenSSH client by adding these commands in your yml file, before the git clone command:

```
- apt update && apt upgrade -y
- apt install openssh-client -y
```

If you don't want to do this in every build, you can also build your own custom Docker image with base image mcr.microsoft.com/dotnet/sdk:6.0 where you install the OpenSSH client. Then push the image you built to DockerHub or another Docker registry, and use it as a build container in your yml:

* <https://support.atlassian.com/bitbucket-cloud/docs/use-docker-images-as-build-environments/#Creating-a-custom-build-environment>

If you have any other questions or issues, please create a new question in community via <https://community.atlassian.com/t5/forums/postpage/board-id/bitbucket-questions>.

This is a very old question from 2021; email notifications are sent on new replies to everyone watching this thread, for an issue that may not be relevant to them. The post can also become convoluted and difficult to follow if we try to troubleshoot multiple users' issues in it. If you need assistance, a new question is better.

Kind regards,  
Theodora


author: brenden_kehren
body: Thank you for the details [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086) . Your comments about adding/installing the SSH client were what was missing from YOUR original answer, the articles you posted, and any documentation I've found from Bitbucket.

While you may think this may not be relevant to the very few people who have subscribed to this post, this is the first result when you search clone multiple repositories bitbucket pipeline. So this tip could help thousands of others possibly having the same issue.

Thanks again.


author: James Merkle
body: If you require multiple repositories to be cloned, I would recommend disabling the automatic clone feature, and manually clone the repositories needed, especially if two repositories need to be "siblings", at the same directory level.

Add the following to the top of your YML file:  

```
clone:
  enabled: false # do not perform a git clone into the build folder by default
```

Within your 'script' section, clone the repos necessary (power-shell environment):  
'jack' is the "current repository", for which BITBUCKET_BRANCH environment will define the branch.  

```
script:
 - env | sort
 - echo "Clone jack and jill repositories"
 - git clone -b $BITBUCKET_BRANCH --depth=1 git@bitbucket.org:my_id/jack.git # clone the branch requested
 - git clone --depth=1 git@bitbucket.org:my_id/jill.git
```

The result will be two repositories, 'jack', and 'jill', both in the build folder.  
Using --depth=1 will limit history, allowing a faster clone process.  

comments: author: Matt Spuddy Quail
body: James' suggestion worked well for my usecase. Specifying **clone.enabled: false** and cloning the two repos into their own directories was particularly insightful.

I my case, I wanted to also be able to do a **git push** to the main repo. Following the advice in [this documentation](https://support.atlassian.com/bitbucket-cloud/docs/push-back-to-your-repository/) lead me to this procedure:

* Use HTTP instead of the SSH (git) protocol.
* Set the http proxy (which will authenticate upstream for you).
* Set the name/email git config (I set them to the same values that Pipelines uses).
* Set push.default to squelch a git warning.

```
script:
- git config --global http.proxy http://localhost:29418
- git clone --branch="${BITBUCKET_BRANCH}" --depth 1 
"${BITBUCKET_GIT_HTTP_ORIGIN}" jack
- git clone --depth 1 http://bitbucket.org/my_id/jill.git
- cd jack
- git config user.name bitbucket-pipelines
- git config user.email commits-noreply@bitbucket.org
- git config push.default simple
- date > timestamp.txt
- git add timestamp.txt
- git commit -m "[skip ci] update timestamp"
- git push
```


Tags: atl-gruland

bitbucket-cloud

bitbucket-pipelines

cloud