SSH from repo's pipeline fails while repo with identical config succeeds: Permission denied (pubkey)
Asked by Basiel Smitz:

I have a repo where I want to do the deploy over the rsync-deploy pipe in my bitbucket-pipeline  

however it keeps failing on SSH with the message: Permission denied (publickey).  
to simplify testing I have added the following line to the start of my deploy script which it keeps failing on as well.

```
- ssh -v $SSH_USER@$SSH_SERVER_LIVE "ls -l /srv"
```

I'll tell you a bit more about how i have set things up.  

The repo has a public/private keypair that is being reused over a number of other repos and servers which all have no issues.  

the public key of this repo is known to the server I am trying to connect to. It is included in the authorized_keys file located in \~/.ssh  

I have added the IP to the known hosts and generated a fingerprint for it. This IP is what is set to the $SSH_SERVER_LIVE  

besides with the IP i also tried it with the Public DNS provided by AWS as wel as a custom DNS we use internally. They all generate the same fingerprint and they all fail when sshing from within the pipeline.   

This setup is identical to other repos which have no SSH issues, I also set up a fresh test repo with the same config that only includes the pipeline yml which only executes the command above and it works fine.  

I will now provide some logging, cleared of sensitive information.

<br />

**failing SSH step in the offending repo:**

```
+ ssh -v $SSH_USER@$SSH_SERVER_LIVE "ls -l /srv"

OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.9 30 May 2023

debug1: Reading configuration data /root/.ssh/config

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files

debug1: /etc/ssh/ssh_config line 21: Applying options for *

debug1: Connecting to {ip-address-removed} [{ip-address-removed}] port 22.

debug1: Connection established.

debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa type -1

debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2

debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.3

debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.3 pat OpenSSH* compat 0x04000000

debug1: Authenticating to {ip-address-removed}:22 as 'ubuntu'

debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug1: SSH2_MSG_KEX_ECDH_REPLY received

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:{fingerprint-known_hosts-removed}

debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory

debug1: Host '{ip-address-removed}' is known and matches the ECDSA host key.

debug1: Found key in /root/.ssh/known_hosts:11

debug1: rekey out after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: rekey in after 134217728 blocks

debug1: Will attempt key: /opt/atlassian/pipelines/agent/ssh/id_rsa explicit

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>

debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey

debug1: Next authentication method: publickey

debug1: Trying private key: /opt/atlassian/pipelines/agent/ssh/id_rsa

debug1: Authentications that can continue: publickey

debug1: No more authentication methods to try.

ubuntu@{ip-address-removed}: Permission denied (publickey).
```

**successful SSH step in the test repo:**

```
+ ssh -v $SSH_USER@$SSH_SERVER "ls -l /srv"

OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.9 30 May 2023

debug1: Reading configuration data /root/.ssh/config

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files

debug1: /etc/ssh/ssh_config line 21: Applying options for *

debug1: Connecting to {ip-address-removed} [{ip-address-removed}] port 22.

debug1: Connection established.

debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa type 0

debug1: identity file /opt/atlassian/pipelines/agent/ssh/id_rsa-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2

debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.3

debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.3 pat OpenSSH* compat 0x04000000

debug1: Authenticating to {ip-address-removed}:22 as 'ubuntu'

debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug1: SSH2_MSG_KEX_ECDH_REPLY received

debug1: Server host key: ecdsa-sha2-nistp256 SHA256: {fingerprint-known_hosts-removed}

debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory

debug1: Host '{ip-address-removed}' is known and matches the ECDSA host key.

debug1: Found key in /root/.ssh/known_hosts:7

debug1: rekey out after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: rekey in after 134217728 blocks

debug1: Will attempt key: /opt/atlassian/pipelines/agent/ssh/id_rsa RSA SHA256:{fingerprint-not-in-known_hosts-removed} explicit

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>

debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey

debug1: Next authentication method: publickey

debug1: Offering public key: /opt/atlassian/pipelines/agent/ssh/id_rsa RSA SHA256:{fingerprint-not-in-known_hosts-removed} explicit

debug1: Server accepts key: /opt/atlassian/pipelines/agent/ssh/id_rsa RSA SHA256:{fingerprint-not-in-known_hosts-removed} explicit

Authenticated to {ip-address-removed} ([{ip-address-removed}]:22) using "publickey".

debug1: channel 0: new session [client-session] (inactive timeout: 0)

debug1: Requesting no-more-sessions@openssh.com

debug1: Entering interactive session.

debug1: pledge: filesystem

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for {ip-address-removed} / (none)

debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for {ip-address-removed} / (none)

debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist

debug1: Remote: /home/ubuntu/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding

debug1: Remote: /home/ubuntu/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding

debug1: Sending environment.

debug1: Sending command: ls -l /srv

Learned new hostkey: RSA SHA256:{fingerprint-not-in-known_hosts-2-removed}

Learned new hostkey: ED25519 SHA256:{fingerprint-not-in-known_hosts-3-removed}

Adding new key for {ip-address-removed} to /root/.ssh/known_hosts: ssh-rsa SHA256:{fingerprint-not-in-known_hosts-2-removed}

Adding new key for {ip-address-removed} to /root/.ssh/known_hosts: ssh-ed25519 SHA256:{fingerprint-not-in-known_hosts-3-removed}

debug1: update_known_hosts: known hosts file /root/.ssh/known_hosts2 does not exist

debug1: pledge: fork

debug1: client_input_channel_req: channel 0 rtype exit-status reply 0

debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0

total 8

drwxrwxrwx 12 root root 4096 Aug 30 14:49 maple-server

drwxr-xr-x 2 root root 4096 Aug 30 14:28 mongo-gateway

debug1: channel 0: free: client-session, nchannels 1

Transferred: sent 5408, received 5044 bytes, in 0.8 seconds

Bytes per second: sent 6792.6, received 6335.4

debug1: Exit status 0
```

**/var/log/auth.log server for offending repo, failed**

```
sshd[30921]: AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:{fingerprint-not-in-known_hosts-removed} failed, status 22
sshd[30921]: Connection closed by authenticating user ubuntu {ip-address-removed} port 46835 [preauth]
```

**/var/log/auth.log server for test repo, success**

```
sshd[30871]: Accepted publickey for ubuntu from {ip-address-removed} port 40194 ssh2: RSA SHA256:{fingerprint-not-in-known_hosts-2-removed}
sshd[30871]: pam_unix(sshd:session): session opened for user ubuntu(uid=1000) by (uid=0)
systemd-logind[376]: New session 249 of user ubuntu.
sshd[30917]: Received disconnect from {ip-address-removed} port 40194:11: disconnected by user
sshd[30917]: Disconnected from user ubuntu {ip-address-removed} port 40194
sshd[30871]: pam_unix(sshd:session): session closed for user ubuntu
systemd-logind[376]: Session 249 logged out. Waiting for processes to exit.
systemd-logind[376]: Removed session 249.
```

fingerprints removed with a placeholder like {fingerprint-not-in-known_hosts-removed} are fingerprints that are not the same as found in settings/ssh-keys on the repo  

I honestly have no idea what is going wrong or what is left for me to try and change. Any input is appreciated. Hope I have ben clear about the issue. I will try to clarify if needed.  

Thanks in advance.


Answers:
author: Saxea _Flowie_
body: Hi [@Basiel Smitz](/t5/user/viewprofilepage/user-id/4714675) ,

Maybe try to compare the files and their permissions in the directory \`/opt/atlassian/pipelines/agent/ssh' between the two pipelines? If contains the same files then check the content of each file. It seems the pub key is missing or it is not the right pair?

comments: author: Basiel Smitz
body: Hi [@Saxea _Flowie_](/t5/user/viewprofilepage/user-id/5230209)   

It appears that the offending repo does have a different id_rsa than the newly created test repo when checking /opt/atlassian/pipelines/agent/ssh/id_rsa in both pipelines it's about half the size of the key in the test repo  

I trippel checked and the public key listed in the SSH Keys settings for both repo's are identical, therefore the private one should be too? Or am i mistaken?  

the offending pipeline still successfully connects to other servers in previous steps with the same provided key.  

Any tips on what to do next?  

<br />


author: Basiel Smitz
body: Resolved the issue by adding other in use public keys to the authorized_keys file on the server and now it works.   

It does still bother me that the key listed under settings/SSH Keys appears to be incorrect.


Tags: aws

bitbucket-cloud

bitbucket-pipelines

cloud

ssh