Newly added SSH keys not working

Hello folks,

I’m currently dealing with a very strange issue on my production phabricator instance which has been running for almost a year now without any problems.

The problem i’m facing is that all newly added SSH keys don’t seem to work. I’m using echo {} | ssh git@phabricator.myorganisation.org conduit conduit.ping for testing. With keys which are already registered in phabricator everything is working fine. But with keys that have been added a few days ago nothing happens and eventually the command times out (no termination or rejection). Interestingly, when I delete an existing key and I add it again (the exact same SSH key, just newly added in phabricator) the problem arises with that key as well. Therefore, it’s really just with newly added keys. The key itself doesn’t seem to be a problem.

I started to debug this by ensuring that the SSH keys hook is returning the list of available SSH keys. That’s working fine. After that I started to run the SSHD with with the debug flag. This is what I get:

debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2k-freebsd  26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:mopB4+7sYtb0DhMbkXpJNIL4mtZbDX331sgAGhhpnYo
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:r4a824JLU26UPQLGesVCLOyxX7dCfC0hShLHiDXPpnY
debug1: private host key #2: ssh-ed25519 SHA256:BJ5cKkOZPab1RGMoa3wRgIKeJ6EJjtp7lreNf8ayaRA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-f'
debug1: rexec_argv[3]='/etc/ssh/sshd_config.phabricator'
debug1: Bind to port 22 on ::.
debug1: Server TCP RWIN socket size: 65536
Server listening on :: port 22.
debug1: Bind to port 22 on 0.0.0.0.
debug1: Server TCP RWIN socket size: 65536
Server listening on 0.0.0.0 port 22.
debug1: fd 5 clearing O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
debug1: res_init()
Connection from 213.144.130.227 port 55732 on 84.22.106.112 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.3
debug1: match: OpenSSH_7.3 pat OpenSSH* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2 FreeBSD-20161230
debug1: permanently_set_uid: 22/22 [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user git service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "git"
debug1: PAM: setting PAM_RHOST to "adsl-130-227.dsl.init7.net"
debug1: userauth-request for user git service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:LHCNXPeq8UMmQdcUkPRwCRpoS8HGohHFP/Rd1cKSw54 [preauth]
debug1: matching key found: file /usr/local/www/phabricator.myorganisation.org/sshd_stuff/phabricator-ssh-hook.sh, line 4 RSA SHA256:LHCNXPeq8UMmQdcUkPRwCRpoS8HGohHFP/Rd1cKSw54

As you can see, the matching key is being found through the script that returns the authorized keys. However, after that line nothing happens and everything just hangs there.
I did start to have a look at the processes during the time where it hangs and I found that the following process is running:

git     1162   0.8  0.9  162132  36300  0  S    17:08   0:00.12 php /usr/local/www/phabricator.myorganization.org/phabricator/bin/ssh-auth git

After a while, the session times out and that process vanishes.

I did make sure that the hook script is owned by root and has permission 755 as noted in the phabricator documentation. But that can’t really be the problem anyway as the key is being found.

Everything else in phabricator is working very well.

I’m completely stuck here. Do you guys have any idea? Any kind of hint & wild guess would be largely appreciated.

My setup:

  • FreeBSD 11.1 64-bit
  • Phabricator updated about a week ago
  • PHP 5.6.32
  • OpenSSH 7.2.p2
1 Like

This appears to be a bug in OpenSSH/FreeBSD. See:

https://secure.phabricator.com/T11827
https://bugzilla.mindrot.org/show_bug.cgi?id=2496

From the FreeBSD task, this is apparently fixed in the OpenSSH upstream:

https://github.com/openssh/openssh-portable/commit/ddd3d34e5c7979ca6f4a3a98a7d219a4ed3d98c2

I’m not sure what the release pipeline looks like between that fix and getting the patch onto a FreeBSD system. T11827 has a possible workaround but since this is a bug in OpenSSH with a fix available in the upstream, users rarely encounter it (you’re the second user to ever report it), and the fix suggests that the workaround shouldn’t really work anyway, I don’t expect to make upstream changes.

1 Like

Hello @epriestley,

Thank you a lot for your quick reply - much appreciated!

I can confirm that what you’re stating is exactly what’s happening here. My FreeBSD server is running OpenSSH 7.2p2.
Out of curiosity I tried nileshgr’s patch shown at the very bottom of T11827 and that work. I’ll look into upgrading OpenSSH to 7.3p1 now.

Hi, we also were facing this bug this week-end when upgrading our instances. We went from ssh 6.6.1 to 7.4 and had the problem. We’re running Centos 7.

Our work around so far is to generate an authorized keys file every minute from the ssh-auth script and use AuthorizedKeysFile instead of AuthorizedKeysCommand.

I had a lot of trouble finding the issue :confused: Also because it was actually working for some user (like me), particulary if we were using an ssh agent with multiple keys. if only I saw this post before hehe. We found the problem by randomly trying stuff and finally stumbling on this bug report https://bugzilla.redhat.com/show_bug.cgi?id=1496467

I must add that there’s no bugfix in Redhat/Centos 7 as far as I know. And this is a very much used distro in the industry (and for us in a university).

Finally, the fix in T11827#202919 doesn’t make it for me, I added a loop to generate more key artificially and it still get stuck.

EDIT: We’re now testing an other solution where the ssh-auth.php script only fetch the key supplied by the user. A little bit like ssh-auth-key.php (which is not maintained btw ?). This needs sshd >6.9 though.