Arc-diff fails with "Segmentation fault: 11"

I’m experiencing an issue with Arcanist that appears to be specific to when I’m contributing to https://code.wildfiregames.com/ and not using a VPN.

$ arc version 
arcanist 7329bc7c32b995a7dce0319723857cbdf4f95a91 (23 May 2019)
libphutil ada64585035e8fbdf5d0dd6abfb6c13e6b160b7d (17 Jun 2019)
$ php --version
PHP 7.2.19
Zend Engine v3.2.0, with Zend OPcache v7.2.19

Arcanist works fine for other Phabricator installs (e.g. at wikimedia.org).
And for wildfiregames.com it works fine as long as I proxy through a (nearby) VPN within the country (UK).

I’ve added some ad-hoc debugging to Arcanist to see what’s going on, but nothing conclusive so far. Results below.

$ git remote -v
origin	git@github.com:0ad/0ad.git (fetch)
origin	git@github.com:0ad/0ad.git (push)

$ cat .arcconfig 
{
  "phabricator.uri" : "https://code.wildfiregames.com/",
  "repository.callsign" : "P"
}

$ git show
commit 141d09f115824a09f04bffd27bbbec98baafffdc (HEAD -> master)

    tests: Follow-up rP22378

diff --git a/source/ps/tests/test_cppformat.h b/source/ps/tests/test_cppformat.h
index df2e2f8380..7c66345f24 100644
--- a/source/ps/tests/test_cppformat.h
+++ b/source/ps/tests/test_cppformat.h
@@ -25,7 +25,8 @@ public:
    void test_basic()
    {
        // Make test behave independent of current host locale
-       char* old = setlocale(LC_ALL, "en_US.UTF-8");
+       char* old = setlocale(LC_ALL, NULL);
+       setlocale(LC_ALL, "en_US.UTF-8");

1s $ arc diff --preview
Segmentation fault: 11

$ arc diff --preview --trace
 ARGV  '/Users/krinkle/Development/arcanist/bin/../scripts/arcanist.php' 'diff' '--preview' '--trace'
 LOAD  Loaded "phutil" from "/Users/krinkle/Development/libphutil/src".
 LOAD  Loaded "arcanist" from "/Users/krinkle/Development/arcanist/src".
Config: Reading user configuration file "/Users/krinkle/.arcrc"...
Config: Did not find system configuration at "/etc/arcconfig".
Working Copy: Reading .arcconfig from "/Users/krinkle/Development/0ad-git/.arcconfig".
Working Copy: Path "/Users/krinkle/Development/0ad-git" is part of `git` working copy "/Users/krinkle/Development/0ad-git".
Working Copy: Project root is at "/Users/krinkle/Development/0ad-git".
Config: Did not find local configuration at "/Users/krinkle/Development/0ad-git/.git/arc/config".
>>> [0] (+0) <http> https://code.wildfiregames.com/api/user.whoami
Segmentation fault: 11

Now, when I enable my VPN, the same command works fine:

$ arc diff --preview --trace
 ARGV  '/Users/krinkle/Development/arcanist/bin/../scripts/arcanist.php' 'diff' '--preview' '--trace'
 LOAD  Loaded "phutil" from "/Users/krinkle/Development/libphutil/src".
 LOAD  Loaded "arcanist" from "/Users/krinkle/Development/arcanist/src".
Config: Reading user configuration file "/Users/krinkle/.arcrc"...
Config: Did not find system configuration at "/etc/arcconfig".
Working Copy: Reading .arcconfig from "/Users/krinkle/Development/0ad-git/.arcconfig".
Working Copy: Path "/Users/krinkle/Development/0ad-git" is part of `git` working copy "/Users/krinkle/Development/0ad-git".
Working Copy: Project root is at "/Users/krinkle/Development/0ad-git".
Config: Did not find local configuration at "/Users/krinkle/Development/0ad-git/.git/arc/config".

>>> [0] (+0) <http> https://code.wildfiregames.com/api/user.whoami
<<< [25] (+4,936) <http> 613,505 us

>>> [26] (+4,937) <conduit> harbormaster.sendmessage() <bytes = 228>
>>> [27] (+4,937) <http> https://code.wildfiregames.com/api/harbormaster.sendmessage
>>> [28] (+4,937) <conduit> harbormaster.sendmessage() <bytes = 228>
>>> [29] (+4,938) <http> https://code.wildfiregames.com/api/harbormaster.sendmessage
<<< [27] (+5,246) <http> 308,460 us
ConduitFuture::didReceiveResult
<<< [26] (+5,246) <conduit> 308,695 us
<<< [28] (+5,347) <conduit> 409,729 us
>>> [30] (+5,347) <exec> $ git log '141d09f115824a09f04bffd27bbbec98baafffdc' --not '4e50b39ebefc53708f2824915f12adc250ae7e73' --format='%H%x01%T%x01%P%x01%at%x01%an%x01%aE%x01%s%x01%s%n%n%b%x02' --
<<< [30] (+5,359) <exec> 11,656 us
>>> [31] (+5,359) <exec> $ git symbolic-ref --quiet HEAD
<<< [31] (+5,367) <exec> 8,037 us
>>> [32] (+5,368) <exec> $ git rev-parse --symbolic-full-name 'master'@{upstream}
<<< [32] (+5,374) <exec> 6,428 us
>>> [33] (+5,375) <conduit> differential.setdiffproperty() <bytes = 284>
>>> [34] (+5,375) <http> https://code.wildfiregames.com/api/differential.setdiffproperty
>>> [35] (+5,375) <conduit> differential.setdiffproperty() <bytes = 787>
>>> [36] (+5,375) <http> https://code.wildfiregames.com/api/differential.setdiffproperty
>>> [37] (+5,375) <conduit> differential.setdiffproperty() <bytes = 325>
>>> [38] (+5,376) <http> https://code.wildfiregames.com/api/differential.setdiffproperty
<<< [34] (+5,555) <http> 180,218 us
<<< [33] (+5,555) <conduit> 180,527 us
<<< [36] (+5,573) <http> 197,355 us
<<< [35] (+5,573) <conduit> 197,558 us
<<< [38] (+5,599) <http> 223,872 us
<<< [37] (+5,599) <conduit> 224,056 us
Created a new Differential diff:
        Diff URI: https://code.wildfiregames.com/differential/diff/8613/

Included changes:
  M       source/ps/tests/test_cppformat.h

I’ve instrumented the code with some crude print points to see where it gets to.

When it fails (VPN off), the first request it tries to make is for user.whoami, which is also where it gets stuck. The call to ConduitClient::callMethodSynchronous never returns.

Looking at HTTPSFuture::isReady:

  1. creates handle (once).
  2. creates curl multi (once).
  3. multi-info-read gets bool(false).
  4. result index not set, return early (false).

Step 3 and 4 are then repeated 522 times until Segmentation fault: 11 appears.

For the case where it works (VPN on), the cycle repeats 15,145 times after the following is finally found:

array(3) {
  ["msg"]=>
  int(1)
  ["result"]=>
  int(0)
  ["handle"]=>
  resource(143) of type (curl)
}

Then the following is printed

<<< [0] (+341) <http> 341,843 us

And after that, HTTPSFuture::isReady sees the result is set and returns it. Then we go back up through ConduitFuture::didReceiveResult and finally ConduitClient::callMethodSynchronous returns.

I’ve updated them to their latest versions, but it is still reproducible after that.

$ arc upgrade
  Upgrading libphutil...
  Upgrading arcanist...
  **Updated!** Your copy of arc is now up to date.
$ arc version       
  arcanist d92fa96366c0ed50e4257508148aa75192d4fb1f (20 Jun 2019)
  libphutil 71666e588072e467858d87e7b4027c6e817bfa80 (25 Jun 2019)

I hit something similar. Looks like there’s a bug in curl-openssl: https://github.com/Homebrew/homebrew-core/issues/40812

I solved this with:

  • brew uninstall --ignore-dependencies curl-openssl
  • brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/e47f3e2682fa3d8a6ee58ded40718a1326e08a6d/Formula/curl-openssl.rb

But your mileage may vary, especially as far as brew install'ing random paste files off the internet goes.