I spent the day chasing mysterious and unpredictable timeouts when creating diffs with ‘arc diff’, ala https://pastebin.com/gycWsTq9. The issue was quite persistent (immediately and repeatedly retrying didn’t help), but only for some requests.
Our setup is in Google Cloud, behind their load balancer which terminates HTTPS then talks HTTP to the backend which runs Phabricator.
It turned out libcurl (within arc) was defaulting to HTTP 2 and this wasn’t working properly with the LB. This patch to libphutil fixed the issue conclusively for me: https://pastebin.com/3b7XA5Ti
I’m not the only one to have run into this, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233177. The root cause seems to be curl just plain not supporting talking HTTP 2 to proxies https://curl.haxx.se/docs/todo.html#Support_HTTP_2_for_HTTP_S_proxi while defaulting to HTTP 2.
I just wanted to raise this and see if you might consider baking the above curl_setopt() call into libphutil until curl supports this properly. Thanks for everything.