Observing phabricator Git repo fails

When configure my installation of Phabricator to observe the phabricator git repo via https I get the following error when it attempts to update:

root@55b4eda869ea:/var/www/phabricator# bin/repository update R21
[2020-08-21 20:54:29] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git ls-remote '********' 'refs/*'

STDOUT
(empty)

STDERR
fatal: not a git repository: /var/www/phabricator/../../.git/modules/phabricator/phabricator
 at [<arcanist>/src/future/exec/ExecFuture.php:421]
arcanist(), phabricator()
  #0 ExecFuture::raiseResultError(array) called at [<arcanist>/src/future/exec/ExecFuture.php:325]
  #1 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:492]
  #2 PhabricatorRepository::execxRemoteCommand(string, PhutilOpaqueEnvelope, array) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:573]
  #3 PhabricatorRepositoryPullEngine::loadGitRemoteRefs(PhabricatorRepository, PhutilOpaqueEnvelope, boolean) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:380]
  #4 PhabricatorRepositoryPullEngine::executeGitUpdate() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:132]
  #5 PhabricatorRepositoryPullEngine::pullRepositoryWithLock() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:40]
  #6 PhabricatorRepositoryPullEngine::pullRepository() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:59]
  #7 PhabricatorRepositoryManagementUpdateWorkflow::execute(PhutilArgumentParser) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:492]
  #8 PhutilArgumentParser::parseWorkflowsFull(array) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:377]
  #9 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]
root@55b4eda869ea:/var/www/phabricator# 

Component versions:

phabricator: 72f149bf399ab1ba3fb3f55c563a7e8506381a8d
arcanist: ceb082ef6b2919d76a90d4a53ca84f5b1e0c2c06
php: 7.4.8
diff: 3.7
git: 2.20.1
hg: 4.8.2
pygmentize: 2.3.1
svn: 1.10.4

  • What are the config values for repository.default-local-path?
  • In the repository page, go to manage ==> Storage. What’s the value of Storage Path?

repository.default-local-path = /var/repo/
Storage Path = /var/repo/21/

I’m puzzled about git trying to work at:
/var/www/phabricator/../../.git/modules/phabricator/phabricator
when it should be running in /var/repo/21/

What’s in /var/www/phabricator/../../.git/modules/phabricator/phabricator?
Does R21 has any submodules? What happens when you manually run git ls-remote in /var/repo/21 ?

If I run git ls-remote in /var/repo/21 I get the following output:

root@55b4eda869ea:/var/repo/21# git ls-remote
From https://github.com/phacility/phabricator.git
72f149bf399ab1ba3fb3f55c563a7e8506381a8d        HEAD
1d9d2d066e222510e54e4d3631eb056ee16e67be        refs/heads/legacy-2019
72f149bf399ab1ba3fb3f55c563a7e8506381a8d        refs/heads/master
82f98747a4e33596f8c027cb574b8ee3cb4c7d6b        refs/heads/stable
cf03b3928b7112ed86b7ff86ddb9486fb1adb5e9        refs/pull/10/head
44a8c78d408a1da9de201f4ce07de00cefd8ba8d        refs/pull/10/merge
e77f7761983198599c4f352cdd7d11219ad2d7f4        refs/pull/103/head
f24aed7af1dc22a57db9eb5f0a2c0dd2f19296bb        refs/pull/103/merge
...
bb2c2b87c1e3f5e3b2cb0ff0462937cd461ffce2        refs/pull/96/merge
d7fcbd7d39c2d3287d1ea42c129cb5ccfd99728e        refs/pull/99/head
6bd215fc5bdc5cf073b519cb3d824bcaec2ac38c        refs/pull/99/merge
root@55b4eda869ea:/var/repo/21#

As you can see this repo is cloned from the official Phabricator repo, which may have submodules.

It may be important to know that when I initially set up this repo it failed with the same error, so I updated it manually from the command line as follows:

cd /var/repo
rm -rf 21
git clone --bare -- https://github.com/phacility/phabricator.git 21
chown daemon.daemon 21
/var/www/phabricator/bin/repository update R21

Which appeared to work, but left the initialisation error in the web interface. When I attempt to update via the web interface I get the git ls-remote error.

I can say with some confidence that https://github.com/phacility/phabricator.git doesn’t have any sub-modules and hadn’t had any in recent history.

It’s possible that in your manual initialization you got something wrong - e.g., I think it should be chown -R.
The best thing you can do to this repository right now is

  • chown daemon.daemon /var/repo itself, and
  • rm -rf /var/repo/21/, so the daemon will try to re-clone it.

Then the error will be shown in the repo’s management page, and we’ll see what the real problem is.

I have removed the repo and forced and update. The daemon initialised the bare repo, but there is nothing in it, and the repo manage page shows the error:

Pull of 'phacility-phabricator' failed: Command failed with error #128! 
COMMAND
git ls-remote '********' 'refs/*'

STDOUT
(empty)

STDERR
fatal: not a git repository: /var/www/phabricator/../../.git/modules/phabricator/phabricator

And if I view the repo I get the following error:

Command failed with error #128!
COMMAND
git log --skip=0 -n 15 --pretty=format:'%H:%P' 72f149bf399ab1ba3fb3f55c563a7e8506381a8d -- 

STDOUT
(empty)

STDERR
fatal: bad object 72f149bf399ab1ba3fb3f55c563a7e8506381a8d

The second error is consistent with a repo that had some data, was deleted, and hadn’t successfully re-cloned yet, so that’s expected.

The first error is consistent with a git repository that’s was cloned as a sub-module and later had its git directory deleted, and is also located at /var/www/phabricator/, which is not expected. Unless…

It’s possible that the ls-remote command is executing without setting cwd first, if the argument hidden is the full url of the remote (and not a name).
If that’s the case, and the daemon just happen to execute with a cwd inside /var/www/phabricator/, and /var/www/phabricator/.git is a file that refers to a location that is corrupt (e.g. doesn’t exist), then git will try to read the “local” git dir and fail.

If that’s the case, then simply deleting the bad .git file (or re-cloning phabricator in /var/www/phabricator) would solve your issue.

The problem turned out to be a .git submodule file in the root of the phabricator code in my docker image. The file is there as a side effect of the way I create my docker image. I check out the phabricator and arcanist code as a submodule of my container project and copy them into the docker image. If I remove or rename the .git file then the observed repo works as expected.

Thanks for your help.