Drydock workingcopies not being deleted after failure to use them

Observed Behavior:
Drydock tries to clone a repository to which the bot user doesn’t have view permission. This leads to the /var/drydock/workingcopy-xxxx directory being created, not filled in, and not deleted afterwards. As this retries every few seconds without limit, it created 250000 empty directories which don’t seem to be expiring.

Expected Behavior:
Delete the /var/drydock/workingcopy-xxxx directory after failing to use it.

Phabricator Version:
phabricator
5dc4e76eb9f8f883e9d3846e084a2ac06da1ef3d (Thu, May 2)
arcanist
9830c9316d38988b2dc283ac1a124b73bc8e6c5f (Mar 7 2019)
phutil
639e4b9cae284717b1ed717dd1e4d11c70744b86 (Apr 12 2019)
php
5.5.9-1ubuntu4.24

Reproduction Steps:

  • set up Almanac Hosts, Drydock, Harbormaster.
  • create a bot user and give him permission to ssh into an Almanac host and to read-write permissions under its /var/drydock.
  • set up a Working Copy Blueprint that checks out code on git push to a certain repository in order to perform some action on it.
  • set up a Harbormaster Build Plan that uses above resources
  • add a Herald trigger to trigger above Build Plan
  • ensure the bot user does not have access to view the above repository
  • push something to above repository
  • /var/drydock on the Almanac host will start filling up with empty workingcopy-xxxx directories in incrementing IDs.
  • also, logs for the Drydock Working Copy Blueprint will show errors of type = “Activation Failed”, as such:
Resource activation failed: [CommandException] Command failed with error #128!
COMMAND
ssh '-o' 'LogLevel=ERROR' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l '********' -p '22' -i '********' '<almanac-host-ip>' -- 'git clone -- '\''ssh://********@phabricator.corp.local/diffusion/<repo>/<repo>.git'\'' '\''/var/drydock/workingcopy-213018/repo/<repo>/'\'''

STDOUT
(empty)

STDERR
Cloning into '/var/drydock/workingcopy-213018/repo/<repo>'...
phabricator-ssh-exec: [Access Denied: Restricted Repository] (Can View) You do not have permission to view this object. // Members of the project "Restricted Project" can take this action.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.