Drydock workingcopies not being deleted after failure to use them

Observed Behavior:
Drydock tries to clone a repository to which the bot user doesn’t have view permission. This leads to the /var/drydock/workingcopy-xxxx directory being created, not filled in, and not deleted afterwards. As this retries every few seconds without limit, it created 250000 empty directories which don’t seem to be expiring.

Expected Behavior:
Delete the /var/drydock/workingcopy-xxxx directory after failing to use it.

Phabricator Version:
5dc4e76eb9f8f883e9d3846e084a2ac06da1ef3d (Thu, May 2)
9830c9316d38988b2dc283ac1a124b73bc8e6c5f (Mar 7 2019)
639e4b9cae284717b1ed717dd1e4d11c70744b86 (Apr 12 2019)

Reproduction Steps:

  • set up Almanac Hosts, Drydock, Harbormaster.
  • create a bot user and give him permission to ssh into an Almanac host and to read-write permissions under its /var/drydock.
  • set up a Working Copy Blueprint that checks out code on git push to a certain repository in order to perform some action on it.
  • set up a Harbormaster Build Plan that uses above resources
  • add a Herald trigger to trigger above Build Plan
  • ensure the bot user does not have access to view the above repository
  • push something to above repository
  • /var/drydock on the Almanac host will start filling up with empty workingcopy-xxxx directories in incrementing IDs.
  • also, logs for the Drydock Working Copy Blueprint will show errors of type = “Activation Failed”, as such:
Resource activation failed: [CommandException] Command failed with error #128!
ssh '-o' 'LogLevel=ERROR' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l '********' -p '22' -i '********' '<almanac-host-ip>' -- 'git clone -- '\''ssh://********@phabricator.corp.local/diffusion/<repo>/<repo>.git'\'' '\''/var/drydock/workingcopy-213018/repo/<repo>/'\'''


Cloning into '/var/drydock/workingcopy-213018/repo/<repo>'...
phabricator-ssh-exec: [Access Denied: Restricted Repository] (Can View) You do not have permission to view this object. // Members of the project "Restricted Project" can take this action.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.