How to have a shallow git mirror?

Git
#1

We are developing features for a project with an enormous git repo. Internally, we would like to host a shallow mirror on phabricator and maintain our branches there. Is is currently supported?

Things I managed to do so far:

  1. Create a shallow clone on the phabricator instance
  2. Push branches to this clone
  3. Browse this repo

Things that don’t work:

  1. Creating an empty repo and pushing to it from a shallow clone (this is how I initially tried to have a shallow clone on the phabricator instance)
  2. Fetch from phabricator instance (the git uplaod-pack proxy fails)
#2

I think these are limitations of git itself, not of Phabricator; You’ll have to understand a little more about how git uses shallow clones/fetch/push to make this work.

I would expect fetch to work, if the client is up-to-date though.

Normally, though, the costs of having the full repo in phabricator are just some disk space and a one-time import time; Why not just have the whole thing there?

#3

It’s not the limitation of git itself afaik b/c when I set the remote to ssh directly to the Phabricator instance (e.g. git remote add raw_phab ssh://yannayl@phab:/var/repo/4) and not via the Phabricator proxy (ssh://phab-vcs-user@phab:2222/diffusion/TST/test.git) - everything works as expected.
Also, as said, fetch doesn’t work, the git upload-pack proxy yields the following error:

Exception: Unexpected "git upload-pack" protocol frame: expected "<hash> <name>\n", got "shallow ae4a9579cedd2609cc37028cf1aaaaaaaaaaa
".
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Git’s limitation is that both branches must have a common ancestor (there were further restrictions in older git versions but they were removed, see https://stackoverflow.com/questions/6900103/why-cant-i-push-from-a-shallow-clone/6900428#6900428 for example).

The amount of storage needed for this repo is enormous and the time to clone it will be days. Also, I am not sure about the resources Phabricator will need to process it (does it process the history of the repo?).

#4

hmm, the protocol frame error could be the proxy - https://secure.phabricator.com/D20381 introduces a protocol proxy, and possibly doesn’t handle shallow - I’ll try to reproduce and upstream a report.

re resources - yes, Phabricator processes all reachable commits, with some exceptions (huge commits might be skipped, and I think “binary files” are not processed). IIRC, most normal commits are also copied into the database as part of the processing, but I might be wrong about that.

#5

Thanks!
Just note that currently there is no straight-forward way to make Phabricator host a shallow clone. I issued the git commands manually.
It might be possible if observing a shallow clone, but I am not sure.

Hosting a shallow copy can be a really nice feature to have :slight_smile:

#6

https://secure.phabricator.com/T13309 discusses the protocol proxying issue.

https://secure.phabricator.com/D20577 should fix the protocol parser so git clone can work if the working copy on disk is shallow.

Shallow working copies are not supported so we’re unlikely to fix anything else here, but the protocol proxy should be able to proxy the entire protocol.

#7

Thanks. I will apply the patch and test it on Sunday.

#8

I applied the patch - fetch and push work.
Thank you