Fatal: Not a git repository - after server power outage

After a power outage, and realizing our UPS batteries are not sufficient, we are left with a few of our repos as not operational.

Of our 80 or so repo’s, about five of them are in-operational. doing a git pull reports this:

fatal: '/var/repo/161/' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This repo worked prior to the server outage. When browsing to the diffusion page we get this:

Unhandled Exception (“CommandException”)

Command failed with error #128!
COMMAND
git log --skip=0 -n 15 --pretty=format:’%H:%P’ ‘ea769cb87cc1addf4e3bb86a4c54bcffdc2594d3’ –

STDOUT
(empty)

STDERR
fatal: Not a git repository (or any of the parent directories): .git

Luckily we can get the code as it’s stored locally on the developers computers, but we would like to not have to init a new repo and have all of our developers have to muck around with git remote …

Any help is greatly appreciated!

Thanks,
Ryan

Phabricator only interacts with Git repositories by running git commands (and a handful of other “definitely safe” things like writing commit hook files), so nothing we do should be able to damage the repository in a way that it isn’t otherwise susceptible to (for example, we never directly modify files in the repository ourselves). So, if you can find guidance about similar cases in general, it may be helpful here: there’s no particular reason to believe that Phabricator does anything to make them more likely to be impacted by a power outage.

The expected structure of these directories (like /var/repo/161/) is that they contain a bare Git repo like this:

/var/repo/12345/ $ ls -alh
total 40
drwxr-xr-x  12 epriestley  staff   384B Sep 20 11:35 .
drwxr-xr-x  18 epriestley  staff   576B Sep  1 13:05 ..
-rw-r--r--   1 epriestley  staff   204B Sep 17 13:30 FETCH_HEAD
-rw-r--r--   1 epriestley  staff    23B Sep 20 11:35 HEAD
drwxr-xr-x   2 epriestley  staff    64B Sep  1 12:08 branches
-rw-r--r--   1 epriestley  staff   262B Sep 20 11:35 config
-rw-r--r--   1 epriestley  staff    73B Sep  1 12:08 description
drwxr-xr-x  13 epriestley  staff   416B Sep  1 12:08 hooks
drwxr-xr-x   4 epriestley  staff   128B Sep  1 12:12 info
drwxr-xr-x  92 epriestley  staff   2.9K Sep 17 13:30 objects
-rw-r--r--   1 epriestley  staff   366B Sep  1 12:09 packed-refs
drwxr-xr-x   4 epriestley  staff   128B Sep  1 12:08 refs

What’s in them now? Do they look like this, or are they totally empty?

If they’re fully gone or you run out of ways to try to restore them to a working state, you can (mostly) restore them like this:

  • Stop the daemons.
  • Put the best clone you can find (e.g., from a developer’s laptop) on disk somewhere.
  • mv 123 123.old to move aside the bad/damaged repository.
  • Run git clone --bare file:///path/to/developer/working-copy 123 to clone a new bare working copy.
  • (Once you’ve restored everything, start the daemons again.)

Depending on what the developer had this may not get 100% of everything back (e.g., some branches or tags may be missing or out of date), but should get you most of the way without needing too much init + push fiddling.

If the repositories are non-empty, you can also go create a support pact on admin.phacility.com, file a support ticket referencing this thread, upload one of them, and I can take a quick look and see if I can figure out what’s up. I don’t specifically have experience restoring crashed Git repositories, but have a sense of what they normally look like and may be able to figure out what has happened.

(You don’t need to actually pay for the pact for this, just don’t expect miracles.)

And, obviously, see also the backup documentation:

https://secure.phabricator.com/book/phabricator/article/configuring_backups/

…and repository clustering documentation:

https://secure.phabricator.com/book/phabricator/article/cluster_repositories/

…but I wouldn’t normally expect a power outage to vanish a ton of data into the aether so who knows.

Thank you for the reply!

I was able to get everything back; for some reason the HEAD file in the server was empty. I was able to add ref: refs/heads/master to the file and everything came back as expected. The repos that were affected were used by different developers in different parts of our process and have no overlap (nor would any of the developers use the other repos).