We use phabs/gits mirror feature to release our products. In most cases a failed mirror is spotted fairly fast. But sometimes a mirror is failing and we just don’t “get” it. Is there some way to get notified of failing daemon tasks like mirroring. Or should we use an own bash script to scan the logs for this kind of failure?
There’s currently no way to get notified of a mirror failing (or an observation failing). See https://secure.phabricator.com/T12896 for some discussion (and https://secure.phabricator.com/T6131 for mirror status, more narrowly).
A product issue is that there isn’t any “obvious” set of users to notify when a repository operation fails. For example, notifying “all administrators” might be completely useless on installs that have thousands of user-created repositories.
There isn’t a great way to test this today even if you want to generate your own notifications. You could look at the logs, as you suggest. You could also try comparing
git ls-remote <master-uri> to
git ls-remote <mirror-uri>, but note that a user may push an arbitrarily large change which takes an arbitrarily long time to mirror.
An upstream version of this change isn’t trivial (e.g., it likely needs some new support for notification configuration, and possibly some way to configure default notifications, and probably some fiddling for “notify after X consecutive failures” vs “notify after mirroring fails for Y minutes”) and there’s currently no customer interest in building out this featureset.
(This is a reasonable thing to want and upstream-aligned, just a priority issue since the actual implementation isn’t entirely straightforward and there’s no current customer interest.)