Integrating other applications with Phabricator via webhooks

I’m currently looking into implementing a webhook receiver in Weblate (OSS translation platform) which needs to be notified about commits in Phabricator.

If I understood correctly (could be wrong) the feed.http-hooks is deprecated and shouldn’t be used anymore? Maybe I’m missing something but I don’t see how the new webhook functionality can replace the feed.http-hooks functionality?

The new webhook functionality does not include details and will always require a call by the webhook receiver in order to determine what the event actually is (as far as I can tell). This makes integration harder because the called application now needs to make a blocking call (the application might not easily allow that in the place where the webhook is received) to Phabricator with and deal with potential problems like retries, failures, etc.

What would be the best way to go about such a thing?

1 Like

The new webhook functionality does not include details and will always require a call by the webhook receiver in order to determine what the event actually is (as far as I can tell).

In most cases, yes.

This makes integration harder because the called application now needs to make a blocking call

Yes.

(the application might not easily allow that in the place where the webhook is received)

I’d expect most HTTP endpoints should be able to make blocking calls. For instance, I believe almost all modern web applications have some kind of blocking calls to some kind of database.

deal with potential problems like retries, failures, etc.

Phabricator provides some support for retry and failure handling, and the support is unchanged from feed.http-hooks.

You will need to have Conduit client code and handle Conduit errors, but my expectation is that this is not a dramatic increase in complexity most of the time. You can call Conduit by launching a raw curl subprocess if necessary and the web UI will guide you through how to format the parameters. Obviously, this is still harder than just getting all the data handed to you, but should not usually be a huge increase in complexity.

What would be the best way to go about such a thing?

The absolute easiest way is to use Herald to send events only when something you care about occurred, then ignore what actually happened. If your hook behavior is something like “I want to do something every time someone pushes a commit to master, but I don’t actually care about the commit”, this might work. I’d discourage this, but I can’t stop anyone.

The easiest recommended way is to write a webhook receiver which can make blocking calls, and do not return HTTP/200 until the event has been completely processed. If you encounter any kind of error, return a non-200 response. Phabricator will then handle retries for you, and continue retrying for 7 days (you can control this by adjusting the GC policy for herald.webhooks).

If you do this, the only major limit to watch out for is that you must return a 200 response within 10 seconds. My expectation is that almost all webhook receivers can complete processing within 10 seconds almost always. (If an occasional request hits a network hang or whatever, that’s fine: it will just be retried. It only needs to complete within 10 seconds once in the next 7 days.)

If you need more sophisticated control over any part of this, write and endpoint which queues the event somewhere and process it asynchronously.

If you’re comfortable working with the Phabricator platform, you can conceivably write a Phabricator extension which adds an Application, route, Controller, and Worker, then have Phabricator make a Webhook request to itself and process the event in the daemon queue, if you don’t have other processing queues you’re more comfortable working with. If you’re not familiar with PHP or prefer not to work with Phabricator, I think most other languages have tools available for handling this kind of thing.

You can also modify Phabricator to make an internal transaction.search API call and send this data with the webhook body, like feed.http-hooks did. Note that doing so will essentially give any user with access to webhooks complete, policy-bypassing read access to everything in Phabricator, which is the major reason that the way this API works has changed.

Thanks for your detailed reply! I’m going to digest it for a bit and decide what way works best for me.

As some food for thought I do think that a lot of applications that offer webhook receiving functionality (like Weblate) assume that when you create an integration there is a Github style push notification they are receiving. Now I’m not arguing that this is a good thing, good design or whatever. I am just saying that most applications seem opinionated on how such a webhook should behave and look.

Taking the above in to consideration I do think that for the overall adoption of Phabricator in the larger software ecosystem it would be beneficial to have a hook which has similar behavior to the Github webhook style (as far as I can tell all software like Gitlab, Gitea, etc. does so).

The Github UI seems to confirm that the push notification is the most widely used since they have separate option for that:

I don’t know for sure, but in Phabricator use cases I’m aware of, I don’t believe push events are the most common kind of webhook event that users want to process.

Based on discussions I’ve had with other users, I’d guess the most common kind of events user are interested in from Phabricator are related to revisions, not pushes (I currently believe the most common webhook goal for Phabricator users is something like “when someone creates a revision [maybe matching certain criteria], post a notification to [some chat service]”).

I can think of several possible reasons that Phabricator users may reasonably have different webhook goals than GitHub users, but a big one is that builds are triggered through Harbormaster in Phabricator, but often through webhooks in GitHub.

Phabricator has roughly the other options that GitHub has (“Firehose” = “Send me everything”; “Disabled” = untick “Active”; and using Herald to trigger webhooks is a more powerful version of “Let me select individual events.”).

If I got a lot of feedback from different users that they specifically wanted to use webhooks to handle push events and doing this with Herald was burdensome I’d consider adding more options to the dropdown to make event selection easier, but I have essentially no evidence in this direction so far.

Another issue is that Phabricator webhooks are global, but GitHub webhooks are per-repository. In Phabricator, a simple implementation of “Send me only push events” would send you push events for every repository, and letting you select only a subset of repositories would pretty much just be a less flexible version of Herald, which is already the UI.

overall adoption of Phabricator in the larger software ecosystem

Overall adoption of Phabricator is not a goal of this project. Phabricator is not a GitHub clone, and I’d encourage you to evaluate GitHub (or another very similar package like GitLab, Gitea, etc) if you’re looking for software which works like GitHub.