We are currently noticing an increase of load on our server that appears to be related to a bunch of search engines crawling our differential pages, among other things.
At a glance the load seems to mostly come from Phabricator invoking many instances of git in the cli to get a diff and do the pygments highlighting pass. We were wondering if there is anyway to possibly disable this by default on a page load, or perhaps by passing a GET url param that we could redirect bots to based on their referrer.
In addition, I noticed that the actual diffs appear to be no-store cache control. Is there any advice or direction on the topic of tossing a caching reverse proxy in front of the pages, possibly even forcing caching of at least the git diffs?
Any help would be appreciated. Thanks!