Phabricator suddenly broke

Hi

We had an old un-updated phabricator install that suddenly stopped working.
We decided to actually update it, so we ran git pull, moved to the stable branch, storage upgraded, updated php to 7.1 and still nothing.

The daemons seem to work and start although there is an issue with a updating from remote repo on two projects but this doesnt seem to stop things.

The big issue is accessing via http, each load seems to get a different error loading specific files. Such as:

[Core Exception/Exception] Source file "/opt/phabricator/phabricator/src/infrastructure/cluster/exception/PhabricatorClusterStrandedException.php" failed to load.

and

[Core Exception/Exception] Source file "/opt/phabricator/arcanist/src/error/PhutilAggregateException.php" failed to load.

the apache logs show the following:

[Tue Apr 13 22:10:24.057654 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196] [2021-04-13 22:10:24] EXCEPTION: (Exception) Source file "/opt/phabricator/phabricator/src/infrastructure/cluster/exception/PhabricatorClusterStrandedException.php" failed to load. at [<arcanist>/src/init/lib/PhutilBootloader.php:268]
[Tue Apr 13 22:10:24.057999 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196] arcanist(head=), phabricator(head=)
[Tue Apr 13 22:10:24.058008 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #0 <#2> PhutilBootloader::executeInclude(string) called at [<arcanist>/src/init/lib/PhutilBootloader.php:207]
[Tue Apr 13 22:10:24.058014 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #1 <#2> PhutilBootloader::loadLibrarySource(string, string) called at [<arcanist>/src/symbols/PhutilSymbolLoader.php:422]
[Tue Apr 13 22:10:24.058019 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #2 <#2> PhutilSymbolLoader::loadSymbol(array) called at [<arcanist>/src/symbols/PhutilSymbolLoader.php:277]
[Tue Apr 13 22:10:24.058024 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #3 <#2> PhutilSymbolLoader::selectAndLoadSymbols() called at [<arcanist>/src/init/init-library.php:23]
[Tue Apr 13 22:10:24.058028 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #4 <#2> __phutil_autoload(string)
[Tue Apr 13 22:10:24.058050 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #5 <#2> spl_autoload_call(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:177]
[Tue Apr 13 22:10:24.058055 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #6 <#2> PhabricatorLiskDAO::raiseUnreachable(string, Exception) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:134]
[Tue Apr 13 22:10:24.058060 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #7 <#2> PhabricatorLiskDAO::newClusterConnection(string, string, string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:72]
[Tue Apr 13 22:10:24.058065 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #8 <#2> PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:852]
[Tue Apr 13 22:10:24.058070 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #9 <#2> LiskDAO::establishConnection(string) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:274]
[Tue Apr 13 22:10:24.058075 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #10 <#2> PhabricatorCursorPagedPolicyAwareQuery::loadStandardPageRows(PhabricatorCalendarEvent) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:269]
[Tue Apr 13 22:10:24.058080 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #11 <#2> PhabricatorCursorPagedPolicyAwareQuery::loadStandardPage(PhabricatorCalendarEvent) called at [<phabricator>/src/applications/calendar/query/PhabricatorCalendarEventQuery.php:162]
[Tue Apr 13 22:10:24.058085 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #12 <#2> PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/people/query/PhabricatorPeopleQuery.php:407]
[Tue Apr 13 22:10:24.058099 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #13 <#2> PhabricatorPeopleQuery::rebuildAvailabilityCache(array) called at [<phabricator>/src/applications/people/query/PhabricatorPeopleQuery.php:201]
[Tue Apr 13 22:10:24.058105 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #14 <#2> PhabricatorPeopleQuery::didFilterPage(array) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:288]
[Tue Apr 13 22:10:24.058110 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #15 <#2> PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/phid/type/PhabricatorPHIDType.php:91]
[Tue Apr 13 22:10:24.058115 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #16 <#2> PhabricatorPHIDType::loadObjects(PhabricatorObjectQuery, array) called at [<phabricator>/src/applications/phid/query/PhabricatorObjectQuery.php:153]
[Tue Apr 13 22:10:24.058120 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #17 <#2> PhabricatorObjectQuery::loadObjectsByPHID(array, array) called at [<phabricator>/src/applications/phid/query/PhabricatorObjectQuery.php:73]
[Tue Apr 13 22:10:24.058125 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #18 <#2> PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/phid/query/PhabricatorHandleQuery.php:46]
[Tue Apr 13 22:10:24.058130 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #19 <#2> PhabricatorHandleQuery::loadPage() called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:251]
[Tue Apr 13 22:10:24.058135 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #20 <#2> PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/applications/maniphest/view/ManiphestTaskListView.php:164]
[Tue Apr 13 22:10:24.058140 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #21 <#2> ManiphestTaskListView::loadTaskHandles(PhabricatorUser, array) called at [<phabricator>/src/applications/maniphest/view/ManiphestTaskResultListView.php:45]
[Tue Apr 13 22:10:24.058145 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #22 <#2> phutil_tag(string, array, array) called at [<phabricator>/src/infrastructure/javelin/markup.php:70]
[Tue Apr 13 22:10:24.058150 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #23 <#2> javelin_tag(string, array, array) called at [<phabricator>/src/view/AphrontTagView.php:161]
[Tue Apr 13 22:10:24.058155 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #24 <#2> phutil_escape_html(array) called at [<phabricator>/src/infrastructure/markup/render.php:97]
[Tue Apr 13 22:10:24.058160 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #25 <#2> phutil_tag(string, array, array) called at [<phabricator>/src/applications/home/view/PHUIHomeView.php:59]
[Tue Apr 13 22:10:24.058165 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #26 <#2> PHUIHomeView::getTagContent() called at [<phabricator>/src/view/AphrontTagView.php:101]
[Tue Apr 13 22:10:24.058169 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #27 <#2> AphrontTagView::render() called at [<phabricator>/src/view/AphrontView.php:222]
[Tue Apr 13 22:10:24.058174 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #28 <#2> AphrontView::producePhutilSafeHTML() called at [<phabricator>/src/infrastructure/markup/render.php:111]
[Tue Apr 13 22:10:24.058179 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #29 <#2> phutil_escape_html(PHUIHomeView) called at [<phabricator>/src/infrastructure/markup/render.php:135]
[Tue Apr 13 22:10:24.058184 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #30 <#2> phutil_escape_html(array) called at [<phabricator>/src/infrastructure/markup/render.php:167]
[Tue Apr 13 22:10:24.058189 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #31 <#2> phutil_implode_html(string, array) called at [<phabricator>/src/view/page/PhabricatorBarePageView.php:58]
[Tue Apr 13 22:10:24.058198 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #32 <#2> PhabricatorBarePageView::willRenderPage() called at [<phabricator>/src/view/page/PhabricatorStandardPageView.php:216]
[Tue Apr 13 22:10:24.058204 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #33 <#2> PhabricatorStandardPageView::willRenderPage() called at [<phabricator>/src/view/page/AphrontPageView.php:46]
[Tue Apr 13 22:10:24.058209 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #34 <#2> AphrontPageView::render() called at [<phabricator>/src/view/page/PhabricatorStandardPageView.php:904]
[Tue Apr 13 22:10:24.058214 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #35 <#2> PhabricatorStandardPageView::produceAphrontResponse() called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:722]
[Tue Apr 13 22:10:24.058219 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #36 <#2> AphrontApplicationConfiguration::produceResponse(AphrontRequest, PhabricatorStandardPageView) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:299]
[Tue Apr 13 22:10:24.058224 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #37 <#2> AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:207]
[Tue Apr 13 22:10:24.058229 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #38 <#2> AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:35]
[Tue Apr 13 22:10:24.058234 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #39 phlog(Exception) called at [<phabricator>/src/aphront/response/AphrontUnhandledExceptionResponse.php:32]
[Tue Apr 13 22:10:24.058239 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196]   #40 AphrontUnhandledExceptionResponse::setException(Exception) called at [<phabricator>/webroot/index.php:46]
[Tue Apr 13 22:10:24.058464 2021] [php7:notice] [pid 21216] [client 80.5.62.95:58196] [Core Exception/Exception] Source file "/opt/phabricator/phabricator/src/infrastructure/cluster/exception/PhabricatorClusterStrandedException.php" failed to load.\n\n#0 /opt/phabricator/arcanist/src/init/lib/PhutilBootloader.php(207): PhutilBootloader->executeInclude('/opt/phabricato...')\n#1 /opt/phabricator/arcanist/src/symbols/PhutilSymbolLoader.php(422): PhutilBootloader->loadLibrarySource('phabricator', 'infrastructure/...')\n#2 /opt/phabricator/arcanist/src/symbols/PhutilSymbolLoader.php(277): PhutilSymbolLoader->loadSymbol(Array)\n#3 /opt/phabricator/arcanist/src/init/init-library.php(23): PhutilSymbolLoader->selectAndLoadSymbols()\n#4 [internal function]: __phutil_autoload('PhabricatorClus...')\n#5 /opt/phabricator/phabricator/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php(177): spl_autoload_call('PhabricatorClus...')\n#6 /opt/phabricator/phabricator/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php(134): PhabricatorLiskDAO->raiseUnreachable('phabricator_cal...', Object(Exception))\n#7 /opt/phabricator/phabricator/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php(72): PhabricatorLiskDAO->newClusterConnection('calendar', 'phabricator_cal...', 'r')\n#8 /opt/phabricator/phabricator/src/infrastructure/storage/lisk/LiskDAO.php(852): PhabricatorLiskDAO->establishLiveConnection('r')\n#9 /opt/phabricator/phabricator/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php(274): LiskDAO->establishConnection('r')\n#10 /opt/phabricator/phabricator/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php(269): PhabricatorCursorPagedPolicyAwareQuery->loadStandardPageRows(Object(PhabricatorCalendarEvent))\n#11 /opt/phabricator/phabricator/src/applications/calendar/query/PhabricatorCalendarEventQuery.php(162): PhabricatorCursorPagedPolicyAwareQuery->loadStandardPage(Object(PhabricatorCalendarEvent))\n#12 /opt/phabricator/phabricator/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php(251): PhabricatorCalendarEventQuery->loadPage()\n#13 /opt/phabricator/phabricator/src/applications/people/query/PhabricatorPeopleQuery.php(407): PhabricatorPolicyAwareQuery->execute()\n#14 /opt/phabricator/phabricator/src/applications/people/query/PhabricatorPeopleQuery.php(201): PhabricatorPeopleQuery->rebuildAvailabilityCache(Array)\n#15 /opt/phabricator/phabricator/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php(288): PhabricatorPeopleQuery->didFilterPage(Array)\n#16 /opt/phabricator/phabricator/src/applications/phid/type/PhabricatorPHIDType.php(91): PhabricatorPolicyAwareQuery->execute()\n#17 /opt/phabricator/phabricator/src/applications/phid/query/PhabricatorObjectQuery.php(153): PhabricatorPHIDType->loadObjects(Object(PhabricatorObjectQuery), Array)\n#18 /opt/phabricator/phabricator/src/applications/phid/query/PhabricatorObjectQuery.php(73): PhabricatorObjectQuery->loadObjectsByPHID(Array, Array)\n#19 /opt/phabricator/phabricator/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php(251): PhabricatorObjectQuery->loadPage()\n#20 /opt/phabricator/phabricator/src/applications/phid/query/PhabricatorHandleQuery.php(46): PhabricatorPolicyAwareQuery->execute()\n#21 /opt/phabricator/phabricator/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php(251): PhabricatorHandleQuery->loadPage()\n#22 /opt/phabricator/phabricator/src/applications/maniphest/view/ManiphestTaskListView.php(164): PhabricatorPolicyAwareQuery->execute()\n#23 /opt/phabricator/phabricator/src/applications/maniphest/view/ManiphestTaskResultListView.php(45): ManiphestTaskListView::loadTaskHandles(Object(PhabricatorUser), Array)\n#24 /opt/phabricator/phabricator/src/view/AphrontView.php(222): ManiphestTaskResultListView->render()\n#25 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(111): AphrontView->producePhutilSafeHTML()\n#26 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(135): phutil_escape_html(Object(ManiphestTaskResultListView))\n#27 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(135): phutil_escape_html(Array)\n#28 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(135): phutil_escape_html(Array)\n#29 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(97): phutil_escape_html(Array)\n#30 /opt/phabricator/phabricator/src/infrastructure/javelin/markup.php(70): phutil_tag('div', Array, Array)\n#31 /opt/phabricator/phabricator/src/view/AphrontTagView.php(161): javelin_tag('div', Array, Array)\n#32 /opt/phabricator/phabricator/src/view/AphrontView.php(222): AphrontTagView->render()\n#33 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(111): AphrontView->producePhutilSafeHTML()\n#34 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(135): phutil_escape_html(Object(PHUIObjectBoxView))\n#35 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(97): phutil_escape_html(Array)\n#36 /opt/phabricator/phabricator/src/applications/home/view/PHUIHomeView.php(59): phutil_tag('div', Array, Array)\n#37 /opt/phabricator/phabricator/src/view/AphrontTagView.php(101): PHUIHomeView->getTagContent()\n#38 /opt/phabricator/phabricator/src/view/AphrontView.php(222): AphrontTagView->render()\n#39 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(111): AphrontView->producePhutilSafeHTML()\n#40 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(135): phutil_escape_html(Object(PHUIHomeView))\n#41 /opt/phabricator/phabricator/src/infrastructure/markup/render.php(167): phutil_escape_html(Array)\n#42 /opt/phabricator/phabricator/src/view/page/PhabricatorBarePageView.php(58): phutil_implode_html('', Array)\n#43 /opt/phabricator/phabricator/src/view/page/PhabricatorStandardPageView.php(216): PhabricatorBarePageView->willRenderPage()\n#44 /opt/phabricator/phabricator/src/view/page/AphrontPageView.php(46): PhabricatorStandardPageView->willRenderPage()\n#45 /opt/phabricator/phabricator/src/view/page/PhabricatorStandardPageView.php(904): AphrontPageView->render()\n#46 /opt/phabricator/phabricator/src/aphront/configuration/AphrontApplicationConfiguration.php(722): PhabricatorStandardPageView->produceAphrontResponse()\n#47 /opt/phabricator/phabricator/src/aphront/configuration/AphrontApplicationConfiguration.php(299): AphrontApplicationConfiguration->produceResponse(Object(AphrontRequest), Object(PhabricatorStandardPageView))\n#48 /opt/phabricator/phabricator/src/aphront/configuration/AphrontApplicationConfiguration.php(207): AphrontApplicationConfiguration->processRequest(Object(AphrontRequest), Object(PhutilDeferredLog), Object(AphrontPHPHTTPSink), Object(MultimeterControl))\n#49 /opt/phabricator/phabricator/webroot/index.php(35): AphrontApplicationConfiguration::runHTTPRequest(Object(AphrontPHPHTTPSink))\n#50 {main}
[Tue Apr 13 22:10:25.616743 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198] [2021-04-13 22:10:25] EXCEPTION: (Exception) Source file "/opt/phabricator/arcanist/src/error/PhutilAggregateException.php" failed to load. at [<arcanist>/src/init/lib/PhutilBootloader.php:268]
[Tue Apr 13 22:10:25.617059 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198] arcanist(head=), phabricator(head=)
[Tue Apr 13 22:10:25.617074 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #0 <#2> PhutilBootloader::executeInclude(string) called at [<arcanist>/src/init/lib/PhutilBootloader.php:207]
[Tue Apr 13 22:10:25.617085 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #1 <#2> PhutilBootloader::loadLibrarySource(string, string) called at [<arcanist>/src/symbols/PhutilSymbolLoader.php:422]
[Tue Apr 13 22:10:25.617096 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #2 <#2> PhutilSymbolLoader::loadSymbol(array) called at [<arcanist>/src/symbols/PhutilSymbolLoader.php:277]
[Tue Apr 13 22:10:25.617106 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #3 <#2> PhutilSymbolLoader::selectAndLoadSymbols() called at [<arcanist>/src/init/init-library.php:23]
[Tue Apr 13 22:10:25.617116 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #4 <#2> __phutil_autoload(string)
[Tue Apr 13 22:10:25.617125 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #5 <#2> spl_autoload_call(string) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:316]
[Tue Apr 13 22:10:25.617135 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #6 <#2> AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:207]
[Tue Apr 13 22:10:25.617146 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #7 <#2> AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:35]
[Tue Apr 13 22:10:25.617156 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #8 phlog(Exception) called at [<phabricator>/src/aphront/response/AphrontUnhandledExceptionResponse.php:32]
[Tue Apr 13 22:10:25.617166 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198]   #9 AphrontUnhandledExceptionResponse::setException(Exception) called at [<phabricator>/webroot/index.php:46]
[Tue Apr 13 22:10:25.617513 2021] [php7:notice] [pid 20406] [client 80.5.62.95:58198] [Core Exception/Exception] Source file "/opt/phabricator/arcanist/src/error/PhutilAggregateException.php" failed to load.\n\n#0 /opt/phabricator/arcanist/src/init/lib/PhutilBootloader.php(207): PhutilBootloader->executeInclude('/opt/phabricato...')\n#1 /opt/phabricator/arcanist/src/symbols/PhutilSymbolLoader.php(422): PhutilBootloader->loadLibrarySource('arcanist', 'error/PhutilAgg...')\n#2 /opt/phabricator/arcanist/src/symbols/PhutilSymbolLoader.php(277): PhutilSymbolLoader->loadSymbol(Array)\n#3 /opt/phabricator/arcanist/src/init/init-library.php(23): PhutilSymbolLoader->selectAndLoadSymbols()\n#4 [internal function]: __phutil_autoload('PhutilAggregate...')\n#5 /opt/phabricator/phabricator/src/aphront/configuration/AphrontApplicationConfiguration.php(316): spl_autoload_call('PhutilAggregate...')\n#6 /opt/phabricator/phabricator/src/aphront/configuration/AphrontApplicationConfiguration.php(207): AphrontApplicationConfiguration->processRequest(Object(AphrontRequest), Object(PhutilDeferredLog), Object(AphrontPHPHTTPSink), Object(MultimeterControl))\n#7 /opt/phabricator/phabricator/webroot/index.php(35): AphrontApplicationConfiguration::runHTTPRequest(Object(AphrontPHPHTTPSink))\n#8 {main}

I have also made a fresh git clone of arcanist and phabricator into a fresh directory and attached it to the same database with no avail.

I’ve tried lifting the ulimits (incase it was too many open files), I have apcu and opcache as apc from pecl doesnt seem to build within php7.1.

We’re running CentOS 7.9, php 7.1 (remi repo), apache 2.4. MariaDB is on a different host and the files are stored on an NFS server.

Does anyone have any ideas where to look?

I’m assuming you’ve restarted Apache and everything else after updating.

This sounds like either a PHP problem or a filesystem problem. is /opt/phabricator a regular filesystem? Is the php version recent?

Either way, I would probably to spin up a new machine, install Phabricator from scratch, and point it to the same database.

The filesystem is standard ext4 sat on an SSD, all report back fine.
PHP is latest 7.1 from Remi and seems to work fine.

I believe it’s something down to database performance, once I pushed the connection limit up on the MySQL server (MariaDB 5.5) then things started to work with the cannot load errors being intermittent along with messages such as “Cannot contact master”.

I can see a load of connections out to (~300+) MySQL being in TIME_WAIT. The database is a fresh install on a new system with the existing data imported (Again, CentOS7 but XFS on SSD). The import took ~4hours for 4gb, which feels far too slow however mysql performs fine when I use it; does phabricator use complex queries or are we too large for phabricator?

After tweaking the local tcp stack, setting tcp_fin_timeout to 2 has greatly increased the reliability.

If feels like there’s too many database connections however, the database can handle much more load than phabricator is giving it. :-/

import took ~4hours for 4gb,

Which import flow exactly did you use? That does sound slow to me, but that’s not something I do often.

Another user did report performance issues with MariaDB recently (Aphront Exception since recent update) which we were unable to reproduce; it could be related.


Phabricator does make a lot of database connections, but there’s also a lot of re-use; I don’t remember what the current state is, but 300 sounds very high for a few page views. Of course, it they break and don’t close, they’ll just keep piling up.
See maybe ◉ Cluster: Databases for possible configurations.


I don’t really see how database connectivity/load can cause the “failed to load” issue, but this is PHP so everything is possible.

Which import flow exactly did you use?

Just mysqldump and mysql < file.sql

I don’t really see how database connectivity/load can cause the “failed to load” issue, but this is PHP so everything is possible.

We were seeing “Too many files open” too but we could verify that was not the case.