After system restarts, cannot start daemons with phd

Hello,
we have been running phabricator for a few years, and periodically need to restart the daemons using the ./bin/phd start command. Our system config has not changed, However, this time issuing the start command, or any phd commands all return only the following command line output:

Content-type: text/html; charset=UTF-8

Our version info is:
phabricator
edda40aa3b931ed064c52536af5b65fd6611e841 (Apr 23 2019) (branched from 02cfcfa0796993e078c8cb632a9df39ef7227e46 on origin)

arcanist
4d22e0f89f10b8a7f47f6ee615e0ccf0f354e582 (Mar 8 2019) (branched from 9830c9316d38988b2dc283ac1a124b73bc8e6c5f on origin)

phutil
3e6a9946bcfc39591dbd9a571d1f56c8c27d4a0a (Apr 17 2019) (branched from 639e4b9cae284717b1ed717dd1e4d11c70744b86 on origin)

php
7.3.28

diff
3.3 at /bin/diff

git
1.8.3.1 at /bin/git

hg
2.6.2 at /bin/hg

pygmentize
2.2.0 at /bin/pygmentize

Any help or suggestions greatly appreciated.

Are you able to upgrade? That version info for phabricator is 2 years out of date and it’s likely whatever the issue is would have been fixed, unless it’s an environment/configuration issue.

I’m not sure the error information you posted was pasted properly. All you’ve indicated for the output of the command is

Content-type: text/html; charset=UTF-8

Which itself seems incomplete. Is that all that prints out on the command line? Is there any information in the phd logs?

Hello. Thanks for your response!
The last time we updated, some features that were pretty important to my team were removed, so I am a little hesitant since everything has been working for us very well. Until this issue. I am guessing it must be some kind of configuration issue, although nothing has changed in our phabricator config and was working fine until this restart.

The output I pasted is complete. That’s all that is the only output given after issuing the command.

the daemon.log shows this:

[14-May-2021 17:06:01 UTC] arcanist(head=stable, ref.master=8f69a5c378c3, ref.stable=4d22e0f89f10), phabricator(head=stable, ref.master=b6bf0f6a3b49, ref.stable=edda40aa3b93), phutil(head=stable, ref.master=ff84abc7d857, ref.stable=3e6a9946bcfc)
[14-May-2021 17:06:01 UTC]   #0 <#2> PHPMailer::SmtpSend(string, string) called at [<phabricator>/externals/phpmailer/class.phpmailer.php:576]
[14-May-2021 17:06:01 UTC]   #1 <#2> PHPMailer::Send() called at [<phabricator>/src/applications/metamta/adapter/PhabricatorMailSMTPAdapter.php:155]
[14-May-2021 17:06:01 UTC]   #2 <#2> PhabricatorMailSMTPAdapter::sendMessage(PhabricatorMailEmailMessage) called at [<phabricator>/src/applications/metamta/storage/PhabricatorMetaMTAMail.php:750]
[14-May-2021 17:06:01 UTC]   #3 <#2> PhabricatorMetaMTAMail::sendWithMailers(array) called at [<phabricator>/src/applications/metamta/storage/PhabricatorMetaMTAMail.php:555]
[14-May-2021 17:06:01 UTC]   #4 <#2> PhabricatorMetaMTAMail::sendNow() called at [<phabricator>/src/applications/metamta/PhabricatorMetaMTAWorker.php:22]
[14-May-2021 17:06:01 UTC]   #5 <#2> PhabricatorMetaMTAWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:124]
[14-May-2021 17:06:01 UTC]   #6 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:158]
[14-May-2021 17:06:01 UTC]   #7 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[14-May-2021 17:06:01 UTC]   #8 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[14-May-2021 17:06:01 UTC]   #9 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]
[14-May-2021 17:06:01 UTC] arcanist(head=stable, ref.master=8f69a5c378c3, ref.stable=4d22e0f89f10), phabricator(head=stable, ref.master=b6bf0f6a3b49, ref.stable=edda40aa3b93), phutil(head=stable, ref.master=ff84abc7d857, ref.stable=3e6a9946bcfc)
[14-May-2021 17:06:01 UTC]   #0 <#2> PHPMailer::SmtpSend(string, string) called at [<phabricator>/externals/phpmailer/class.phpmailer.php:576]
[14-May-2021 17:06:01 UTC]   #1 <#2> PHPMailer::Send() called at [<phabricator>/src/applications/metamta/adapter/PhabricatorMailSMTPAdapter.php:155]
[14-May-2021 17:06:01 UTC]   #2 <#2> PhabricatorMailSMTPAdapter::sendMessage(PhabricatorMailEmailMessage) called at [<phabricator>/src/applications/metamta/storage/PhabricatorMetaMTAMail.php:750]
[14-May-2021 17:06:01 UTC]   #3 <#2> PhabricatorMetaMTAMail::sendWithMailers(array) called at [<phabricator>/src/applications/metamta/storage/PhabricatorMetaMTAMail.php:555]
[14-May-2021 17:06:01 UTC]   #4 <#2> PhabricatorMetaMTAMail::sendNow() called at [<phabricator>/src/applications/metamta/PhabricatorMetaMTAWorker.php:22]
[14-May-2021 17:06:01 UTC]   #5 <#2> PhabricatorMetaMTAWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:124]
[14-May-2021 17:06:01 UTC]   #6 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:158]
[14-May-2021 17:06:01 UTC]   #7 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[14-May-2021 17:06:01 UTC]   #8 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[14-May-2021 17:06:01 UTC]   #9 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]

Is your PHP install new?
Funny installs of PHP can decide to output HTTP headers when running.

You can run php -i to see the configuration and which INI files are loaded; for me, it starts with

phpinfo()
PHP Version => 7.4.3

System => Linux vm19 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64
Build Date => Oct  6 2020 15:47:56
Server API => Command Line Interface
Virtual Directory Support => disabled
Configuration File (php.ini) Path => /etc/php/7.4/cli
Loaded Configuration File => /etc/php/7.4/cli/php.ini
Scan this dir for additional .ini files => /etc/php/7.4/cli/conf.d

note the cli everywhere - see PHP: The configuration file - Manual for details.

This wouldn’t explain why there’s no other output at all - does ./bin/phd file look normal?

Everything looks normal to me. The strangest thing is that we have been running this version for a few years with no changes to config.

What all did you check with the file? Can you confirm:

  • ./bin/phd is a symlink to ./scripts/daemon/manage_daemons.php
  • manage_daemons.php has the executable flag set
  • running file manage_daemons.php indicates:
manage_daemons.ph: a /usr/bin/env php script, ASCII text executable
  • the status of the phabricator repository indicates there are no modifications to the application folder
  • you can run /usr/bin/env php -i and get the same php information as your expected installation – note that this is how the manage_daemons.php file indicates how it should be executed, check its first line
  • are you able to run other scripts in ./bin/ or is it just this one which fails?
1 Like