Storage Upgrade issues (new install)

I am trying to do an Phabricator install on a new system.

I am using:
Ubuntu Server 20.04 x86_64
MySQL Ver 8.0.19-0ununtu5
PHP 7.4.3
Apache/2.4.41

I used the Phabricator setup script to install all the components.
Apache seems to work. It delivers a Phabricator page asking to run server update.

I had an issue with MySQL config. Apparently the newer versions don’t want you to use the root user. I created a new user and authorized it in MySQL. Now Phabricator appears to be able to use MySQL.

When I run ./bin/storage upgrade the first time, I get:

barry@qms:/opt/phabricator$ ./bin/storage upgrade
Before running storage upgrades, you should take down the Phabricator web
interface and stop any running Phabricator daemons (you can disable this
warning with --force).

    Are you ready to continue? [y/N] y

Loading quickstart template onto "localhost"...

It never returns. I have waited over 10 minutes.
I ctrlC to exit.

If I try to run ./bin/storage upggrade again, I get:

barry@qms:/opt/phabricator$ ./bin/storage upgrade
Before running storage upgrades, you should take down the Phabricator web
interface and stop any running Phabricator daemons (you can disable this
warning with --force).

    Are you ready to continue? [y/N] y

Loading quickstart template onto "localhost"...
[2020-04-24 15:47:50] EXCEPTION: (AphrontQueryException) #1050: Table 'almanac_binding' already exists at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:386]
arcanist(head=master, ref.master=68f050bd14e0), phabricator(head=master, ref.master=d05d8f655896)
  #0 AphrontBaseMySQLDatabaseConnection::throwQueryCodeException called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:320]
  #1 AphrontBaseMySQLDatabaseConnection::throwQueryException called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:216]
  #2 AphrontBaseMySQLDatabaseConnection::executeQuery called at [<phabricator>/src/infrastructure/storage/xsprintf/queryfx.php:8]
  #3 queryfx called at [<phabricator>/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:274]
  #4 PhabricatorStorageManagementAPI::applyPatchSQL called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:985]
  #5 PhabricatorStorageManagementWorkflow::doUpgradeSchemata called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:903]
  #6 PhabricatorStorageManagementWorkflow::upgradeSchemata called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementUpgradeWorkflow.php:78]
  #7 PhabricatorStorageManagementUpgradeWorkflow::didExecute called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:107]
  #8 PhabricatorStorageManagementWorkflow::execute called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:485]
  #9 PhutilArgumentParser::parseWorkflowsFull called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:376]
  #10 PhutilArgumentParser::parseWorkflows called at [<phabricator>/scripts/sql/manage_storage.php:249]

Then if I run ./storage destroy I get :

barry@qms:/opt/phabricator$ ./bin/storage destroy -f
 DESTROY  Dropping database "phabricator_almanac" on host "localhost"...
 DESTROY  Dropping database "phabricator_application" on host "localhost"...
 DESTROY  Dropping database "phabricator_audit" on host "localhost"...
 DESTROY  Dropping database "phabricator_auth" on host "localhost"...
 DESTROY  Dropping database "phabricator_badges" on host "localhost"...
 DESTROY  Dropping database "phabricator_cache" on host "localhost"...
 DESTROY  Dropping database "phabricator_calendar" on host "localhost"...
 DESTROY  Dropping database "phabricator_chatlog" on host "localhost"...
 DESTROY  Dropping database "phabricator_conduit" on host "localhost"...
 DESTROY  Dropping database "phabricator_config" on host "localhost"...
 DESTROY  Dropping database "phabricator_conpherence" on host "localhost"...
 DESTROY  Dropping database "phabricator_countdown" on host "localhost"...
 DESTROY  Dropping database "phabricator_daemon" on host "localhost"...
 DESTROY  Dropping database "phabricator_dashboard" on host "localhost"...
 DESTROY  Dropping database "phabricator_differential" on host "localhost"...
 DESTROY  Dropping database "phabricator_directory" on host "localhost"...
 DESTROY  Dropping database "phabricator_diviner" on host "localhost"...
 DESTROY  Dropping database "phabricator_doorkeeper" on host "localhost"...
 DESTROY  Dropping database "phabricator_draft" on host "localhost"...
 DESTROY  Dropping database "phabricator_drydock" on host "localhost"...
 DESTROY  Dropping database "phabricator_fact" on host "localhost"...
 DESTROY  Dropping database "phabricator_feed" on host "localhost"...
 DESTROY  Dropping database "phabricator_file" on host "localhost"...
 DESTROY  Dropping database "phabricator_flag" on host "localhost"...
 DESTROY  Dropping database "phabricator_fund" on host "localhost"...
 DESTROY  Dropping database "phabricator_harbormaster" on host "localhost"...
 DESTROY  Dropping database "phabricator_herald" on host "localhost"...
 DESTROY  Dropping database "phabricator_legalpad" on host "localhost"...
 DESTROY  Dropping database "phabricator_maniphest" on host "localhost"...
 DESTROY  Dropping database "phabricator_meta_data" on host "localhost"...
 DESTROY  Dropping database "phabricator_meta_data" on host "localhost"...
 DESTROY  Dropping database "phabricator_metamta" on host "localhost"...
 DESTROY  Dropping database "phabricator_multimeter" on host "localhost"...
 DESTROY  Dropping database "phabricator_nuance" on host "localhost"...
 DESTROY  Dropping database "phabricator_oauth_server" on host "localhost"...
 DESTROY  Dropping database "phabricator_owners" on host "localhost"...
 DESTROY  Dropping database "phabricator_packages" on host "localhost"...
 DESTROY  Dropping database "phabricator_passphrase" on host "localhost"...
 DESTROY  Dropping database "phabricator_paste" on host "localhost"...
 DESTROY  Dropping database "phabricator_pastebin" on host "localhost"...
 DESTROY  Dropping database "phabricator_phame" on host "localhost"...
 DESTROY  Dropping database "phabricator_phid" on host "localhost"...
 DESTROY  Dropping database "phabricator_phlux" on host "localhost"...
 DESTROY  Dropping database "phabricator_pholio" on host "localhost"...
 DESTROY  Dropping database "phabricator_phortune" on host "localhost"...
 DESTROY  Dropping database "phabricator_phragment" on host "localhost"...
 DESTROY  Dropping database "phabricator_phrequent" on host "localhost"...
 DESTROY  Dropping database "phabricator_phriction" on host "localhost"...
 DESTROY  Dropping database "phabricator_phurl" on host "localhost"...
 DESTROY  Dropping database "phabricator_policy" on host "localhost"...
 DESTROY  Dropping database "phabricator_ponder" on host "localhost"...
 DESTROY  Dropping database "phabricator_project" on host "localhost"...
 DESTROY  Dropping database "phabricator_releeph" on host "localhost"...
 DESTROY  Dropping database "phabricator_repository" on host "localhost"...
 DESTROY  Dropping database "phabricator_search" on host "localhost"...
 DESTROY  Dropping database "phabricator_slowvote" on host "localhost"...
 DESTROY  Dropping database "phabricator_spaces" on host "localhost"...
 DESTROY  Dropping database "phabricator_system" on host "localhost"...
 DESTROY  Dropping database "phabricator_timeline" on host "localhost"...
 DESTROY  Dropping database "phabricator_token" on host "localhost"...
 DESTROY  Dropping database "phabricator_user" on host "localhost"...
 DESTROY  Dropping database "phabricator_worker" on host "localhost"...
 DESTROY  Dropping database "phabricator_xhpast" on host "localhost"...
 DESTROY  Dropping database "phabricator_xhpastview" on host "localhost"...
 DESTROY  Dropping database "phabricator_xhprof" on host "localhost"...
 DONE  Storage on "localhost" was destroyed.

After the destroy I can try upgrade again and it hangs just like the first time. I have been through this 3 step process several times. The results are always the same.

I would sure appreciate any hints.

  • Run it with --trace.
  • While it’s running, connect to MySQL and use:
mysql> SHOW FULL PROCESSLIST;

…to see what it’s doing. If you wait a few seconds and run this command again and the output changes (it shows the connection doing something new), it is not hung. If it eventually hangs on some particular statement, that may point you at the issue.

Thank you for the --trace advice. Running with trace I could see that modifications were in process and after 25 minutes, the update was complete.

I just was not waiting long enough. It might be good to provide a warning the process is lengthy, or offer some sort of status update.

I really appreciate the help.

The process takes 8.5 seconds on my system, much of which shows progress information:

$ time ./bin/storage upgrade --namespace test123456 --force
Loading quickstart template onto "127.0.0.1:3306"...
Applying patch "phabricator:db.paste" to host "127.0.0.1:3306"...
Applying patch "phabricator:20190523.myisam.01.documentfield.sql" to host "127.0.0.1:3306"...
Applying patch "phabricator:20190718.paste.01.edge.sql" to host "127.0.0.1:3306"...
Applying patch "phabricator:20190718.paste.02.edgedata.sql" to host "127.0.0.1:3306"...
...                          
Completed applying all schema adjustments.
 ANALYZE  Analyzing tables...                                                 
Done.                                                                         
 ANALYZED  Analyzed 578 table(s).
                                                                              
real	0m8.527s
user	0m0.496s
sys	0m0.176s

Interesting. Thanks for the data.

Here is the last line from my --trace
<<< [2333] (+1,383,847) 786,344,011 us

I assume that means 1,383 seconds. I didn’t watch it the entire time, but that seems about right.

I am running Phabricator on a VM installed on a server. I never saw the server load go very high. It must be some issue with disk I/O. My disk setup is also non-typical as it is an UnRaid system with some data redundancy. Possibly MySQL performance it horrible in this configuration.

Serving the Phabricator web pages seems very snappy. We are a small team and I expect very light loads, so it might be just fine.

1 Like