Storage upgrade issue

I’ve encounter upgrade issues when executing storage upgrade. Based on “Managing Storage Adjustments” documentation I have moved each table that is marked with “Surplus” to a separate database. But I’m still left with the tables that are “missing”. Also our projects are lost after moving all tables marked with “Surplus”.

Reproduction Instructions

root@phabricator:~# sudo /var/www/phabricator/phabricator/bin/storage upgrade                                                                                                                    [685/1596]
Before running storage upgrades, you should take down the Phabricator web                                                                                                                                   
interface and stop any running Phabricator daemons (you can disable this                                                                                                                                    
warning with --force).                                                                                                                                                                                      
    Are you ready to continue? [y/N] y                                                                                                                                                                      
Storage is up to date. Use "storage status" for details.                                                                                                                                                    
Synchronizing static tables...                                                                                                                                                                              
Verifying database schemata on "localhost"...                                                                                                                                                               

Database                 Table                 Name         Issues
phabricator_differential differential_revision key_phid     Surplus Key
phabricator_differential differential_revision key_modified Surplus Key
phabricator_differential differential_revision phid         Missing Key
phabricator_project      project_column        key_trigger  Surplus Key

Found 4 adjustment(s) to apply, detailed above.

You can review adjustments in more detail from the web interface, in Config > Database Status. To better understand the adjustment workflow, see "Managing Storage Adjustments" in the documentation.

MySQL needs to copy table data to make some adjustments, so these migrations may take some time.

    Apply these schema adjustments? [y/N] y

Applying schema adjustments...
Completed applying all schema adjustments.

Target                                                   Error                                                                                                                                              
phabricator_dashboard.dashboard_dashboard_fdocument      Surplus
phabricator_dashboard.dashboard_dashboard_ffield         Surplus
phabricator_dashboard.dashboard_dashboard_fngrams        Surplus
phabricator_dashboard.dashboard_dashboard_fngrams_common Surplus
phabricator_dashboard.dashboard_panel_fdocument          Surplus
phabricator_dashboard.dashboard_panel_ffield             Surplus
phabricator_dashboard.dashboard_panel_fngrams            Surplus
phabricator_dashboard.dashboard_panel_fngrams_common     Surplus
phabricator_dashboard.dashboard_portal                   Surplus
phabricator_dashboard.dashboard_portal_fdocument         Surplus
phabricator_dashboard.dashboard_portal_ffield            Surplus
phabricator_dashboard.dashboard_portal_fngrams           Surplus
phabricator_dashboard.dashboard_portal_fngrams_common    Surplus
phabricator_dashboard.dashboard_portaltransaction        Surplus
phabricator_dashboard.dashboard_install                  Missing
phabricator_dashboard.dashboard_dashboard_ngrams         Missing
phabricator_dashboard.dashboard_dashboardpanel_ngrams    Missing
phabricator_fact.fact_chart                              Surplus
phabricator_phortune.phortune_accountemail               Surplus
phabricator_phortune.phortune_accountemailtransaction    Surplus
phabricator_phortune.phortune_merchant.viewPolicy        Missing
phabricator_phortune.phortune_paymentmethodtransaction   Surplus
phabricator_phortune.phortune_subscriptiontransaction    Surplus
phabricator_project.project_column.triggerPHID           Surplus
phabricator_project.project_trigger                      Surplus
phabricator_project.project_triggertransaction           Surplus
phabricator_project.project_triggerusage                 Surplus
phabricator_search.search_documentfield                  Missing
phabricator_pastebin                                     Missing


The schemata have errors (detailed above) which the adjustment workflow can
not fix.

Some of these errors are caused by surplus schemata (extra tables or columns
which Phabricator does not expect). These are not serious. For information on
resolving these issues, see the "Surplus Schemata" section in the "Managing
Storage Adjustments" article in the documentation.

Phabricator/Arcanist Version
Unfortunately we do a git archive from each repository and can’t see the version info anymore. (we didn’t know that versioning was based on commit sha hash)
But based on the change ticket I found out that current version (pre upgrade) based on

The new version i’m trying to upgrade to:

arcanist bac2028421a4be6e34e08764bbbda49e68b3a604 (19 Nov 2019)
libphutil 1750586fdc50a6cd98adba4aa2f5a7649bd91dbe (30 Sep 2019)
phabricator c4b4a53cad7722f031b725f8b41511e9d341d033 (13 Dec 2019)

You appear to be running bin/storage upgrade with an old version of the code on disk, against a newer version of the schema: you are actually downgrading storage. This won’t do any permanent damage (unless you also manually modify things), but it won’t work correctly.

To fix this:

  • Figure out why your unusual git archive workflow has put an old version of code on disk, and fix it. Phabricator does not have reverse migrations and can not downgrade cleanly, so it’s important that you never deploy backwards.
  • Undo all the manual schema changes you’ve made – in particular, put the “surplus” tables back where they came from.
  • Upgrade the version of Phabricator on disk to stable correctly. The last file in resources/sql/autopatches/ should be
  • Run bin/storage upgrade. If you’ve undone all your manual interventions, this will fix all the issues and put the database in the right state. (If it raises a small number of warnings, they might point at manual interventions you didn’t fully undo.)

Weird. But done a restore from the instance and redo action and works fine now. Thanks for helping me.