Storage upgrade error

edshrock · April 15, 2020, 1:57pm

I probably broke my Phabricator installation during upgrade (last upgrade April 2014)… Any help would be appreciated. I did not make a backup of my database…

While running “phabricator/bin/storage upgrade”, I got:

Populating Phriction policies.
Migrating document 1 to project policy RT339196: ePortfolio - document edit...
[2020-04-15 13:02:02] EXCEPTION: (AphrontSchemaQueryException) #1054: Unknown column 'contentPHID' in 'field list' at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:356]
arcanist(head=master, ref.master=377ed2ed8db7), phabricator(head=master, ref.master=0ea6d131e072)
  #0 AphrontBaseMySQLDatabaseConnection::throwCommonException(integer, string) called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:379]
  #1 AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:320]
  #2 AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:216]
  #3 AphrontBaseMySQLDatabaseConnection::executeQuery(PhutilQueryString) called at [<phabricator>/src/infrastructure/storage/xsprintf/queryfx.php:8]
  #4 queryfx(AphrontMySQLiDatabaseConnection, string, PhrictionDocument, array, string, string)
  #5 call_user_func_array(string, array) called at [<phabricator>/src/infrastructure/storage/connection/AphrontDatabaseConnection.php:58]
  #6 AphrontDatabaseConnection::query(string, PhrictionDocument, array, string, string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:989]
  #7 LiskDAO::update() called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:910]
  #8 LiskDAO::save() called at [<phabricator>/resources/sql/autopatches/20141107.phriction.policy.2.php:49]
  #9 require_once(string) called at [<phabricator>/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:291]
  #10 PhabricatorStorageManagementAPI::applyPatchPHP(string) called at [<phabricator>/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:245]
  #11 PhabricatorStorageManagementAPI::applyPatch(PhabricatorStoragePatch) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:1157]
  #12 PhabricatorStorageManagementWorkflow::doUpgradeSchemata(array, NULL, boolean, boolean) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:903]
  #13 PhabricatorStorageManagementWorkflow::upgradeSchemata(array, NULL, boolean, boolean) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementUpgradeWorkflow.php:78]
  #14 PhabricatorStorageManagementUpgradeWorkflow::didExecute(PhutilArgumentParser) called at [<phabricator>/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:107]
  #15 PhabricatorStorageManagementWorkflow::execute(PhutilArgumentParser) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:485]
  #16 PhutilArgumentParser::parseWorkflowsFull(array) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:376]
  #17 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/sql/manage_storage.php:249]

epriestley · April 15, 2020, 2:54pm

This is a bug with Phabricator: the migration is not written properly. It is generally unsafe to use save() in migrations, but a few older migrations didn’t realize this and use it anyway.

Your data is almost certainly fine and this is relatively easy and routine to fix. It has only escaped notice until now because the migration is so old.

This migration changes the view and edit policies of Wiki documents under projects/ to explicitly be “Members of Project X”. Prior to this migration, this policy was implicit. This feature was removed, and this migration keeps policies the same across the feature removal.

If you haven’t already, taking a backup now might be a good idea.

Once you have a backup, here are some possible ways to proceed:

Skip the Migration

If you do not use Phriction heavily and/or are comfortable resetting the policies on every page to “All Users can View / Edit”, you can skip this migration by making this change:

diff --git a/resources/sql/autopatches/20141107.phriction.policy.2.php b/resources/sql/autopatches/20141107.phriction.policy.2.php
index 5e00bd7a40..2b7cc251de 100644
--- a/resources/sql/autopatches/20141107.phriction.policy.2.php
+++ b/resources/sql/autopatches/20141107.phriction.policy.2.php
@@ -1,5 +1,7 @@
 <?php
 
+return;
+
 $table = new PhrictionDocument();
 $conn_w = $table->establishConnection('w');

Then run bin/storage upgrade again. If it succeeds, you may be fine as-is, or may need to adjust some policies manually in the database. I can walk you through this in more detail if you end up here.

Migrate in Steps

Downgrade to a version of Phabricator where this migration worked. These are likely working versions (the state of the repositories on Nov 7, 2014):

phabricator 3fa8b152b63541753f1d07af72fa63187fab8a0c
arcanist 52947cfd92b331916adffa1ff3233546446c881b
libphutil 43a0b6d17bb097d76ef5ef936c91fa7448212cc2

To migrate in steps:

Run git checkout <hash> in each repository.
Run bin/storage upgrade, which should make it through 20141107.phriction.policy.2.php.
Run git checkout in each repository to get back to the modern version (master or stable).
Run bin/storage upgrade again.

Wait for a Fix

This is likely easy to fix, but it may be a few hours before I can get a fix into the upstream.

(If you get completely stuck at some point, go to admin.phacility.com and create a new support pact, don’t pay any invoices, and upload your database dump. I can upgrade it for you. This is not normally in the scope of free support, but the issue you’re encountering is a Phabricator bug that’s strictly the upstream’s fault.)

epriestley · April 15, 2020, 3:07pm

I believe I’ve fixed this in https://secure.phabricator.com/D21124. This fix is now available in master and stable, so these steps are likely to work now:

Run git pull in phabricator/.
Run bin/storage upgrade again.

edshrock · April 15, 2020, 5:25pm

Thanks Evan,

Unfortunately did not work. Got:

Applying patch “phabricator:20160223.paste.fileedges.php” to host “localhost”…
[2020-04-15 17:24:15] EXCEPTION: (AphrontSchemaQueryException) #1146: Table ‘phabricator_paste.paste’ doesn’t exist at [/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:356]
arcanist(head=stable, ref.master=377ed2ed8db7, ref.stable=729100955129), phabricator(head=stable, ref.master=0ea6d131e072, ref.stable=14409a32e74e), phutil(head=stable, ref.master=720c8116845b, ref.stable=034cf7cc3994)
#0 AphrontBaseMySQLDatabaseConnection::throwCommonException(integer, string) called at [/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:379]
#1 AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:320]
#2 AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:216]
#3 AphrontBaseMySQLDatabaseConnection::executeQuery(PhutilQueryString) called at [/src/infrastructure/storage/xsprintf/queryfx.php:8]
#4 queryfx(AphrontMySQLiDatabaseConnection, string, PhabricatorPaste, integer, integer, PhutilQueryString)
#5 call_user_func_array(string, array) called at [/src/infrastructure/storage/xsprintf/queryfx.php:13]
#6 queryfx_all(AphrontMySQLiDatabaseConnection, string, PhabricatorPaste, integer, integer, PhutilQueryString)
#7 call_user_func_array(string, array) called at [/src/infrastructure/storage/connection/AphrontDatabaseConnection.php:52]
#8 AphrontDatabaseConnection::queryData(string, PhabricatorPaste, integer, integer, PhutilQueryString)
#9 call_user_func_array(array, array) called at [/src/infrastructure/storage/lisk/LiskDAO.php:536]
#10 LiskDAO::loadRawDataWhere(string, integer, integer)
#11 call_user_func_array(array, array) called at [/src/infrastructure/storage/lisk/LiskDAO.php:478]
#12 LiskDAO::loadAllWhere(string, integer, integer) called at [/src/infrastructure/storage/lisk/LiskMigrationIterator.php:37]
#13 LiskMigrationIterator::loadPage() called at [/src/utils/PhutilBufferedIterator.php:129]
#14 PhutilBufferedIterator::next() called at [/src/utils/PhutilBufferedIterator.php:84]
#15 PhutilBufferedIterator::rewind() called at [/resources/sql/autopatches/20160223.paste.fileedges.php:10]
#16 require_once(string) called at [/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:291]
#17 PhabricatorStorageManagementAPI::applyPatchPHP(string) called at [/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:245]
#18 PhabricatorStorageManagementAPI::applyPatch(PhabricatorStoragePatch) called at [/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:1157]
#19 PhabricatorStorageManagementWorkflow::doUpgradeSchemata(array, NULL, boolean, boolean) called at [/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:903]
#20 PhabricatorStorageManagementWorkflow::upgradeSchemata(array, NULL, boolean, boolean) called at [/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementUpgradeWorkflow.php:78]
#21 PhabricatorStorageManagementUpgradeWorkflow::didExecute(PhutilArgumentParser) called at [/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementWorkflow.php:107]
#22 PhabricatorStorageManagementWorkflow::execute(PhutilArgumentParser) called at [/src/parser/argument/PhutilArgumentParser.php:457]
#23 PhutilArgumentParser::parseWorkflowsFull(array) called at [/src/parser/argument/PhutilArgumentParser.php:349]
#24 PhutilArgumentParser::parseWorkflows(array) called at [/scripts/sql/manage_storage.php:249]

epriestley · April 15, 2020, 5:58pm

You can safely skip that one, it fixes an issue introduced in 2015 which you can’t possibly be affected by since you’ve never run code from 2015.

Apply this patch:

diff --git a/resources/sql/autopatches/20160223.paste.fileedges.php b/resources/sql/autopatches/20160223.paste.fileedges.php
index 915a2fcd0e..6352b41c48 100644
--- a/resources/sql/autopatches/20160223.paste.fileedges.php
+++ b/resources/sql/autopatches/20160223.paste.fileedges.php
@@ -1,5 +1,7 @@
 <?php
 
+return;
+
 // For a while in November 2015, attachment edges between pastes and their
 // underlying file data were not written correctly. This restores edges for
 // any missing pastes.

…then run bin/storage upgrade again.

edshrock · April 15, 2020, 6:04pm

Hi Evan,

I did a little digging and it looks like the mysql *.MYD and .MYI files got nuked under phabricator_. The only files in those directories are *.opt and *.frm.

I’m checking to see if we have a backup.

epriestley · April 15, 2020, 6:09pm

I filed the paste migration issue upstream here: https://secure.phabricator.com/T13510.

Phabricator (mostly) uses InnoDB as a storage engine, not MyISAM, so I do not expect any .myd or .myi files to exist on disk. Data will be stored in ibdata1 or in .ibd files, depending on the innodb_file_per_table configuration setting.

This second error is also a bug in Phabricator, not a problem with your data. Phabricator would fail long before reaching this migration code if the database data was missing or corrupt.

epriestley · April 15, 2020, 6:16pm

Specifically, Phabricator has approximately 1,050 migrations to apply between 2014 and today.

The first migration error was roughly in migration #250. This second migration error is roughly in migration #570. So you’ve successfully applied about 568 migrations and run into errors with 2. You have about 500 migrations remaining. It is less likely (but still possible) that you will encounter errors in the more recent migrations than in older migrations.

It is very unlikely that there is any problem with the data itself: you’ve successfully applied 560+ migrations to it. At a minimum, neither of these errors indicate problems with the data, and the absence of .myd and .myi files also does not indicate a problem with the data.

edshrock · April 15, 2020, 7:38pm

Got entire migration to complete. I’m able to view/navigate Phabricator in my browser now.

However… Most of the commits/files I see are really crusty (2016). On the dashboard, I see recent commits under “Active Repositories” - GOOD. If I I click on a recent commit I see recent commit - GOOD. However, if I click on the name of a repo under “Active Repositories” (goes into diffusion), I see no recently dated files (all are 2016 or earlier). Clicking on Branches, the master branch shows a crusty commit and 2 other crusty branches.

I have not yet turned on the daemon.

epriestley · April 15, 2020, 7:50pm

Activating the daemons will probably fix this.

Older versions of Phabricator sometimes cloned repositories as full working copies rather than --bare working copies. If your repositories are Git or Mercurial repositories hosted elsewhere (for example, you import them from GitHub), you can move the working copies on disk somewhere else (like ~/old-repo-backup) and let Phabricator re-clone them. Phabricator will not automatically convert non-bare working copies into --bare working copies, and support for non-bare working copies is slowly being removed.

Phabricator should be able to automatically fix all other reasonable problems with repository state for repositories hosted elsewhere.

edshrock · April 15, 2020, 8:04pm

Started daemon. Golden now. Thanks again!