Database error after upgrade whole Phabricator at 02.13


#1

Last update was from Jan 29 so today we decide to upgrade.
Using all known bash script for that (git update to stable branch):

phabricator 7dd3b53a4b93e1f6d2ddf6ff1b4f213c62e6235a (Sat, Feb 3) (branched from 032f5b22941f85340b70da6d164d9dc17cb1b0f9 on origin) arcanist 886f6e6360ac6069ca8b8af12f69523deee6feda (Fri, Jan 19) (branched from 2e02332216c692e9bcedfa95d30602173050d7ca on origin) phutil fb082fb35626707d8620d5e486a03dbbd2c33019 (Fri, Feb 2) (branched from cc343d0c81335bd6c882a21c4e76ea7e9e988367 on origin)

Now all interaction with task and projects like edit/create ends with

Unhandled Exception (“AphrontSchemaQueryException”)
#1054: Unknown column ‘originalTitle’ in ‘field list’

NOTE: This usually indicates that the MySQL schema has not been properly upgraded. Run ‘bin/storage upgrade’ to ensure your schema is up to date.

After second run of update script:

MySQL needs to copy table data to make some adjustments, so these migrations may take some time.

Apply these schema adjustments? [y/N] y

Applying schema adjustments…
Done.

Target Error
phabricator_harbormaster.harbormaster_buildmessage.key_buildtarget #1072: Key column ‘buildTargetPHID’ doesn’t exist in table

Failed to make some schema adjustments, detailed above.
For help troubleshooting adjustments, see “Managing Storage Adjustments” in the documentation.

Target Error
phabricator_differential.differential_hunk.dataType Surplus
phabricator_differential.differential_hunk.dataEncoding Surplus
phabricator_differential.differential_hunk.dataFormat Surplus
phabricator_differential.differential_hunk.data Surplus
phabricator_differential.differential_hunk.changes Missing
phabricator_differential.differential_hunk_modern Missing
phabricator_harbormaster.harbormaster_buildmessage.receiverPHID Surplus
phabricator_harbormaster.harbormaster_buildmessage.buildTargetPHID Missing

SCHEMATA ERRORS

The schemata have errors (detailed above) which the adjustment workflow can
not fix.

Some of these errors are caused by surplus schemata (extra tables or columns
which Phabricator does not expect). These are not serious. For information on
resolving these issues, see the “Surplus Schemata” section in the “Managing
Storage Adjustments” article in the documentation.

If you are not developing Phabricator itself, report this issue to the
upstream.

How to proceed with the update?
bin/storage upgrade don’t resolve that issue.
Thank you in advance


#2

Your database scheme seems broken - you have missing tables and missing columns - but some of the tables seem to be broken and the upgrade workflow doesn’t even notice.

  1. Did you restart the webserver and php-fpm after upgrading? That might explain the exception in the UI.
  2. Did you ever run any modifications on the DB using any way other than bin/storage upgrade? Did you have errors in the upgrade script you may have ignored?
  3. Did you upgrade your install to master, and then changed to stable after running bin/storage upgrade?

#3

1.Yes
2.No and no errors during update
3.If I recall right that could be in 2014


#4

visit Config -> Database Issues, to see a list of all the problems you have in your db schema.

I think you had an instance of (3) in the last few days, because the errors shown in your ./bin/storage upgrade log match running the patches that are currently in master but not in stable, and then changing the code to 7dd3b53a4b93e1f6d2ddf6ff1b4f213c62e6235a. Specifically:

  • patches 20180210.hunk.01.droplegacy.sql and 20180210.hunk.02.renamemodern.sql removes differential_hunk_modern and modify differential_hunk, and
  • 20180212.harbor.01.receiver.sql renames a field from buildTargetPHID to receiverPHID.

My guess is that upgrading phabricator repo to the current master, running bin/storage upgrade and restarting phabricator will bring you into a working state. Next time you wish to switch to stable, make sure to pick a point that’s ahead of your then instance, and switch directly to that.


#5

Thank you for response.
At status we have problems with phabricator_differential, phabricator_harbormaster,phabricator_herald,phabricator_maniphest etc…
Issues are a lot:


And yes I think i’d catch your point. Could be arcanist or libphutil was at different branch (production Phab vm was copied from test one).
Will switch all modules to latest master and will check. Feedback asap.

#6

Switch to master, upgrade went fine, storage upgrade ok.
Restart Apache and phd daemons. Storage is ok.
The interface still shows error

#1054: Unknown column ‘originalTitle’ in ‘field list’

NOTE: This usually indicates that the MySQL schema has not been properly upgraded. Run ‘bin/storage upgrade’ to ensure your schema is up to date.

Database issues in Config are not less :confused:


#7

Additional digging:
Uknown originalTitle is in

phabricator/resources/sql/autopatches# grep originalTitle *
20180207.mail.01.task.sql: DROP originalTitle;
20180207.mail.02.revision.sql: DROP originalTitle;
That patches are applied
/home/phabricator/phabricator/bin/storage status | grep '20180207.mail.01.task.sql|20180207.mail.02.revision.sql’
phabricator:20180207.mail.01.task.sql localhost Applied 297,605 us sql /home/phabricator/phabricator/resources/sql/autopatches/20180207.mail.01.task.sql
phabricator:20180207.mail.02.revision.sql localhost Applied 46,123 us sql /home/phabricator/phabricator/resources/sql/autopatches/20180207.mail.02.revision.sql

Could I re-apply that patches from 20180207 ??


#8

After few back and forth (move table, re-apply sql/php patches) and manual creation of fields,ends up with:

Target Error
phabricator_differential.differential_hunk.changes Wrong Nullable Setting

Could you advice me how to recreate differential database or just hunk table proper way.
It’s not important table for us but isn’t a option bin/storage quickstart.
In quickstart.sql I’m seeing table definition but can’t find variables like $COLLATE_TEXT and can’t recreate that field.


#9

./bin/storage adjust should fix the Nullable setting, or you can fix it manually (alter table phabricator_differential.differential_hunk changes..... NOT NULL).

To get the proper schemas, you can do a fresh install on a different machine, and then use describe create phabricator_differential.differential_hunk to get the right schema for the table, and then apply it to your production table; But the Database Status page already provides you with the exact schema it expects, so that wouldn’t really help.

originalTitle doesn’t appear anywhere in the current codebase - it looks like your webserver is still running old code, which means it wasn’t restarted properly.


#10

Thank you so much for effort. I feel somekind lost :frowning:
Made switch to master, update to latest push (Thu Feb 15) onto all components - phab,arcanist,libphutil.
bin/storage update pass ok, no errors:

Storage is up to date. Use “storage status” for details.
Synchronizing static tables…
Verifying database schemata on “localhost”…
Found no adjustments for schemata.
ANALYZE Analyzing tables…
Done.
ANALYZED Analyzed 500 table(s).
BUT in config/dbissue there is alot of errors. Im worried about Maniphest table because that’s important to us.


bin/storage adjust don’t change anything with that.
and at the end:

I’ll think to report it in mainstream at secure.phabricator.com
I’m realize that is not a code/database problem but that happens in real life database.
EDIT: Wow I can’t anymore create task for bug at secure.phabricator.com ? Ok than…

Let’s digg little deeper. Grep all autopaches and database setup for maniphest_task.originalTitle.

/home/phabricator/phabricator/resources/sql/autopatches/20180207.mail.02.revision.sql:  DROP originalTitle;
/home/phabricator/phabricator/resources/sql/autopatches/20180207.mail.01.task.sql:  DROP originalTitle;
/home/phabricator/phabricator/resources/sql/patches/threadtopic.sql:  ADD originalTitle varchar(255) NOT NULL AFTER title;
/home/phabricator/phabricator/resources/sql/patches/threadtopic.sql:  originalTitle = title;
/home/phabricator/phabricator/resources/sql/patches/threadtopic.sql:  ADD originalTitle text NOT NULL AFTER title;
/home/phabricator/phabricator/resources/sql/patches/threadtopic.sql:  originalTitle = title;
/home/phabricator/phabricator/resources/sql/quickstart.sql:  `originalTitle` varchar(255) COLLATE {$COLLATE_TEXT} NOT NULL,
/home/phabricator/phabricator/resources/sql/quickstart.sql:  `originalTitle` longtext COLLATE {$COLLATE_TEXT} NOT NULL,

If I assume right - in fresh install originalTitle is created, but at last autopatch from 20180207 is droped.
But who and from where still looking for that field ?

EDIT: We have custom filed in task form, could be that a problem ???


#11

Copied problem table structure from another Phab install, storage adjust and storage upgrade.
Problem solved.
Thank you for the support !