Data truncated when pushing into repository

Hello,

I have a problem with phabricator’s git hosted repositories. I’m unable to push anything to the repo if I don’t disable the pre-receive git hook, and I push from anywhere but PHPStorm. Don’t know what’s the difference, but in PHPStorm it works, but I have the same error in everywhere else (console, smartgit, gitkraken, …). The error message is:

➜ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 294 bytes | 294.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: [2020-02-27 18:28:00] EXCEPTION: (AphrontQueryException) #1265: Data truncated for column ‘duration’ at row 1 at [/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:386]
remote: arcanist(head=stable, ref.master=d3b77af8a526, ref.stable=729100955129), phabricator(head=stable, ref.master=e70c9f72a48e, ref.stable=ff6f24db2bc0), phutil(head=stable, ref.master=5b341cc09ca9, ref.stable=034cf7cc3994)
remote: #0 AphrontBaseMySQLDatabaseConnection::throwQueryCodeException(integer, string) called at [/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:320]
remote: #1 AphrontBaseMySQLDatabaseConnection::throwQueryException(mysqli) called at [/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:216]
remote: #2 AphrontBaseMySQLDatabaseConnection::executeQuery(PhutilQueryString) called at [/src/infrastructure/storage/xsprintf/queryfx.php:8]
remote: #3 queryfx(AphrontMySQLiDatabaseConnection, string, PhutilQueryString, HeraldTranscript, array, array) called at [/src/infrastructure/storage/connection/AphrontDatabaseConnection.php:58]
remote: #4 AphrontDatabaseConnection::query(string, PhutilQueryString, HeraldTranscript, array, array) called at [/src/infrastructure/storage/lisk/LiskDAO.php:1103]
remote: #5 LiskDAO::insertRecordIntoDatabase(string) called at [/src/infrastructure/storage/lisk/LiskDAO.php:939]
remote: #6 LiskDAO::insert() called at [/src/infrastructure/storage/lisk/LiskDAO.php:908]
remote: #7 LiskDAO::save() called at [/src/applications/herald/engine/HeraldEngine.php:265]
remote: #8 HeraldEngine::getTranscript() called at [/src/applications/diffusion/engine/DiffusionCommitHookEngine.php:365]
remote: #9 DiffusionCommitHookEngine::applyHeraldRules(array, HeraldPreCommitRefAdapter) called at [/src/applications/diffusion/engine/DiffusionCommitHookEngine.php:327]
remote: #10 DiffusionCommitHookEngine::applyHeraldRefRules(array) called at [/src/applications/diffusion/engine/DiffusionCommitHookEngine.php:165]
remote: #11 DiffusionCommitHookEngine::execute() called at [/scripts/repository/commit_hook.php:199]
To phabricator.xxxx.com:/srv/git/util.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to ‘git@phabricator.xxx.com:/srv/git/util.git’

We are two people pushing to several repositories hosted in phabricator, and we have the same problem in our two computers and all the different repositories.

I tried to find that duration row, but didn’t find it. Do you know what can I do or where can I try to find it?

I see there’s an old task with the same error, but I don’t understand what means with I'm change phabricator/.devinercache owner. Phabricator’s instance and git repositories have the same owner and group and are in the same server.


Reproduction Instructions
I really don’t know why it began to fail. It was working well for several years, then sudently I had the problem but my colleague could work with it, and now we both have the error. I updated phabricator and arcanist 2-3 times to the last version, and the whole server was restarted some days ago.

Phabricator/Arcanist Version

Library Version Date Branchpoint
phabricator ff6f24db2bc0 Wed, Feb 12 2327578adc94
arcanist 729100955129 Thu, Jan 30 21a1828ea06c
phutil 034cf7cc3994 Thu, Jan 30 cc2a3dbf5903

Server: CentOS Linux release 7.7.1908 (Core)
PHP 7.3.14 (cli) (built: Jan 21 2020 09:59:50) ( NTS )
Zend Engine v3.3.14, Copyright © 1998-2018 Zend Technologies
with Zend OPcache v7.3.14, Copyright © 1999-2018, by Zend Technologies

duration is a column on the herald_transcript table. The type of this column is “double”.

Normally, “data truncated” means that that a too-large value is being inserted into a too-small column. For example, we’re attempting to insert a string of length 567 into a varchar(100) column.

I can’t immediately figure out how to make MySQL emit this error for insertion into a double column. These insertions are all destructive (the original value is not preserved exactly) but none raise errors:

mysql> create temporary table t (d double);
Query OK, 0 rows affected (0.01 sec)

mysql> insert into t (d) values (1.000000000000000000000000000000000000000000000000000000000001);
Query OK, 1 row affected (0.00 sec)

mysql> insert into t (d) values (1.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001);
Query OK, 1 row affected (0.00 sec)

mysql> insert into t (d) values (1.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001);
Query OK, 1 row affected (0.00 sec)

mysql> insert into t (d) values (999999999999999999999999999999999999999999999);
Query OK, 1 row affected (0.00 sec)

mysql> insert into t (d) values (3333333333.333333333333333333333333333333333333333333333333333333333);
Query OK, 1 row affected (0.00 sec)

mysql> select * from t;
+--------------------+
| d                  |
+--------------------+
|                  1 |
|                  1 |
|                  1 |
|               1e45 |
| 3333333333.3333335 |
+--------------------+
5 rows in set (0.00 sec)

There may be some MySQL mode which controls this behavior, I’ll see what I can dig up.

I’m not having much luck coming up with a theory on how to build a reproduction case. Possibly helpful:

  • What version of MySQL are you running? Is it vanilla MySQL, or some other SQL (MariaDB, RDS, Google Cloud stuff, etc)?
  • What does this command show in MySQL?
mysql> SHOW VARIABLES LIKE 'sql_mode';

We are using MariaDB 10.3 on a Centos7, installed as this:

MariaDB-client.x86_64 10.3.22-1.el7.centos @mariadb
MariaDB-common.x86_64 10.3.22-1.el7.centos @mariadb
MariaDB-compat.x86_64 10.3.22-1.el7.centos @mariadb
MariaDB-devel.x86_64 10.3.22-1.el7.centos @mariadb
MariaDB-server.x86_64 10.3.22-1.el7.centos @mariadb
MariaDB-shared.x86_64 10.3.22-1.el7.centos @mariadb
galera.x86_64 25.3.28-1.rhel7.el7.centos @mariadb

This is the output of the command:

Variable_name Value
sql_mode IGNORE_SPACE,STRICT_TRANS_TABLES,STRICT_ALL_TABLES

I’ll try to put MariaDB in debug mode and see if I can catch any insert / update transaction raising an error or warning when I push.

I’ve tried with log_warnings=9 and I don’t see any error or warning. Then I’ve tried with SET GLOBAL general_log=ON; and found an error. This is the query:

INSERT INTO `phabricator_herald`.`herald_transcript` (`objectTranscript`, `ruleTranscripts`, `conditionTranscripts`, `applyTranscripts`,
`time`, `host`, `duration`, `objectPHID`, `dryRun`, `garbageCollected`, `phid`)
VALUES ('O:22:\"HeraldObjectTranscript\":6:{s:7:\"\0*\0phid\";s:30:\"PHID-PSHL-bmevpyzieqb7ikuhxpni\";s:7:\"\0*\0type\";s:25:\"HeraldPreCommitRefAdapter\";s:7:\"\0*\0name\";s:14:\"Push Log (Ref)\";s:9:\"\0*\0fields\";a:0:{}s:26:\"\0*\0appliedTransactionPHIDs\";a:0:{}s:10:\"\0*\0profile\";a:0:{}}',
'a:0:{}', 'a:0:{}', 'a:0:{}', '1582886043', 'cloud', '0,00067496299743652', 'PHID-PSHL-bmevpyzieqb7ikuhxpni', '0', '0', 'PHID-HLXS-clhsboymvrxlrmbidd4t');

Looks like the problem is that it’s sending a 0,00067496299743652 instead of 0.00067496299743652. I’ve tried the latter manually and that works.

Ah, thanks. Can you show me the output of this command on your system?

$ php -r 'var_dump(setlocale(LC_ALL, "en_US.UTF-8"));'

I suspect this is a special case of https://secure.phabricator.com/T7339.

In particular, we set LC_ALL to en_US.UTF-8 at startup and this suggests that’s is sufficient to avoid MySQL-incompatible number formatting if it succeeds:

<?php

setlocale(LC_NUMERIC, 'de_DE');

echo (string)1.23;
echo "\n";

setlocale(LC_ALL, 'en_US.UTF-8');

echo (string)1.23;
echo "\n";
$ php -f locale.php 
1,23
1.23

This locale issue is significantly more severe than previous locale issues (which generally required us to pass UTF8 multibyte characters to subprocess argument lists before they would manifest).

A possible workaround on systems which do not have “en_US.UTF-8” is to special-case phutil_string_cast() to always format floating point numbers with periods, then use phutil_string_cast() to perform all string casting.

This is totally ridiculous, but we already live in a world where (string) has questionable behavior.

This is the output:

$ php -r 'var_dump(setlocale(LC_ALL, "en_US.UTF-8"));'
string(11) "en_US.UTF-8"

Locale in server:

$ locale
LANG=es_ES.UTF-8
LC_CTYPE=es_ES.UTF-8
LC_NUMERIC=es_ES.UTF-8
LC_TIME="es_ES.UTF-8"
LC_COLLATE="es_ES.UTF-8"
LC_MONETARY=es_ES.UTF-8
LC_MESSAGES="es_ES.UTF-8"
LC_PAPER="es_ES.UTF-8"
LC_NAME="es_ES.UTF-8"
LC_ADDRESS="es_ES.UTF-8"
LC_TELEPHONE="es_ES.UTF-8"
LC_MEASUREMENT=es_ES.UTF-8
LC_IDENTIFICATION="es_ES.UTF-8"
LC_ALL=

I changed server’s locale and the push command worked, so that seems to be the error. I changed:

$ export LC_NUMERIC=en_US
$ export LANG=en_US
$ localectl set-locale LANG=en_US.utf8
$ localectl status
   System Locale: LANG=en_US.utf8
       VC Keymap: es
      X11 Layout: es

And the mysql locale (I’m not sure, but I think I didn’t change it)

$ echo "show variables like 'lc_messages';" | mysql
Variable_name   Value
lc_messages     en_US

I will keep that configuration while there’s no formal solution. I think that won’t break anything in the server. Do you know a better solution?

Thanks. I also added some more details here:

https://secure.phabricator.com/T7339

In your case, I think the most immediate issue is that we’re doing locale adjustments in PhabricatorStartup.php, but this code is only actually in the startup flow for web requests, not script/CLI requests like pushes over SSH. So a simple fix that will cover your case is to just move the existing locale adjustment code earlier in the startup process. This is a desirable change anyway.

1 Like

Great! I’m subscribed to T7339, so I’ll see if you update the task with new information or if there’s any change in source code that I can try to see if it works. Thank you!