[INVALID]All files fail with HMAC-SHA256 can only digest strings

Hi,

Recently I discovered that all images was gone (not showing) from our phabricator instance. It all traces back to the fact that every file in the files-application thorws an exception stating that “HMAC-SHA256 can only digest strings”. We’re using an S3 bucket as backend.

Reproduction Instructions

  1. Upload a file to the files application
  2. Try to retrieve it
  3. Get Unhandled Exception (“Exception”): HMAC-SHA256 can only digest strings.

Phabricator/Arcanist Version
phabricator
0c3f59fd688a5488cadfb9286d11fbedff766d5d (Wed, Sep 4) (branched from 533a5535b6f0104c13266400735cb563d64525e7 on origin)

arcanist
feb5f4d42c4fe0001e76428e80d5e88262308802 (Jun 22 2019) (branched from d92fa96366c0ed50e4257508148aa75192d4fb1f on origin)

phutil
794ded9857110cdc7b0fb7bedb4a64438d2d873e (Sat, Aug 31) (branched from 71e8d7a4cf8e9f56b1427c27b3684ae17a3ea7c7 on origin)

php
7.2.19-0ubuntu0.18.04.2

diff
3.6 at /usr/bin/diff

git
2.17.1 at /usr/bin/git

hg
Not Available

pygmentize
2.2.0 at /usr/bin/pygmentize

svn
Not Available

I’m unable to reproduce this by following your reproduction steps. At step (3), I get the original file.

Well… There must be something wrong at least… If it is of any help, here’s what I find in apache’s error logs:

And here’s the output if I try to “bin/files cat” a file

Hold on… On further investigation, I strongly suspect this is an S3 issue…

I’m so sorry. This was a configuration error related to the S3 backend. The error occured becasue the HMAC function didn’t actually receive any data at all. Please close this bug.

It sounds like we might be able to handle the error better – do you know how I can reproduce the S3 configuration issue offhand?

Our setup is using the S3 API implemented in Openstack Swift, and our Swift installation was temporarly unavailable, causing all of phabricator’s attempts to retrieve data to fail with an http 404 error… Maybe you can replicate it by setting up a working S3 configuration, store some files and then delete them in S3 “manually” (outside of phabricator)?

Thanks! I’ll see if I can reproduce that locally.

I filed this upstream as https://secure.phabricator.com/T13407 until I can look at it in more detail.