Conduit API: file.uploadchunk issue

I have trouble understanding how the file.uploadchunk exactly should be used.

My understanding is that you first call the file.allocate method, from which you can retrieve the filePHID, to be used as parameter in the file.uploadchunk method.

The dataEncoding parameter of the file.uploadchunk is set to “base64”, byteStart parameter is set to 0 for the first chunk.
But I don’t know what should be in the data parameter: what is the size of a chunk ?
I suppose the data parameter contains the bytes to be uploaded formatted as a base64 string.

If I use a chunk of 2MB or more, I get the following (PHP?) error message:

escapeshellarg(): Argument exceeds the allowed length of 2097152 bytes

If I use a chunk of less than 2MB, I get a Conduit error message like this:

ERR-CONDUIT-CORE
Chunk end byte is 4194304, not 284.

This API is not really intended to be user-facing and we’ll likely provide a simpler API in the future. That said, the general shape of the API is:

To initialize an upload:

  • Call file.allocate to get a PHID.

To start or resume an upload:

  • Call file.querychunks to get a list of chunks the server has and needs.
  • Call file.uploadchunk once for each chunk, providing the data for the chunk.

This generally supports clients that can resume uploads and looks something like:

file_phid = get_saved_phid();
if (!file_phid) {
  file_phid = file.allocate();
  save_phid_for_resume();
}

need_chunks = file.querychunks();
foreach-in-parallel (need_chunks as chunk) {
  file.uploadchunk(chunk, chunk_data);
}

clear_saved_phid();

In Arcanist, the class ArcanistFileUploader has a practical implementation of these methods and isn’t //too// much longer than this pseudocode (~300 lines). (But note that it uses content hashing with file.allocate to support resumes, not saving the file PHID.)

The advantages of this API are:

  • clients can upload multiple chunks in parallel; and
  • clients can resume an upload from any state.

However, the API is relatively complex.

It’s likely that we’ll add an alternate, simpler API in the future (for example, an API which only supports sequential upload and does not support resumes, but allows you to just POST data in a loop in the common case where you don’t care about these things).

See also https://secure.phabricator.com/T13337.

Thanks, I got it working.now.