Maniphest searches based on custom due date attribute are broken

Observed Behavior:
We have a custom attribute named “easypost.due-date” with a the following type definition:

  "easypost.due-date": {
    "name": "Due Date",
    "type": "date",
    "copy": true,
    "search": true
  },

We have (for more than a year) had a saved search with the following definition: Due Date Before “7 days from now”, filtered by Any Open Status, ordered by Due Date. Since the most recent phabricator bump we did, this now fails with the following exception:

 [2019-06-06 10:16:58] EXCEPTION: (Exception) Map returned by "newPagingMapFromCursorObject()" in class "ManiphestTaskQuery" omits required key "custom.easypost.due-date". at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:228]
 arcanist(), phabricator(custom=6), phutil()
   #0 <#2> PhabricatorCursorPagedPolicyAwareQuery::getPagingMapFromCursorObject(PhabricatorQueryCursor, array) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:636]
   #1 <#2> PhabricatorCursorPagedPolicyAwareQuery::buildPagingClause(AphrontMySQLiDatabaseConnection) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:567]
   #2 <#2> PhabricatorCursorPagedPolicyAwareQuery::buildPagingWhereClause(AphrontMySQLiDatabaseConnection) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:489]
   #3 <#2> PhabricatorCursorPagedPolicyAwareQuery::buildWhereClauseParts(AphrontMySQLiDatabaseConnection) called at [<phabricator>/src/applications/maniphest/query/ManiphestTaskQuery.php:355]
   #4 <#2> ManiphestTaskQuery::buildWhereClauseParts(AphrontMySQLiDatabaseConnection) called at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:479]
   #5 <#2> PhabricatorCursorPagedPolicyAwareQuery::buildWhereClause(AphrontMySQLiDatabaseConnection) called at [<phabricator>/src/applications/maniphest/query/ManiphestTaskQuery.php:244]
   #6 <#2> ManiphestTaskQuery::loadPage() called at [<phabricator>/src/infrastructure/query/policy/PhabricatorPolicyAwareQuery.php:248]
   #7 <#2> PhabricatorPolicyAwareQuery::execute() called at [<phabricator>/src/infrastructure/query/policy/PhabricatorCursorPagedPolicyAwareQuery.php:374]
   #8 <#2> PhabricatorCursorPagedPolicyAwareQuery::executeWithCursorPager(AphrontCursorPagerView) called at [<phabricator>/src/applications/search/engine/PhabricatorApplicationSearchEngine.php:1038]
   #9 <#2> PhabricatorApplicationSearchEngine::executeQuery(ManiphestTaskQuery, AphrontCursorPagerView) called at [<phabricator>/src/applications/search/controller/PhabricatorApplicationSearchController.php:255]
   #10 <#2> PhabricatorApplicationSearchController::processSearchRequest() called at [<phabricator>/src/applications/search/controller/PhabricatorApplicationSearchController.php:91]
   #11 <#2> PhabricatorApplicationSearchController::processRequest() called at [<phabricator>/src/aphront/AphrontController.php:29]
   #12 <#2> AphrontController::handleRequest(AphrontRequest) called at [<phabricator>/src/aphront/AphrontController.php:71]
   #13 <#2> AphrontController::delegateToController(PhabricatorApplicationSearchController) called at [<phabricator>/src/applications/maniphest/controller/ManiphestTaskListController.php:20]
   #14 <#2> ManiphestTaskListController::handleRequest(AphrontRequest) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:286]
   #15 phlog(Exception) called at [<phabricator>/src/aphront/handler/PhabricatorDefaultRequestExceptionHandler.php:41]
   #16 PhabricatorDefaultRequestExceptionHandler::handleRequestThrowable(AphrontRequest, Exception) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:744]
   #17 AphrontApplicationConfiguration::handleThrowable(Exception) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:298]
   #18 AphrontApplicationConfiguration::processRequest(AphrontRequest, PhutilDeferredLog, AphrontPHPHTTPSink, MultimeterControl) called at [<phabricator>/src/aphront/configuration/AphrontApplicationConfiguration.php:209]
   #19 AphrontApplicationConfiguration::runHTTPRequest(AphrontPHPHTTPSink) called at [<phabricator>/webroot/index.php:35]

This appears to only fail with all three constraints: removing the filter on Any Open Status makes it not crash, ordering by anything other than Due Date makes it not crash, and removing the filter on Due Date makes it not crash, but having all three in place makes it crash 100% of the time.

Expected Behavior:
Returning the report that it used to return

Phabricator Version:
Internal fork based on 2019 Week 21.

Reproduction Steps:

  1. Add a custom type with the appropriate definition
  2. Create some tasks with a non-empty Due Date
  3. Attempt to perform an advanced Maniphest search with the filters described

Your minimum repro doesn’t seem to be working for me. My maniphest.custom-field-definitions:

And the custom query I’m running:

Are there some additional steps required to reproduce this issue?

Okay.

I wiped my testing VM and recreated it.

The traceback is around pagination, so that lead me to the conclusion that the missing reproduction step was volume.

Easiest way:

  • create the custom attribute
  • create 10 tasks with a due date of tomorrow
  • create the search as described in the original ticket
  • set the “Page Size” in the custom search field to 5
  • press the “Next Page” button

That triggers the same exception with the same traceback.

I imagine it’s being generated rendering the first page when the number of tickets exceeds some other pagination threshold, which is what it’s doing in production (and in my previous testing VM which had hundreds of tickets from reproducing various other bug reports).

bump

bump