Arc unit has different rendering behavior based on how it is configured


#1

Observed Behavior:
arc unit have different test results echoing behaviors based on if you use .arcconfig or .arcunit.

If you use .arcunit, the unit test results are not rendered.

Expected Behavior:
It should have the same output no matter which file you used to configure it.

Phabricator Version:
Not related. This cause can be found in git master of arcanist repo.

Reproduction Steps:

  1. Clone this repo: https://gitlab.com/itszero/phabricator-unit-test-rendering-bug-report
  2. Follow the README.md to reproduce it.

Thanks! I would be happy to contribute code myself too but I’ve read the contribution guideline and it seems like the community would prefer a bug report first so here it is!


#2

To work around this, implement this method on PytestConfiguredTestEngine:

  public function shouldEchoTestResults() {
    return true;
  }

With that method implemented, I get consistent output:

$ ./run_test.sh 
==> Test case 1: Using .arcconfig
==> Running 'arc unit': Expected to see unit test results
   FAIL  test_error.test_answer
def test_answer():
>       assert inc(3) == 5
E       assert 4 == 5
E        +  where 4 = inc(3)

test_error.py:5: AssertionError


COVERAGE REPORT
      0%     .gitignore
      0%     README.md
      0%     arcanist-pytest/.arcconfig
      0%     arcanist-pytest/__phutil_library_init__.php
      0%     arcanist-pytest/__phutil_library_map__.php
      0%     arcanist-pytest/unit/PytestConfiguredTestEngine.php
      0%     run_test.sh
    100%     test_error.py



==> Test case 2: Using .arcunit
==> Running 'arc unit': Unit test results are gone but is expected
   FAIL  test_error.test_answer
def test_answer():
>       assert inc(3) == 5
E       assert 4 == 5
E        +  where 4 = inc(3)

test_error.py:5: AssertionError


COVERAGE REPORT
      0%     .gitignore
      0%     README.md
      0%     arcanist-pytest/.arcconfig
      0%     arcanist-pytest/__phutil_library_init__.php
      0%     arcanist-pytest/__phutil_library_map__.php
      0%     arcanist-pytest/unit/PytestConfiguredTestEngine.php
      0%     run_test.sh
    100%     test_error.py

This overall behavior isn’t ideal, but is not likely to get changes until the changes connected to T13098 land. The “toolsets” change substantially rewrites result reporting.


#3

Thanks for the reply! That only happens to work because I had to fork the PytestTestEngine from the one in arcanist, because the one in arcanist did not implement getEngineConfigurationName.

I understand that there’s a big change coming up. I’ll try to keep a tab on that ticket then. :slight_smile: