Re: [PATCH] test: report summary even when aborting

Subject: Re: [PATCH] test: report summary even when aborting

Date: Sat, 25 May 2019 22:41:58 +0300

To: Daniel Kahn Gillmor, Notmuch Mail

Cc:

From: Tomi Ollila

On Sat, May 25 2019, Daniel Kahn Gillmor wrote:

> In certain cases of test suite failure, the summary report was not
> being printed.  In particular, any failure on the parallel test suite,
> and any aborted test in the serialized test suite would end up hiding
> the summary.
>
> It's better to always show the summary where we can (while preserving
> the return code).

This is an improvement, but it looks like it is possible
aggregate-results.sh prints everything good (like all 3 tests passed)
since it may be that there are only good logs in test-results/.

That might, in some cases, be confusing (exit value of notmuch-test
is not always visible). 

One option could be giving $RES to aggregate-results.sh and if it is
not zero then that could print different text -- just that it requires
a bit more coding...

Tomi

>
> Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
> ---
>  test/notmuch-test | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/test/notmuch-test b/test/notmuch-test
> index 50ed8721..d835e152 100755
> --- a/test/notmuch-test
> +++ b/test/notmuch-test
> @@ -45,6 +45,8 @@ else
>  fi
>  
>  trap 'e=$?; kill $!; exit $e' HUP INT TERM
> +
> +RES=0
>  # Run the tests
>  if test -z "$NOTMUCH_TEST_SERIALIZE" && command -v parallel >/dev/null ; then
>      test -t 1 && export COLORS_WITHOUT_TTY=t || :
> @@ -58,7 +60,6 @@ if test -z "$NOTMUCH_TEST_SERIALIZE" && command -v parallel >/dev/null ; then
>      RES=$?
>      if [[ $RES != 0 ]]; then
>          echo "parallel test suite returned error code $RES"
> -        exit $RES
>      fi
>  else
>      for test in $TESTS; do
> @@ -69,7 +70,7 @@ else
>          RES=$?
>          testname=$(basename $test .sh)
>          if [[ $RES != 0 && ! -e "$NOTMUCH_BUILDDIR/test/test-results/$testname" ]]; then
> -            exit $RES
> +            echo "Aborting on $testname (returned $RES)"
>          fi
>      done
>  fi
> @@ -78,7 +79,11 @@ trap - HUP INT TERM
>  # Report results
>  echo
>  $NOTMUCH_SRCDIR/test/aggregate-results.sh $NOTMUCH_BUILDDIR/test/test-results/*
> -ev=$?
> +if [ "$RES" = 0 ]; then
> +    ev=$?
> +else
> +    ev=$RES
> +fi
>  
>  # Clean up
>  rm -rf $NOTMUCH_BUILDDIR/test/test-results
> -- 
> 2.20.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
_______________________________________________
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Thread: