Re: Test suite timing issues?

Subject: Re: Test suite timing issues?

Date: Tue, 15 Feb 2022 09:51:01 +0100

To: David Bremner, Tomi Ollila


From: Michael J Gruber

Tomi Ollila venit, vidit, dixit 2022-02-14 23:53:54:
> On Mon, Feb 14 2022, David Bremner wrote:
> > Tomi Ollila <> writes:
> >
> >>
> >> Looked notmuch-new.c -- time_t (seconds since epoch) is used as timestamp
> >> comparisons (which would indicate the subsecond resolution most fs' provide
> >> is not used)...
> >>
> >> ... and if so, I wonder why some of our tests are not failing all the time
> >> for everyone...?
> >
> > Not claiming everything is fine, but there is code there targetted at
> > the failure mode you mentioned:
> >
> >     /* If the directory's mtime is the same as the wall-clock time
> >      * when we stat'ed the directory, we skip updating the mtime in
> >      * the database because a message could be delivered later in this
> >      * same second.  This may lead to unnecessary re-scans, but it
> >      * avoids overlooking messages. */
> >     if (fs_mtime != stat_time)
> >       _filename_list_add (state->directory_mtimes, path)->mtime = fs_mtime;
> This sure had to be tested... :D
> so I outcommented the line as // if (fs_mtime != stat_time)
> and then build and run test -- a lot of failures...
> also
> ./test/ 2>&1 | grep -e PASS -e FAIL
>  PASS   Single new gzipped message
>  PASS   Single new gzipped message (full-scan)
>  FAIL   Multiple new messages, one gzipped
>  FAIL   Multiple new messages, one gzipped (full-scan)
>  FAIL   Renamed (gzipped) message
>  PASS   notmuch search with partially gzipped mail store
>  FAIL   notmuch search --output=files with partially gzipped mail store
>  PASS   show un-gzipped message
>  PASS   show un-gzipped message (format mbox)
>  PASS   show un-gzipped message (format raw)
>  FAIL   show gzipped message
>  FAIL   show gzipped message (mbox)
>  FAIL   show gzipped message (raw)
>  PASS   new doesn't run out of file descriptors with many gzipped files
> (above was "lucky" run, usually that 6th test, ...partially gzipped...
>  test also FAILed (I'd guess second happened to change there)).
> then restored the fs_mtime != stat_time line -- then all of 750 passed.
> (finally, run that 750-gzip in a loop (dropped last, slow test), hundreds
>  of times already -- no FAILures... (ecryptfs on ext4))
> Tomi
> > BTW, I have so far run the test suite 68 times in a row without failures
> > on a Debian s390x host. The file system is ext4, mounted relatime. It
> > would be interesting to know what file system is yielding the failures
> > Michael is seeing.

Thanks for the detailed analysis. It convinces me that on notmuch's side
everything is OK.

This very much boils down to fs and stat issues.

If I remember correctly, I've seen this isse once on a different release
frpm epel-8 but only on epel-8 otherwise. This is on a build
infrastructure where you may end up getting different hosts on each run,
primed from the same "chroot". I'll try to find out more.

Until now, we have been building notmuch release packages on Fedora
without running the tests during the release build, because there used
to be issues with the test suite (and before that that the separate test
corpus download). With everything looking robust for a while, I'm
switching tests on for Fedora release builds now, and at the same start
packaging those extra packages for RedHat's enterprise platform. I want
to avoid spurious release build failures, of course - otoh having those
tests run is a good thing.

So I'll try out a bit more, and if everything else fails, then running
the test suite with --full-scan is still better than not running it at
all. (I would have suggested a make variable, but it's really a corner
case. given the stat/mtime check).

notmuch mailing list --
To unsubscribe send an email to