bug: "no top level messages" crash on Zen email loops

Hi!

Here's a fun bug for you Xapian tricksters.

Two emails attached make notmuch crash when trying to display the
folder.

$ notmuch show thread:0000000000000001
Internal error: Thread 0000000000000001 has no toplevel messages.
 (notmuch-show.c:1012)

Those are the two messages:

$ notmuch search --output messages  thread:0000000000000001
id:9379QM5Z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sprut@zendesk.com
id:9379QM5Z39_5aa86b1350504_174eb3fc97a2cb98c71674_sprut@zendesk.com

`notmuch show` on either messages crashes the same way:

$ notmuch show id:9379QM5Z39_5aa86b1350504_174eb3fc97a2cb98c71674_sprut@zendesk.com
Internal error: Thread 0000000000000001 has no toplevel messages.
 (notmuch-show.c:1012)

Note that displaying the messages weith `--format raw` doesn't crash, so
it's really the thread structure that's broken. Obviously, emacs can't
display the messages either and doesn't touch the unread tags when
trying to load the message, which is to be expected I guess.

Xapian is also unhappy with the database created by notmuch new:

$ xapian-check gitlab/.notmuch/xapian/
docdata:
blocksize=8K items=1 firstunused=1 revision=7 levels=0 root=0
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=12 firstunused=4 revision=7 levels=0 root=3
xapian-check: DatabaseError: 1 unused block(s) missing from the free list, first is 0

Valgrind is not particularly unhappy with notmuch, so it doesn't seem
like a memory error:

==26723== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

I tried to track this down in gdb, i got as far as finding that, in
`notmuch_thread_get_toplevel_messages`, the `list` object is corrupt (?)
already (`list->head == NULL`) which obviously makes it hard to, er,
list messages in a thread. :p I lost the exact backtrace and so on, but
I'm not sure there's much we can get from gdb: it seems the problem
might be in notmuch-new, but I'm a little out of my depth to debug
*that* without any further pointers.

This is with 0.26-1~bpo9+1 on Debian stretch, but I can also reproduce
with 0.23 on another Debian stretch machine, using a similar mail
spool.

My guess is that those messages are somewhat special: notice how the
reply-to identifiers *loop* between the two messages?

Message one:

Message-ID: <9379QM5Z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sprut@zendesk.com>
In-Reply-To: <9379QM5Z39@zendesk.com>
 <9379QM5Z39_5aa86b1350504_174eb3fc97a2cb98c71674_sprut@zendesk.com>

Message two:

Message-ID: <9379QM5Z39_5aa86b1350504_174eb3fc97a2cb98c71674_sprut@zendesk.com>
In-Reply-To: <9379QM5Z39@zendesk.com>
 <9379QM5Z39_5aa86b134fcfb_174033fc97a2cb98c7198d_sprut@zendesk.com>

And indeed, a mailbox with only *one* of those messages doesn't cause
the crash. But also: the original thread is now made of *three*
messages, and taking any one of the two messages above with that *third*
message doesn't cause the crash:

Message three:

Message-ID: <9379QM5Z39_5aaf79c126a_94233ffb30ecb9982187c0_sprut@zendesk.com>
In-Reply-To: <9379QM5Z39@zendesk.com>
 <9379QM5Z39_5aa86b1350504_174eb3fc97a2cb98c71674_sprut@zendesk.com>

This message also shows correctly threaded with message two if present,
otherwise the thread is (obviously) broken with only message one and
three.

Mutt displays those messages as a "normal" three-level thread:

   1   ! mar 14 GitLab Support  (4,7K) Your GitLab support request has been received
   2   ! mar 14 GitLab Support  (4,5K) └>comments not showing up?
   3 O ! mar 19 XXXXXXXXXXXXXXX (7,9K)  └>[GitLab, Inc.] Re: comments not showing up?

The numbers on the left (1, 2, 3) correspond to the labeling I used
above as well (one, two, three).

The third message is not included here because it's an actual reply from
a human from GitLab (yay gitlab! :) which I'd need approval before
sharing here. The first message is an automated response so I thought it
was fair game to share publicly. The second is a copy of my own message
which triggered the autoreply, which is probably the source of the
loop. The software generating this mess is Zendesk.com. I haven't had
that problem with other interactions with Zendesk, maybe because I
never talked with a Zendesk that sent autoreplies.

To reproduce this, untar the attachment anywhere (say $HOME) and then
hack a notmuch config file pointing there, e.g.:

$ diff .notmuch-config*
15c15
< path=/home/anarcat/Maildir/
---
> path=/home/anarcat/gitlab/

Then point notmuch to that config (export
NOTMUCH_CONFIG=~/.notmuch-config-test) and run notmuch new (which should
find only two messages). Then run the commands from the above of this
email, of course. :)

Thanks for any input,

A.

PS: I must say I am grateful and impressed by the reliability of
notmuch. I've been using notmuch for *years* now and it's the *first*
time, for as long as I remember, that I had to go back to mutt to read
email. So kudos to the team, good job. :)

-- 
Si les élections n'étaient pas indispensables à la prospérité du
capital, on ne nous les servirait pas partout, toujours, à coup de
fric, à coup de flics.
                        - René Binamé

bug: "no top level messages" crash on Zen email loops

Thread: