On Tue 2016-02-09 20:01:43 -0500, Daniel Kahn Gillmor wrote:
>> I just wanted to mention that I think there's a problem with the reindex
>> functionality introduced in this patch (or in 16/16).  It looks like
>> this function irrevocably busts apart threads.  dkg and I are
>> investigating.
>
> it doesn't appear to be irrevocable to me, but it is definitely doing
> something weird with threading.

OK, this is definitely tickling some problems with threading, but those
are problems that are present already in existing versions of notmuch,
unrelated to this series.

When removing a message from the database, its earlier presence doesn't
become a ghost message, and as a result anything that points to it
doesn't get assembled into the prior thread properly.

The attached tarball has a python test showing this behavior with a
simple thread of two messages:

0 dkg@frigg:~/src/notmuch/threading-test$ ./run-test 
Found 2 total files (that's not much mail).
Processed 2 total files in almost no time.
Added 2 new messages to the database.
Threads: 1
removing and re-adding a@example.com
Threads: 2
removing and re-adding b@example.com
Threads: 1
0 dkg@frigg:~/src/notmuch/threading-test$ 

the relevant python function is:


def remove_and_readd(db, mid):
    print('removing and re-adding', mid)
    m = db.find_message(mid)
    f = m.get_filename()
    db.remove_message(f)
    db.add_message(f)



I think when a message is removed from the database, we need to know
whether anything else (in its same thread?) refers to it.  If so, we
should keep it around as a ghost message instead of fully removing it.

does this sound like the right approach?

     --dkg