Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

Subject: Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

Date: Wed, 18 Nov 2020 10:29:40 -0300

To: notmuch@notmuchmail.org

Cc:

From: Jorge P. de Morais Neto

Hi.  I use Notmuch 0.31.2 on Emacs 27.1.50 (manually compiled on
2020-11-02) with matching version-pinned MELPA Stable Notmuch package on
updated Debian buster.  I have enabled buster-proposed-updates,
buster-updates and buster-backports.  I manually backport notmuch
according to <https://wiki.debian.org/SimpleBackportCreation>.  I use
OfflineIMAP 7.3.3 (Python 2 pip), afew 3.0.1 (pip3), Bogofilter 1.2.4
(buster) and a custom Python 3 script based on the ~notmuch~ module.

This Monday (when still on Notmuch 0.31) I noticed that, when I tagged a
message or thread, the fido-mode completion offered many weird candidate
tags that shouldn't exist in the database.  Also, on the Notmuch Hello
screen the ~All tags~ section would error out.  I then dumped the
database (~notmuch dump~) and noticed many lines associating weird tags
to weird message ids.  In almost every case, both the weird tags and the
weird Message-Id contained uncommon characters, often ASCII control
characters.

One of the weird lines was " -- id:8"---specifying a message with
Messaged-ID "8" and no tags.  Invoking ~notmuch show id:8~ yielded
internal error---something like "message with document ID <SOME_NUMBER>
has no thread ID".

I then upgraded Notmuch to 0.31.2 and compacted the database but the
error persisted.  I then manually cleaned up the database dump, deleted
the ~/offlineimap/Jorge-Disroot/.notmuch/xapian/ directory, invoked
~notmuch new~, and ~notmuch restore~.  I checked my backups from
2020-11-09 (not affected) and 2011-11-16.  That latest backup was from
before I /noticed/ the corruption, but sadly it was affected too.  I
then diffed the latest backup with the previous; and then the latest
backup with the current dump.  The diffs suggest that the error involved
only the addition of invalid information; I suspect and hope that valid
information was not lost.

I attached my post-new Bash script and the Python 3 script it invokes.
So you can see the weird lines I mentioned, I provide for download (URL
below) the xz-compressed output of the command:

    diff -u notmuch_dump--manually_fixed notmuch_dump--corrupted > diff_notmuch_dump__manually_fixed--corrupted

https://upload.disroot.org/r/vSGNbFrN#8OCTbQTDKsrpRNjBsFzkv6rimPqEFE/UhGm14MypY0o=

I have also saved the binary corrupted database.  If you want to see it,
then tell me and I may upload it to Disroot's Lufi instance.  It should
probably be shown to as few people as possible for the sake of my
privacy.

Finally, my notmuch config includes the following directives (the other
directives are probably irrelevant to you):

    [new]
    tags=new
    ignore=

    [search]
    exclude_tags=deleted;spam;trash

    [maildir]
    synchronize_flags=true

Regards
post-new (application/x-shellscript)
simplemuch.py (text/x-python)
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

_______________________________________________
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-leave@notmuchmail.org

Thread: