Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

Subject: Corrupted database (notmuch 0.31, Emacs 27.1.50, afew 3.0.1, OfflineIMAP, Python)

Date: Tue, 17 Nov 2020 16:17:12 -0300

To: notmuch@notmuchmail.org

Cc:

From: Jorge P. de Morais Neto

Hi.  I use Notmuch 0.31.2 on Emacs 27.1.50 (manually compiled on
2020-11-02) with matching version-pinned MELPA Stable Notmuch package on
updated Debian buster.  I have enabled buster-proposed-updates,
buster-updates and buster-backports.  I manually backport notmuch
according to <https://wiki.debian.org/SimpleBackportCreation>.  I use
OfflineIMAP 7.3.3 (Python 2 pip), afew 3.0.1 (pip3), Bogofilter 1.2.4
(buster) and a custom Python 3 script based on the ~notmuch~ module.

Yesterday (when still on Notmuch 0.31) I noticed that, when I tagged a
message or thread, the fido-mode completion offered many weird candidate
tags that shouldn't exist in the database.  Also, on the Notmuch Hello
screen the ~All tags~ section would error out.  I then dumped the
database (~notmuch dump~) and noticed many lines associating weird tags
to weird message ids.

In almost every case, both the weird tags and the weird Message-Id
contained uncommon characters, often ASCII control characters.  One of
the weird lines was " -- id:8"---specifying a message with Messaged-ID
"8" and no tags.  I tried ~notmuch show id:8~ and got an internal
error---something like "message with document ID <SOME_NUMBER> has no
thread ID".

I then upgraded Notmuch to 0.31.2 and compacted the database but the
error persisted.  I then manually cleaned up the database dump, deleted
the ~/offlineimap/Jorge-Disroot/.notmuch/xapian/ directory, invoked
~notmuch new~, and ~notmuch restore~.  I checked my backups from
2020-11-09 (no corruption) and 2011-11-16.  That latest backup was from
before I /noticed/ the corruption, but it was affected too.  I then
diffed backup 2020-11-09 with backup 2020-11-16; and then backup
2020-11-16 with the current dump.  The diffs suggest that the error
involved only the addition of invalid information; I suspect and hope
that valid information was not lost.

I attached my post-new Bash script and the Python 3 script it invokes.
So you can see the weird lines I mentioned, I also attached the
xz-compressed output of the command:

    diff -u notmuch_dump--manually_fixed notmuch_dump--corrupted > diff_notmuch_dump__manually_fixed--corrupted

I have also saved the binary corrupted database.  If you want to see it,
then tell me and I may upload it to Disroot's Lufi instance.  It should
probably be shown to as few people as possible for the sake of my
privacy.

Finally, my notmuch config includes the following directives (the other
directives are probably irrelevant to you):

    [new]
    tags=new
    ignore=

    [search]
    exclude_tags=deleted;spam;trash

    [maildir]
    synchronize_flags=true

Regards
diff_notmuch_dump__manually_fixed--corrupted.xz (application/x-xz)post-new (application/x-shellscript)
simplemuch.py (text/x-python)
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

_______________________________________________
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-leave@notmuchmail.org

Thread: