out of memory on idle machine (was: Re: consistent database corruption with notmuch new)

Subject: out of memory on idle machine (was: Re: consistent database corruption with notmuch new)

Date: Sat, 30 Jan 2021 09:54:32 +0100

To: notmuch

Cc:

From: Gregor Zattler

Hi notmuch developers,,
* Gregor Zattler <telegraph@gmx.net> [14. Dez. 2020]:
> notmuch new still corrupts the database, the second notmuch new
> invocation finds emails the first did not find.

I'm still searching for the reason notmuch chokes on my mails.

I assembled a HP MicroServer, installed basic debian buster and
notmuch from the debian buster repo, rsynced my mail to a
separate file system symlinked to the same location as on my
laptop.

There are now
grfz@mic:~/Mail$ find -type f | wc -l
1209419
files on this file system.  no other process touches this
file system, actually the machine is otherwise ilde.

I did notmuch new several times in a row:

grfz@mic:~/Mail/.notmuch$ rm -rf xapian
grfz@mic:~/Mail/.notmuch$ notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210127T114210.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 1183682 total files in 16h 43m 27s (19 files/sec.).
Added 1091038 new messages to the database.
grfz@mic:~/Mail/.notmuch$ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Processed 1169095 total files in 16h 52m 48s (19 files/sec.).
Added 1077686 new messages to the database.
grfz@mic:~/Mail/.notmuch$ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Processed 1151900 total files in 16h 59m 36s (18 files/sec.).
Added 1050106 new messages to the database.
grfz@mic:~/Mail/.notmuch$ notmuch new
add_file: Out of memory files/sec.).
Processed 205 total files in 6s (29 files/sec.).
Added 193 new messages to the database.
Note: A fatal error was encountered: Out of memory
grfz@mic:~/Mail/.notmuch$


notmuch new processes not all files, later invocations
re-add most of the files and the fourth notmuch new
invocation runs out of memory after a few seconds.

This machine has

grfz@mic:~/Mail$ free
total        used        free      shared  buff/cache   available
Mem:       16394760      222272     2293404        5464    13879084    15831448
Swap:      15622140       15104    15607036

enough memory.  It's ECC Memory.


Any ideas on what is the cause of this?  Even if we ignore the
out-of-memory condition: Why does notmuch re-add a million
already added files?


Ciao; Gregor














































































+ my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
+ my-fetchmail.services stop
/home/grfz/bin/my-notmuch-renew: line 4: my-fetchmail.services: command not found
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-05-29.1559166684
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1519322404.860_1.len:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1525795204.16507_1.len:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1514242825.4116_1.len:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_405.pit:2,
Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_525.pit:2,
Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_685.pit:2,
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190529T215124.gz...
Your notmuch database has now been upgraded.
Processed 1036956 total files in 2h 21m 25s (122 files/sec.).
Added 945879 new messages to the database.
+ notmuch tag --batch -new -inbox -unread '*'
Can't specify both cmdline and stdin!
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634
Warning: cannot apply tags to missing message: notmuch-sha1-e584281f598522d172e73617da020e2f914ccb7e
Warning: cannot apply tags to missing message: notmuch-sha1-5b56ca3568715b006eddd35a597ca7654be045f9
Warning: cannot apply tags to missing message: notmuch-sha1-00ab3ac6e631287f0f6dffaf0037bb7ef86f9e47
Warning: cannot apply tags to missing message: notmuch-sha1-72bde4c427b04c1cbac4a2c2b3a94402616917d7
Warning: cannot apply tags to missing message: notmuch-sha1-dbd41b3c6736943ef17eff9c6862dfa3703af020
Warning: cannot apply tags to missing message: notmuch-sha1-f17b642c650ba54b67b9d4ff5a338143a73f8b7b
Warning: cannot apply tags to missing message: notmuch-sha1-e1cf1219c1fb10a55ba30ca590b93eaf33fdae27
Warning: cannot apply tags to missing message: notmuch-sha1-0d5aaa420dc9e9505cced7c5d8f4c15e0c88c5ef
Warning: cannot apply tags to missing message: notmuch-sha1-a31480a6ef2520153b778dc9531f2c982a79ba32
Warning: cannot apply tags to missing message: notmuch-sha1-606e94dbe6e46b9df83d77ffa788429523e39646
Warning: cannot apply tags to missing message: notmuch-sha1-bb586b264c94f23d9e826071d8841d542ca95c75
Warning: cannot apply tags to missing message: notmuch-sha1-06d4d1492c70aa544b1deeaf7895ca85fe634670
Warning: cannot apply tags to missing message: notmuch-sha1-253991559a5e3c2d5cf1234b761e52ea09aa693f
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559177807
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-05-30.1559177848
Compacting database...
compacting table postlist
     Reduced by 36% 908856K (2488312K -> 1579456K)
compacting table docdata
     Reduced by 49% 312K (632K -> 320K)
compacting table termlist
     Reduced by 47% 1701792K (3551384K -> 1849592K)
compacting table position
     Reduced by 45% 3375368K (7372720K -> 3997352K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559177848.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559214505
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch new
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
+ my-fetchmail.services stop
+ rm -rf /home/grfz/Mail/.notmuch/xapian
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch config set index.header.To To
+ notmuch config set index.header.Cc Cc
+ notmuch config set index.header.Sender Sender
+ notmuch config set index.header.InReplyTo In-Reply-To
+ notmuch config set index.header.ReplyTo Reply-To
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190530T135332.gz...
Your notmuch database has now been upgraded.
Processed 1037014 total files in 2h 42m 32s (106 files/sec.).
Added 945937 new messages to the database.
+ notmuch tag -new -inbox -unread '*'
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559241909
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-05-30.1559241950
Compacting database...
compacting table postlist
     Reduced by 36% 1001496K (2707600K -> 1706104K)
compacting table docdata
     Reduced by 51% 336K (656K -> 320K)
compacting table termlist
     Reduced by 47% 1837496K (3836392K -> 1998896K)
compacting table position
     Reduced by 45% 3875240K (8440616K -> 4565376K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559241950.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
++ date
+ echo Start @ Sa 1. Jun 16:50:23 CEST 2019
Start @ Sa 1. Jun 16:50:23 CEST 2019
+ /home/grfz/bin/my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-06-01.1559400744
+ rm -rf /home/grfz/Mail/.notmuch/xapian
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch config set index.header.To To
+ notmuch config set index.header.Cc Cc
+ notmuch config set index.header.Sender Sender
+ notmuch config set index.header.InReplyTo In-Reply-To
+ notmuch config set index.header.ReplyTo Reply-To
+ notmuch config set index.header.Received Received
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190601T145224.gz...
Your notmuch database has now been upgraded.
Processed 1037381 total files in 2h 54m 22s (99 files/sec.).
Added 946288 new messages to the database.
+ notmuch tag -inbox -unread '*'
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559419342
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-06-01.1559419387
Compacting database...
compacting table postlist
     Reduced by 37% 1077536K (2904400K -> 1826864K)
compacting table docdata
     Reduced by 48% 304K (624K -> 320K)
compacting table termlist
     Reduced by 48% 2004432K (4166496K -> 2162064K)
compacting table position
     Reduced by 46% 4498968K (9748816K -> 5249848K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-06-01.1559419387.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
+ my-fetchmail.services start
/home/grfz/bin/my-notmuch-renew: line 48: my-fetchmail.services: command not found
++ date
+ echo End @ Sa 1. Jun 22:10:51 CEST 2019
End @ Sa 1. Jun 22:10:51 CEST 2019
++ date
+ echo Start @ So 2. Jun 00:45:52 CEST 2019
Start @ So 2. Jun 00:45:52 CEST 2019
+ /home/grfz/bin/my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-02.1559429235
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-02.1559429235
++ date
+ echo Start @ Di 4. Jun 20:54:03 CEST 2019
Start @ Di 4. Jun 20:54:03 CEST 2019
+ /home/grfz/bin/my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-06-04.1559674574
+ rm -rf /home/grfz/Mail/.notmuch/xapian
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch config set index.header.To To
+ notmuch config set index.header.Cc Cc
+ notmuch config set index.header.Sender Sender
+ notmuch config set index.header.InReplyTo In-Reply-To
+ notmuch config set index.header.ReplyTo Reply-To
+ notmuch config set index.header.Received Received
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190604T185615.gz...
Your notmuch database has now been upgraded.
Processed 1034784 total files in 3h 16m 21s (87 files/sec.).
Added 944525 new messages to the database.
+ notmuch tag -inbox -unread '*'
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-05.1559694905
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-06-05.1559694949
Compacting database...
compacting table postlist
     Reduced by 37% 1073696K (2900248K -> 1826552K)
compacting table docdata
     Reduced by 50% 328K (648K -> 320K)
compacting table termlist
     Reduced by 48% 2003640K (4164728K -> 2161088K)
compacting table position
     Reduced by 46% 4495744K (9743384K -> 5247640K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-06-05.1559694949.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
+ my-fetchmail.services start
++ date
+ echo End @ Mi 5. Jun 02:44:58 CEST 2019
End @ Mi 5. Jun 02:44:58 CEST 2019
From: Gregor Zattler <telegraph@gmx.net>
To: notmuch <notmuch@notmuchmail.org>
Cc:
Bcc: Gregor Zattler <grfz@localhost>
Subject: out of memory on "empty" machine (was: Re: consistent database
 corruption with notmuch new)
Reply-To:
In-Reply-To: <20201214192251.GA7858@no.workgroup>

Hi notmuch,
* Gregor Zattler <telegraph@gmx.net> [14. Dez. 2020]:
> Hi notmuch developers,
> * Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
>> * Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
>>> I now do a notmuch new with libxapian30 version 1.4.17-1
>>> and will report back in a few hours.
>>
>> The result is only slightly different from version 1.4.11:
>
> actually now I realized, that notmuch was not linked against
> libxapian v1.4.17-1.
>
> Now I build notmuch from master and linked it with
> libxapian 1.4.17-1.  The result is almost the same, though:
> 0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan
> Welcome to a new version of notmuch! Your database will now be upgraded.
> This process is safe to interrupt.
> Backing up tags to /home/grfz/Mail/.notmuch/dump-20201214T124836.gz...
> Your notmuch database has now been upgraded.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
> Processed 1176599 total files in 3h 31m 23s (92 files/sec.).
> Added 1102787 new messages to the database.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
> Processed 125008 total files in 44m 59s (46 files/sec.).
> Added 83898 new messages to the database.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> add_file: A Xapian exception occurred).
> A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
> Processed 122480 total files in 42m 1s (48 files/sec.).
> Added 83907 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> 1 (master *) grfz@no:~/Mail/.notmuch$
>
> notmuch new still corrupts the database, the second notmuch new
> invocation finds emails the first did not find.
>
>
> Ciao, Gregor
> --
>  -... --- .-. . -.. ..--.. ...-.-
>
>

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-
_______________________________________________
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-leave@notmuchmail.org

Thread: