Re: out of memory on idle machine

Subject: Re: out of memory on idle machine

Date: Sun, 31 Jan 2021 21:21:39 +0100

To: notmuch

Cc:

From: Gregor Zattler


Hi David, notmuch developers,
* Gregor Zattler <telegraph@gmx.net> [31. Jan. 2021]:
> I'll redo idea #2 on my laptop and will report it's results.

Now on the laptop:

grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210131T080546.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Processed 1184742 total files in 3h 33m 35s (92 files/sec.).
Added 1115147 new messages to the database.

0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
1612091679
0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%y"  ~/Mail/inbox/cur
2021-01-31 12:14:39.771049424 +0100
0 (master *) grfz@no:~/Mail/.notmuch$
0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
bash: quest: command not found
127 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:
1114128: [0]
inbox/cur

0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian
Value 0 for record #1114128: 1.61208e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur


So I think that's OK on my laptop, after the first notmuch
new.  Now I do another one.

0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Processed 121712 total files in 45m 4s (45 files/sec.).
Added 85345 new messages to the database. Removed 3 messages.
0 (master *) grfz@no:~/Mail/.notmuch$

The Problem remains, but at a different scale: In between
these two notmuch new runs there were only a few hours,
there's no way I received 121712 or 85345 emails in this time
frame.

0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612101986.719_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612105584.10821_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612109187.21153_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612112775.31326_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612116375.9385_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612120297.21133_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
terminate called after throwing an instance of 'Xapian::DatabaseCorruptError'
Aborted
134 (master *) grfz@no:~/Mail/.notmuch$

So here is the database corruption again, while doing the
third notmuch new.


I only now realize I did not do the tests after the second
notmuch new, but I copied the database so here is it:

0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
Value 0 for record #1114128: 1.61209e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
1612091679
0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian-2/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:
1114128: [0]
inbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
Value 0 for record #1114128: 1.61209e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$


I don't have a clue on how to interpret this.  Any ideas?

Thanks for your attention, Gregor


P.S.: These are the properties of the laptops root filesystem:

0 (master *) grfz@no:~$ sudo tune2fs -l /dev/mapper/vg-lv--root
[sudo] password for grfz:
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   root
Last mounted on:          /
Filesystem UUID:          801d2942-610d-4f43-9824-b275642e75de
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              14082048
Block count:              56308736
Reserved block count:     2815436
Free blocks:              23262312
Free inodes:              11957831
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stripe width:        8191
Flex block group size:    16
Filesystem created:       Sat Nov 23 11:59:12 2019
Last mount time:          Sun Jan 31 08:43:15 2021
Last write time:          Sun Jan 31 08:43:00 2021
Mount count:              1
Maximum mount count:      23
Last checked:             Sun Jan 31 08:43:00 2021
Check interval:           0 (<none>)
Lifetime writes:          7448 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       13902671
Default directory hash:   half_md4
Directory Hash Seed:      22422f4f-fec4-4a9b-8ee6-5be99ecb21f4
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x69629743
_______________________________________________
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-leave@notmuchmail.org

Thread: