Re: Xapian commits unexpectedly slow

Subject: Re: Xapian commits unexpectedly slow

Date: Sun, 26 May 2024 09:12:57 -0300

To: Matthew Schauer, notmuch@notmuchmail.org

Cc:

From: David Bremner


Matthew Schauer <matthew.schauer@e10x.net> writes:

>
> Nifty!  Here are the results -- I assume you know how to interpret them
> better than I do:
>
> T00-new.sh: Testing notmuch new                         [0.4 large]
>                         Wall(s) Usr(s)  Sys(s)  Res(K)  In/Out(512B)
>   Initial notmuch new   1163.05 854.26  45.97   444304  2343120/13645200
>   notmuch new #2        2.23    0.02    0.03    9384    2144/8
>   notmuch new #3        0.01    0.01    0.00    9460    0/8
>   notmuch new #4        0.01    0.01    0.00    9428    0/8
>   notmuch new #5        0.01    0.00    0.00    9468    0/8
>   notmuch new #6        0.01    0.01    0.00    9692    0/8
>   new (52374 mv)        1351.01 537.75  235.45  959524  1027288/8531616
>   new (52374 mv back)   834.15  489.27  213.97  967040  184/4754016
>   new (52374 cp)        747.23  284.03  105.51  941992  0/4007120
>

Apologies, it looks like I never replied to this thread. Probably you
are not longer interested, but I can make a few observations, mainly
that there are a few relevant improvements in later notmuch.

1) This is about 3x slower than my current benchmark machine [1]. My
current machine is probably 4 years newer, so I would expect some
improvement in performance. 

2) I don't know if this is typical for spinning rust, but about about
25% of the time is (apparently) IO wait, since it it does not show up in
CPU time.  I do have access to a machine with both SSD and spinning
rust, but the latter is in some complicated RAID formation, so I don't
know how representative the results would be.

3) Some time after you reported these issues I implemented an
"autocommit" parameter, which should should help avoid large Xapian
large commits.

4) Your results show that notmuch new could be extra slow when dealing
with moving files on disk. This should be somewhat improved by changes
in notmuch 0.32 (I also see fairly dramatic impovements in
notmuch-reindex relative to notmuch new, but the underlying cause is
less clear).

[1] e.g. https://notmuchmail.org/perf-test-results/2024-05-26-minkowski/
_______________________________________________
notmuch mailing list -- notmuch@notmuchmail.org
To unsubscribe send an email to notmuch-leave@notmuchmail.org

Thread: