On Sat, 28 Nov 2009 10:02:03 -0600, "Jeffrey C. Ollie" <jeff@ocjtech.us> wrote:
> This is the beginning of a test suite. It uses the Check[1] unit
> testing framework to handle the testing. There's one basic test of
> the SHA1 routines, obviously many more will need to be added.
Hi Jeffrey,
Thanks very much for this contribution. I'm always highly in favor of
improving our testing infrastructure. And a good test suite was on of
the first items I put on the notmuch/TODO list.
However, it's not clear to me that this kind of C-based unit testing
will be able to easily support much of what I want from testing for
notmuch. It seems that most of the bugs I encounter, or features that I
want to test, require a lot of state to be setup that would be really
painful to do in C code.
For example, a recent bug I hit was:
After moving an (old) existing file into the mail store,
"notmuch new" won't pick it up.
And I don't see an easy way to test something like that from a typical
"check" test. Meanwhile, I can't see a lot of use from the kinds of
things that *could* be checked from a check test. We could easily check
little things like that after calling notmuch_message_add_tag that
notmuch_message_get_tags has the same tag in its list. But if we have to
do higher-level testing anyway, (for cases like the bug above), then
it's easy to cover fine-grained stuff like this there as well.
For the recent rename-support work I did I wrote the attached
notmuch-test script. It does the kind of high-level state setup I
described above, (and actually contains a test for precisely the bug I
described above).
I haven't added this to the repository yet since it needs a bunch of
work before being usable. Here are some things it needs:
* Automated verification that tests are working, (currently I'm just
manually verifying that the results are as expected---obviously
that's not practical in the long term).
* Modularization so that each test can be maintained as a small,
independent snippet.
* The ability to run an individual test in isolation without running
the whole suite.
* A fix in notmuch to get rid of the stupid sleep calls that are
slowing down the current notmuch-test. (One approach would be to
stop using mtime altogether. That might slow things down
unacceptably in general use, but might be just fine for the test
suite where the mail store can generally be quite tiny. Another
approach would be to use a filesystem like ext4 with sub-second
mtime support).
Anyway, just wanted to share some thoughts and some preliminary code for
a notmuch test suite.
-Carl
#!/bin/sh
set -e
find_notmuch_binary ()
{
dir=$1
while [ -n "$dir" ]; do
bin=$dir/notmuch
if [ -x $bin ]; then
echo $bin
return
fi
dir=$(dirname $dir)
if [ "$dir" = "/" ]; then
break
fi
done
echo notmuch
}
# Generate a new message in the mail directory, with
# a unique message ID and subject.
#
# The filename of the message generated is available as
# $gen_msg_filename
gen_msg_cnt=0
gen_msg_filename=""
generate_message ()
{
gen_msg_cnt=$((gen_msg_cnt + 1))
gen_msg_name=msg-$(printf "%03d" $gen_msg_cnt)
if [ "$#" = "0" ]; then
gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
else
gen_msg_filename="${MAIL_DIR}/$1/$gen_msg_name"
mkdir -p $(dirname $gen_msg_filename)
fi
cat <<EOF >$gen_msg_filename
From: Notmuch Test Suite <test_suite@notmuchmail.org>
To: Notmuch Test Suite <test_suite@notmuchmail.org>
Message-Id: <msg-${gen_msg_cnt}@notmuch-test-suite>
Subject: Test message ${gen_msg_filename}
Date: Tue, 05 Jan 2010 15:43:57 -0800
This is just a test message at ${gen_msg_filename}
EOF
}
do_sleep ()
{
sleep 1
}
TEST_DIR=$(pwd)/test.$$
MAIL_DIR=${TEST_DIR}/mail
export NOTMUCH_CONFIG=${TEST_DIR}/notmuch-config
NOTMUCH=$(find_notmuch_binary $(pwd))
rm -rf ${TEST_DIR}
mkdir ${TEST_DIR}
cd ${TEST_DIR}
mkdir ${MAIL_DIR}
cat <<EOF > ${NOTMUCH_CONFIG}
[database]
path=${MAIL_DIR}
[user]
name=Notmuch Test Suite
primary_email=test_suite@notmuchmail.org
EOF
echo "### Testing \"notmuch new\" with no messages"
$NOTMUCH new
echo "### Testing \"notmuch new\" with 1 new message"
do_sleep
generate_message
$NOTMUCH new
echo "### Testing \"notmuch new\" with 2 new messages"
do_sleep
generate_message
generate_message
$NOTMUCH new
echo "### Testing \"notmuch new\" with no new messages (and a non-empty database)"
$NOTMUCH new
echo "### Testing \"notmuch new\" with two new directories (one mail)"
rm -rf ${MAIL_DIR}/* ${MAIL_DIR}/.notmuch
mkdir ${MAIL_DIR}/def
mkdir ${MAIL_DIR}/ghi
generate_message def
$NOTMUCH new
echo "### Testing \"notmuch new\" with two new directories (one mail)---opposite inode order"
rm -rf ${MAIL_DIR}/.notmuch
mv ${MAIL_DIR}/ghi ${MAIL_DIR}/abc
rm ${MAIL_DIR}/def/*
generate_message abc
$NOTMUCH new
echo "### Testing \"notmuch new\" with 1 old message moved into the mail store"
rm -rf ${MAIL_DIR}/* ${MAIL_DIR}/.notmuch
generate_message
tmp_msg_filename=tmp/$gen_msg_filename
mkdir -p $(dirname $tmp_msg_filename)
mv $gen_msg_filename $tmp_msg_filename
do_sleep
$NOTMUCH new > /dev/null
do_sleep
mv $tmp_msg_filename $gen_msg_filename
$NOTMUCH new
echo "### Testing \"notmuch new\" with 1 renamed message"
do_sleep
generate_message
$NOTMUCH new > /dev/null
do_sleep
mv $gen_msg_filename ${gen_msg_filename}-renamed
$NOTMUCH new
echo "### Testing \"notmuch new\" with 1 deleted message"
do_sleep
rm ${gen_msg_filename}-renamed
$NOTMUCH new
echo "### Testing \"notmuch new\" with a new directory with 3 messages"
do_sleep
generate_message dir
generate_message dir
generate_message dir
$NOTMUCH new
echo "### Testing \"notmuch new\" with a renamed directory of 3 messages"
do_sleep
mv ${MAIL_DIR}/dir ${MAIL_DIR}/dir-renamed
$NOTMUCH new
echo "### Testing \"notmuch new\" with a deleted directory of 3 messages"
do_sleep
rm -rf ${MAIL_DIR}/dir-renamed
$NOTMUCH new
echo "### Testing \"notmuch new\" with a new directory with 3 messages (tail of list)"
do_sleep
generate_message zzz
generate_message zzz
generate_message zzz
$NOTMUCH new
echo "### Testing \"notmuch new\" with a deleted directory of 3 messages (tail of list)"
do_sleep
rm -rf ${MAIL_DIR}/zzz
$NOTMUCH new
echo "### Testing \"notmuch new\" with a symlink to an external directory of 1 message"
rm -rf ${MAIL_DIR}/.notmuch
mv ${MAIL_DIR} ${TEST_DIR}/actual_maildir
mkdir ${MAIL_DIR}
ln -s ${TEST_DIR}/actual_maildir ${MAIL_DIR}/symlink
$NOTMUCH new
echo "### Testing \"notmuch new\" with a symlink to an external file"
do_sleep
generate_message
external_msg_filename=${TEST_DIR}/external/$(basename $gen_msg_filename)
mkdir -p $(dirname $external_msg_filename)
mv $gen_msg_filename $external_msg_filename
ln -s $external_msg_filename $gen_msg_filename
$NOTMUCH new
echo "### Testing \"notmuch new\" with a two-level directory with 3 files"
do_sleep
generate_message two/levels
generate_message two/levels
generate_message two/levels
$NOTMUCH new
echo "### Testing \"notmuch new\" with deletion of two-level directory (3 files)"
do_sleep
rm -rf ${MAIL_DIR}/two
$NOTMUCH new
cat <<EOF
Notmuch test suite complete.
Intermediate state can be examined in:
${TEST_DIR}
EOF