On Wed, 01 Feb 2012 14:37:53 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> On Wed, 01 Feb 2012 12:18:08 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> > 
> > There are at least these options here
> > 
> > 1) go through all ~100 places where test_expect_equal_file is used
> >    and fix the call order: quick look tells that the offending uses
> >    are in dump-restore, hooks, search-limiting and symbol-hiding.
> > 
> > 2) enforce "expected" filename has some format *and* fix all current
> >    uses of it. Add testbed_error () function which yells loudly ane exits...
> > 
> > 3) guess which is output and which is expected from args so that 
> >    machine helps tester here (for both diff output & copied files)a
> > 
> > 4) just copy compared files to some directory, those are named as
> >    basename of the original -- diff order still inconsistent.
> > 
> > 
> > I'd just go with option 1 and fix new *violations* when stumble upon one.
> > 
> 
> Option 1 does not solve the problem.  New violations would apper and
> need to be fixed again.  I am for option 2.

How is enforcing use of a particular filename better and more robust
than enforcing argument order?  You'll still have to force an arbitrary
heuristic.  And you'll still be vulnerable to people messing up the file
names (which actually seems easier to get wrong than messing up the
order).  And you'll have to have more code to parse the argument
strings.  And you'll still get inconsistent diffs.

If this is really a problem, I vote for 1.  In general, I am not in
favor of making the test suite more complicated than it needs to be.

jamie.