If you're following the red-green-refactor cycle properly, then you'll have seen test failures on DRYed tests. Most testing frameworks let you customize the failure message. It's usually a simple matter of adding more information about which part of it failed.

I won't meta-program for tests, but I will do things like make a list of classes, or symbols or whatever, to pass to a loop. Just keep it simple.

> I won't meta-program for tests, but I will do things like make a list of classes, or symbols or whatever, to pass to a loop. Just keep it simple.

This is a case where metaprogramming for tests came in handy.

I had a bug recently that involved someone making a change that violated an invariant property of a class. To codify this invariant, I was tempted to do what you did, to make a list of symbols to feed into my test to ensure the invariant was obeyed. However this bug was caused precisely by someone adding a new symbol, a method, that didn't obey this property. The test using this design wouldn't catch this failure. I instead opted to do some introspection (it's Python, so it was dead simple) on the class to ensure all of its methods obeyed this invariant. It took a little extra time to implement but in the end it worked.

With the caveat that I haven't invested the full amount of time necessary to strongly state this opinion publicly, I do feel that metaprogrammed tests are a case of adding abstraction unnecessarily (which, incidentally, is why I haven't taken the time to try to extensively use them on a project.)

A test should be dead simple to read and understand when someone else new to the project needs to understand what it tests. Further, when a test fails, the output should clearly indicate exactly on what line of code an error occurred.

Metaprogramming tests feels to me like a case of a desire for or predilection for cleverness getting in the way of what the task is actually for.

Tests should tell a story. They should not be subjected to the same methods of abstraction used in the code they themselves are supposed to test and verify.

>Meta-programming to generate tests has only lead to pain in my experience.

Likewise, although I view it as a deficiency of testing frameworks that require metaprogramming for parameterized testing. They shouldn't.

And on the other hand, I'm always suspicious of code that does something programming-like that is specific to the test framework.

If you need a loop, ideally you write the loop using the language, rather than using a test framework's special loop construct. For parameterization, ideally you'd use a function, rather than a special test framework concept for parameterization.

Having duals of every abstraction feature of a language in the test framework increases the cognitive load and ramp up time to become effective in a codebase, and there's always the risk of incomplete knowledge, leading to mixing and matching different abstractions - and test framework abstractions are usually more leaky and less well thought out than language abstractions.

Furthermore, they decrease readability and clarity of tests themselves. I am of the strong opinion that if you need to test 20 inputs, you should write 20 asserts, not a loop - when the test fails, you should be able to immediately tell _which_ input(s) failed, not just that a failure has occurred.

I came to this opinion by way of desiring to be clever in my tests and then finding myself having to insert `printf(input)` inside a loop in order to quickly figure out which input failed.

Code should be written so that it doesn't need to be modified. Feelings of aesthetics of desires to be clever should be avoided and are, ultimately, I think, often just vanity.

I'm thinking of rspec in particular, as it's the framework I write most tests in. It's trivial to include the loop variable in the string interpolation that describes the "context" block for the loop inner test setup. But usually I wouldn't be using a plain loop, I'd be iterating over entries from a hash map.

The same thing in something like junit is less workable, and copy and paste becomes more viable simply because the language is so clumsy and inexpressive. I've seen various extensions and annotations that let you parameterize junit tests but I don't think they make things much clearer.

A key benefit of driving the test cases off a table of inputs to outputs is that missing cases in cross products are much easier to spot. Visualising the state space and its coverage is easier when you can see a map of the covered space in one block of code.

The one place where I've had good results from meta programming / DRY for tests is table-based validation; where you have a large cross-product of different states to test, mapping to expected outcomes.