Lucas Richter

How to write unit tests that work

Mentor: Hey kid! How do you tell when your automated test is broken?

Protege: When it fails, of course! How stupid do you think I am?

Mentor: I think you’re pretty damn stupid!

Protege: WTF?!??! Why?

Mentor: Because you only got half of the answer. A test is broken if it behaves in either of 2 ways. The first one is obvious: if it fails when it should pass. The second is less obvious, but equally valid: a test is broken if it passes when it should fail!

Protege: Hold up a minute, old guy. So you’re saying that my tests could suck, even though they pass?

Mentor: Yep.

This was the realisation I came to a short while ago. And it’s a bugger. My tests could not only suck, they could suck invisibly. So, I thought, what’s the solution?

Back when I was first reading about Test-Driven Development (TDD), I thought, “OK. So I have to write my test so it fails, then rewrite it so it passes, then refactor it. Easy.”

Not so easy, in fact. Because you don’t have to do that just at the test level. You have to do it for every single assertion you make. Otherwise you don’t know whether you have a broken assertion.

Protege: But that’s a pain in the arse! It’s so inefficient! It will take me ages!

Mentor: That ain’t necessarily so. You just need to figure out a way to do it so that it doesn’t take a lot of effort.

Protege: Come on, sensei. I know you have answers. Give it up!

Unfortunately for me, I don’t have the kind of mentor that just gives me answers. So here’s what I came up with (by the way, I’m using a Test-After Development (TAD) methodology):

1. Write the code.
2. Write your test, but set it up to fail at every opportunity.
   That is, where something should be true, assert that it’s false. If two data are meant to match, assert that they don’t. If it should be an “equal or greater than” (>=) comparison, make it a “less than” (<). This is how you make sure your test will fail when it’s supposed to.
3. Put a breakpoint on each “inverted” assertion.
   This not only serves as a convenient marker (at least if you have an IDE), it means you don’t have to set one when your test screws up (i.e. when your “inverted” assertion passes instead of failing) and you have to debug to see what the problem is.
4. Run your test, and make sure it fails at the first “inverted” assertion.
   That’s important. If it skips any, then something’s wrong.
   
5. Once you’re satisfied, “un-invert” the assertion.
   Or invert it again, if you like. This will make it pass when it should.
   
6. Remove the breakpoint. You’re ready to move on to the next assertion.
7. Repeat steps 4-6 until all your assertions are “the right way up.”
8. Enjoy your solid, reliable test.

DISCLAIMER

I’m not an authority on unit testing, or any aspect of coding. Far from it. I’m just a guy. I’m writing down what I figure out, so that I don’t forget it. If this is useful to you, that’s fantastic. Please let me know so I can enjoy the warm fuzzy feeling (seriously). If it’s crap, please let me know, so I can be liberated from my ignorance (equally seriously).

Finally, some word on Twitter outages

I just read an article about Twitter’s downtime problems on their blog. And here’s what I think:

Finally.

I’m really pleased that they’re talking about this. It gives me far more confidence that this is, if not a user friendly service (due to the aforementioned downtime problems), then at least a business with user friendly attitudes.

I’m as pissed as anyone else about not being able to use Twitter when I want to. That will remain the case as long as there’s downtime. But while I’m no more confident that the problem will be solved than I was before I read that post, I’m far more certain that I want to see Twitter succeed. And not just the “make a profit, get rich” kind of succeed. I mean the “make Google jealous” kind of succeed.

And that’s what acknowledging obvious problems can do for a business.

What I’d really love to see from here is the adoption of this attitude on their outage page. The current “we’re working on it, it’ll be back soon” message really doesn’t cut it with me. I’d rather just be told that the error’s been logged, to be honest. As long as there’s some follow-up about what the error was, so I can subscribe to a maintenance blog or whatever if I want. Unlikely as that is to happen, it would be an exemplary demonstration of transparency and honesty with users.

Please, Twitter. Don’t stop here. More of the same. Please.