Saturday, September 10, 2011

Release notes as test cases

I've been testing bug fixes and writing release notes for them for years now. A release note is the high level documentation detailing a problem that used to exist in the software and explains how it has been corrected in the latest version. These notes are compiled and sent to the customer along with the software so that they can see what changes have been made.

I like writing release notes, but I never realized how important they are, and how they can be used to help test. I had just finished testing a bug fix and was going through the motions of completing the story.  The release note is typically the final act.  I had to explain to the imaginary customer what we had just changed, and how they could verify it.

The way that I explained the problem and its correction in text caused me to stop and ask myself a question. This question led to another test. That test led to a potential problem. So there I was, ready to close the story that I felt was sufficiently tested and move onto the next one, when writing the release note sent me back to the drawing board.

Crap.

Well, not crap. I found a problem.  But could I have found it sooner? My tests passed according to the acceptance criteria. My destructive tests didn't turn out to be that destructive. I felt that I had covered my bases when I started writing the release note and by the time I was finished found that I was wrong. I'm never wrong.* Thinking like a user is different than thinking like a tester. If someone was to explain a correction for a bug to you, what questions would you ask?

I had my car in for service recently because the check engine light was on.  The service technicians explained that I needed a new fuel leak detection pump. Or something. Whatever. I signed the papers and paid the bill and my only question when the work was finished was 'Is the check engine light off?'.

The next time I test a bug I'm going to try writing the release note for it first. It's not just acceptance criteria, its the information that can and will be questioned by the customer.





*I'm usually wrong.