A Geek’s Lens

There are many times when repeating things is necessary and helpful. That is certainly the case with a strong QA/testing/feedback process in software development. The key is getting feedback early and then often. In my most recent project at Fog Creek, we took a product from concept to release in 6 weeks. It was a substantial amount of work. It adds some great new functionality, but my best guess is that it took us fifteen to twenty percent longer then it “should have.”

The problem was simply that our rock star QA analyst was away on vacation at [what we failed to recognize as] a pivot point in the development cycle. It happened just as we finished the first pass of Fog Creek Copilot OneClick’s functionality, before any of the chrome was applied. The people around to give us feedback had a strong technical understanding of the product and the problem that it was solving. As a result, we spent a full week without serious first-time user feedback.

It is a distinct advantage, that QA analysts aren’t programmers (usually). It’s awesome that FC attracted a few that put themselves in the shoes of lead users in a way that is very hard once you are deeply involved in the architecture of the software. Without their feedback, we made decisions about features and interaction design, as well as assumptions about the readiness of the software for delivery to market, that were wrong.

We could have solved this problem without Alison (QA analyst extraordinaire) if we had recognized it. Steve Krug has a great methodology for dealing with this exact issue, through quick and cheap usability testing, but we didn’t think it was dangerous to delay the collection of feedback just a few days. Its not that we had to undo all of the decisions that were made in that intervening week, but some made it harder to incorporate the necessary changes that came out of Ali’s testing. Even hallway usability testing failed us, to some extent, because we had explained too much about the product and the process to our colleagues. Without credible non-tech-centric feedback, our blind spots grew, but our awareness of them receded. More importantly, we lost momentum because we started to feel like it was done even though we were only really at the halfway point, in terms of time.

The day Ali returned from vacation and started filing bugs, our productivity went right back up, our sense of urgency returned, and our blind spots started shrinking. 2 weeks later, we were cleaning the dust out of the corners and putting the final touches on the product. It’s comforting to know that this is a problem that can be hedged by proactively improving our process. Next time around, we will be sure to have either QA or user feedback built-in to the “initial pass complete” phase of our development cycle.