recent posts
categories
archives
great blogs
tools
The effect of bias in DIY usability testing
We’re seeing a lot of people talking about conducting their own usability tests in house. The topic of DIY testing has been around for some time now with advocates such as Steve Krug and Jakob Nielson. The general principle being that any testing is better than no testing which is an idea we fully believe in.
As a company who offer usability testing it will probably come as no surprise that we would put forward an argument against companies doing their testing in house. But, before I do that I’d like to add that I think its excellent that people are taking user centredness so seriously these days and if teams feel that they want to take testing in house then that can only be a good thing for the future of the industry. If internal development teams feel so strongly about getting usability and user experience right, then of course we would fully support them in doing so.
My concern with what appears to be a rise in DIY usability testing is one of bias. One of the main benefits our clients get from using us is that we’re independent (combined with the fact we’ve been usability testing for nearly 10 years now).
Our word of warning to all those people looking to test their own projects is don’t underestimate the power of bias when testing your own stuff. When we come into a project and observe users interacting with a product or service we don’t have any of the baggage of internal politics, why it has been designed the way it has, who made the decision to put that button there, why its not possible to do this. We simply see things from a fresh, independent perspective which allows us to really see what’s going on.
When a project team comes to observe us testing their product or service with users, at the end of the day when we talk about the findings what they saw is often quite different to what we saw. They saw the detail rather than the bigger picture, they picked up on evidence to strengthen their own pre-existing beliefs, they played out discussions and arguments they’d had when designing it. We didn’t. We had the luxury of seeing what was really happening.
Not being able to see the wood from the trees is something we can all identify with at times. When we are in the day to day detail its really hard to step back and see things from a new perspective. Testing the project you’ve been working on carriers the danger that you may just see what you want to see. You may see things that you instantly dismiss because of the history of how it has been developed, but the real key to improving the experience may be hidden here and you simply can’t see it. Sometimes it takes someone else to spot the patterns going on right in front of your eyes.
Overall the increase in DIY testing has to be a good one because ultimately the winners will be the users. An increase in awareness and appreciation for improving user experience is something we would fully support, so for people with no budget to bring in independent consultants we’d fully recommend giving DIY testing a go. But, DIY testers must be aware of the dangers of only seeing evidence to support what they already believe to be true. Sometimes an independent expert review can be more valuable than an DIY usability test, but that’s a post for another day.
Is your perception of usability in your product or service accurate?
Related services: Usability testing, and User experience audit
March 26th, 2009 at 4:12 pm
I think the argument for DIY testing is appropriate when you look at the individual project, the size, requirements, budget, and the project lifecycle. DIY testing is paramount in getting a project moving, and to build a picture of the user before applying standard methodologies to developing the product. But you're quite right that the project team will naturally become blinded by their own good work to not see critical user issues. So in addition, external user testing is important when the designs or the product are at a point that needs external user validation (final concept designs, later in the prototyping phase, etc.) before moving on to the next phase.