This is a fantastic video from Penny-Arcade about the virtues of testing concepts and designs early and often with users. The video is aimed at games designers but applies to any designer and translates particularly well to web designers.

It’s not often that we share videos on our blog, but this one is too good not to shout about it. It delivers the message we tell our clients, but in a much better and more enjoyable medium. If you don’t mind sitting through the annoying advert at the beginning, you’ll find the reward is absolutely worth it. Enjoy!

What is it?

Loop11 is an online, remote usability testing tool. In simple terms Loop 11 allows you to create an online survey that includes tasks for users to complete on a website of your choice. You have the ability to mix questions with tasks in any order you wish, and for each task you can set a goal for the user to achieve i.e. find a size 10, blue women’s jumper for under £30. You then have to set the URL that they will start on at the beginning of the task. There is a demo of how it works on the website which takes you through what your users will see when they complete a test.
 

A step by step process of setting up your test

 
Below are two examples showing how the test looks to the user when they are completing a task (Figure 1) or answering a question (Figure 2). On each task the objective is always displayed at the top along with the progression buttons. When asking a question, Loop11 gives you various question types which allow the users to answer in an assortment of different ways, the example I have shown represents a normal text field answer/comment box.
 

 Figure 1. An example of how a task would look like to a user 

Figure 2. You can insert questions after, or before the tasks, above is an example of a text field question

What are its advantages?

This is the closest tool we have found so far to an actual usability test. Loop11 is one of the only sites we have found that really lets you create something similar to the usability tests we regularly carry out. You have control over the design of a proper test script which is the main reason we have found it to be so useful.

You don’t have to be there with the user (in the same room). Loop11 tests are un-moderated so all you have to do is design it, then set it live and spread the word by sending a link to your selected participants. Users can then complete the tasks in their own home at their own pace. Its main advantage over face to face usability tests is that it allows you to test as many users as you want.

Loop11 isn’t free, but it is cheaper than moderated usability testing. You also don’t have to spend much money on incentives as users are participating when they want, and where they want (we’ve written more about this on UXbooth). We still included a small incentive as a thank you for users spending their time completing the test, and as we made sure the test itself wasn’t long it worked out rather well for everyone.

You can test Loop11 on any device that browses the internet. We haven’t tested this but we would assume that in most cases this would be correct as Loop11 uses the browser and is fully online so it shouldn’t matter too much what device you use it on.

View ready made reports on participants completing your test. After you have set the test live and participants start completing it you can view reports which show how many people succeeded in completing tasks, how long it took them, and how many pages they went through to get there. This information comes in 3 exportable formats (Excel XML, PDF, CSV, and Excel XML with participants) so that you have access to the original data to do with as you please. The PDF option also exports the report version which includes graphically presented results for the overall test, and for each task itself, however we found that the excel document of raw data was the most useful as it allowed us to work with the data to produce a report with the information we required. We could then brand it and use it within our projects.

A sample version of the PDF report that Loop11 exports after a test is finished

What are its limitations?

Once a test is published there is no going back. You can’t edit the test after publishing it; you can only delete it altogether, so you better get it right the first time! We would recommend thinking carefully about the order of your test before adding them to the test list. Re-test, and double check as many times as possible in preview mode before putting it live.

Judging the success or failure rate can be tricky. The site tests live websites so for each task you have to set the URL you want users to start from. In order to track success and fail tracking you need to add URLs which specify the pages users reach which you consider a to count as a success. This can be problematic if your task can be completed in a variety of different ways. If you don’t anticipate them all, you could record a failure even if they succeed.

Tasks need to be carefully designed. The design of each task becomes critical when doing an un-moderated test especially with this tool as it needs to be much less fluid than a typical face to face usability test where normally we would deliberately allow users the freedom to complete the task as they would naturally. With Loop11 you are forced to be more quantitative with your approach to get a more defined fail/success criterion. Therefore we found that the tool forced us to design tasks to be much more basic and definite.  For example, in a face to face test we might ask users to define the problem they typically face and then show us how they would solve that problem using the site. With Loop 11 we would design the task to answer a very specific question we know they can find an answer to on a specific page.

Slow loading times on some websites. Sometimes the website you may want to use will perform slowly which could affect how likely people are to complete the test. We also noticed it crashed on a few sites too. We recommend checking how well a site performs in the preview mode before you commit to purchase. Loop11 do provide a piece of JavaScript that you can insert into the body text of the site to enhance the speed. Unfortunately we couldn’t do this on the site we were using which is a drawback if you are testing a site without access to the backend.

You might get more people participating than you bargained for. One quite costly drawback, especially for those of us who incentivise participants, is that despite setting a limit for the number of tests it only cuts them off after that number have completed the form. However, it cannot then stop people who have already started the test whilst the 20^th (in this example) person was completing it. Therefore you may end up paying out for slightly more people than you originally wanted (we set a limit of 20 but had to pay 24 participants).

You don’t know who really completed the test properly. Probably the most obvious limitation when it comes to un-moderated testing is. Obviously you would like to think that anyone who sat down to do the test would want to do it properly, but you will get those who just rush through the tasks and questions to gain the incentive at the end. Normally you can look at the results and usually spot the person who you think didn’t complete it properly, the people who took the shortest time to complete their tasks and didn’t write any comments. The question you then have to ask yourself is whether you discount them from the testing?
 

When we recommend using it

If you’re looking for a quick, high level view of how a website is doing then this would be a good solution for you. It also helps if you are having problems testing users who are geographically spread or too time poor to meet one to one, by using Loop11 you can overcome this by testing anywhere that has internet accessibility anytime they want. Due to Loop11’s nature it is a useful tool for benchmarking a site, finding out where users are running into an issue to then carrying out some face to face usability testing to find out exactly where the problem lies. The scenario where we are likely to use it is to monitor the usability of sites we have already worked on. However, as the tool has strict success/fail metrics it is only really suited to carefully designed tasks which have a clear answer.
 
See anything I’ve missed? Just let me know!

I was talking to someone the other day about the new BBC homepage which employs a way to navigate through content from left to right. I thought it was well implemented, and knowing the BBC they would have user tested it thoroughly before making the call.

Anyway, the conversation swiftly moved on to horizontal scrolling and how as a ‘usability dude’ I must hate anything with a horizontal scroll bar. I tried to  explain that most of us ‘usability’ people are not against unconventional design, but we just like to see it implemented for the right reasons (because it fits with the user goal) and not for the wrong reasons (because the designer likes it and wants back slaps from peers).

Vertical scrolling, Yes. Horizontal scrolling, No.

In a talk, Jakob Nielsen once demonstrated his thoughts on horizontal scrolling by nodding his head up and down saying, "vertical scrolling, yes", then, shaking his head left and right saying, "horizontal scrolling, no".  A clever way to make a point, but digital design is never as simple as just following a rule or guideline from a so called ‘guru’. There are of course situations where a design works perfectly well going against conventions (which are typically outdated anyway). So we ask: When should you use horizontal scrolling?

Of course there’s no easy answer to that question. But when understanding the context of use and the goals users have when using the site it can become easier to decide whether to use horizontal scrolling or not.

When using horizontal scrolling can be beneficial to users

Although it’s not something we would always recommend, these examples may suit horizontal scrolling if you are keen on using it:

• Displaying a variety of visual images i.e. a photography site or design portfolio
• Displaying information in a large visual area that is not easy to see at a glance – i.e. think of a map or Google’s street map which employs horizontal scrolling to good effect
• Displaying discreet sections or slides of information – Tablets and smartphone apps employ the notion of swiping and when this is used to move from one screen from left to right it can work really well and feel completely natural. Similarly applications such as Slideshare work well in the horizontal plane (although it is arguable that this constitutes scrolling)
• Displaying a large catalogue of products or items where scrolling horizontally could display different product categories

Why you should be careful in using it

• Most mice have a vertical scrolling wheel, few have an easy way to scroll horizontally. This means most users have to manually operate the scroll mechanism. This is slowly changing with smart mice like the Apple Magic Mouse but may still take some time before they’re mainstream. In fact if you consider physical ergonomics, it’s much easier to move a finger up and down, than it is left to right. Thumbs and hands on the other hand are much more adept at a horizontal scrolling motion, so this type of navigation is likely to depend on innovations in gesture interfaces
• The experience of scrolling horizontally on some sites makes the screen judder and can have that headache inducing feel to the experience
• Controlling the speed of the scroll can be problematic, with some content whizzing past and others taking forever. Giving users the right amount of control can be difficult to get right
• We’re so used to reading left to right within the confines of a page where we make our way slowly downwards, introducing a horizontal scroll could break a fairly rigid western convention so should be used with care when reading is a core part of the user journey

Examples around the web

Interestingly Abercrombie, Hollister, and Superdry have made the decision to move away from horizontal scrolling to vertical. Shopstyle on the other hand have employed horizontal scrolling well on their site.

Like all new functionality, it should be thought through carefully and of course tested with users before taking the plunge.

Here’s some examples from around the web that we’ve found using horizontal scrolling in some way:

• http://www.bbc.co.uk/
• http://www.kiapicanto.co.za/
• http://www.cooper.com
• http://www.marianaonate.com/
• http://www.mareenfischinger.com/
• http://www.dennys.com/
• http://www.shopstyle.co.uk/

What do you think? Do you have any good examples to share?

Santa’s job gets more difficult every year. Less and less kids write to him making the elve’s lives more difficult to keep up with the latest toy trends, the population grows more each year making Santa work harder, and more and more houses are opting for wood burners or no chimneys at all.

This year Mrs Claus contacted us to make Santa’s life a little easier when he’s on deliveries. Of course we were more than happy to help in the hope we would make the Nice list this year.

Mrs Claus was worried about Santa

Mrs Claus was our client, but Santa was our user so we first had to establish the aims for the project and the client’s requirements. In our meetings with Mrs Claus it became clear why she wanted our help:

• Every year she’s up all night worrying about Santa and whether he’s safe – she wanted a way to keep track of his whereabouts without calling him all the time
• She’s worried about Santa’s growing waistline so she wanted a way to remind Santa not to eat so many mince pies on his rounds
• She was also worried that Santa should be careful not to take too many sips of sherry on his visits
• Her primary concern was that the route the elves draw up for Santa was getting more and more complex each year and she was worried Santa would get lost and miss deliveries

Santa’s requirements

Before coming up with a solution we wanted to talk with the user of anything we designed, so we had a chat with Santa to understand more about the context of use. It quickly became clear that he had a different list of requirements:

•  He wanted to keep his sightings to a minimum. With the growth of Facebook and Twitter in recent years, he was worried that he was becoming increasingly vulnerable to people being able to track him
• Rather than a route planner or sat nav, Santa would prefer a pre-defined chimney stops so he could go from chimney to chimney with the route already planned out
• Santa has been struggling to remember the sleeping places for the scary dogs, which houses had difficult roofs to land on, which chimneys were too narrow and so on. He wanted a simple way to receive all that information as he left one chimney on the way to the next
• Santa sometimes gets bored with listening to the Reindeer bickering so wanted a way to set up and manage his playlists
• Santa need a way to track his time and see how he was progressing with his delivery plan to make sure he remained on track
• Santa needed some clearly marked stops where he and the Reindeer could have a ‘comfort break’

In coming up with our solution we had to take into account the context of use:

• It would be cold so anything he used would have to be easily operated outdoors with gloves on – i.e. large buttons
• Santa would need a sleigh mounted device as well as a mobile device to update and consult whilst down a chimney

Wireframes for Santa’s interfaces

We took away all these requirements and wireframed a solution using a tablet device mounted to the sleigh console as well as a smart phone device which synched to it when Santa was on the ground. In phase 2 we will look at a separate monitoring interface for Mrs Claus and the elves to track Santa.

Santa’s Sleigh Mounted Tablet Interface

• Sightings alert  which monitors Facebook, Twitter and SMS chatter
• Next chimney stop with suggested landing places and up-to-date house intelligence
• Playlist controls
• Local time and delivery progress monitor

Sleigh mounted interface - click for fullscreen

Santa’s Smartphone Interface

• Checklist to tick off deliveries as he goes
• Ability to post updates to house intelligence including chimney dimensions, dog sleeping places…
• Mince pie and sherry sips update reminder – with an external breathalyser (we felt this was less priority so have planned this for phase 2)

Santa's smartphone interface - click for fullscreen view

Next steps – Prototype testing with Santa

We’re having to move fast on this project as you can imagine. We’ve only got a couple of weeks left! So now we’ve created the wireframes we need to test them in a prototype with Santa on a few test runs out in the sleigh with his gloves on. We’re looking at stitching finger and thumb pads into the tips of his gloves first. After some user tests we’ll refine our prototype and then start work on the visual design. We’ll keep you posted on how we get on. Wish us luck!

Tesco recently released details of a new feature to their online offering. This feature allows users to view a product in 3D, using augmented reality, as if they are holding it in their hands. Sounds great huh?

The objective here is to help online shoppers make a buying decision. The example Tesco gives is of a busy mum who is looking to buy a new TV, and needs to ensure it will fit well in the living room, and will have enough ports for her kids’ gaming machines.

However, there are a number of flaws with the process that result in a poor user experience:

• Firstly, there is a process:
• Accept Active X
• Download and install a plug-in
• Accept the licence
• Have a ‘marker’ to hand – or print one out
• Ensure I have a webcam, and it is on
• Hold up the marker to the webcam
• Finally, I can look at something. However, the 3d image is poor quality
• Holding it in the palm of your hand does not really allow you to envisage how it will look in your living room

So let’s look at some of these:

You have to download a plug in to your machine

Users are now familiar with shopping online, using websites without the need to download external software to run. By including a requirement to download something Tesco has added a step in the process that users are not expecting, and many are likely to be uncomfortable with.

You need to have a specific physical item in your hand

Now the user has downloaded the plug-in they now have to have a physical item in their hand. This can be something you already own, but if not you can print this out. This is another stage in the process that users are unfamiliar with. The perceived hassle for an unknown gain plays a big part in usability. Forms are normally the biggest culprit, but we imagine that this step will prove too much for most users.

The 3d image is poor quality

So now the user has gotten through the process they are now ready to view the product in the palm of their hand. Disappointingly, the quality of the product graphics is poor and doesn’t really do the product justice. The user can’t really imagine a TV sitting in the corner of the living room when it is sat in the palm of their hand. Ok, the user can look at the ports on the back of the TV, but they may be left wanting a quality image of the TV to validate the pixelated version they’re looking at.

So, what do users need?

When we’ve tested retail websites the product information pages are often key to the user making a buying decision. Imagery is often poor, leaving users guessing or unsure. At this point many users will leave the site and try elsewhere until they are satisfied enough to buy it (from a different website).

This new augmented reality feature from Tesco is clearly looking to solve this problem. But the user research we’ve done has shown that by simply providing images of the product from different angles, with an option to enlarge the image, works perfectly for users. 3D images with rotations and video are sometimes available too and with better image quality than the augmented reality solution.

I’m very interested to see how successful this is, but it looks to me that gimmick and technology has been prioritised ahead of usability and the user experience.  Is this trying to solve a problem that already has a perfectly acceptable solution?