Justifying Rough Sketches

The use of paper-based prototypes and rough sketches of interfaces is generally seen as a good way to get early user feedback on task and interaction designs. (This may have originally come from the architects you spoke of in last week's column.)

Pencil sketches and half-built mockups offer an "obviously unfinished" image to our candidate users. We believe that users so presented are less inhibited about giving useful critiques and constructive feedback than they would be if samples looked more polished.

However, there seem to be very few solid, experimentally-confirmed quantitative studies to support this hypothesis. A CHI'95 paper by Jutta Schumann is the only formal study (of 54 architects) that I have come across - most other authors in this area focus primarily on models and methodologies, rather than data.

But why is this important?

I'm concerned about the design of safety-critical systems such as health information systems and process control centers, where some of the interfaces are not necessarily "traditional" computer UIs. For such systems, the products of every step of the design process can be subject to strict quality control requirements, regulatory approval, and long-term project data archiving. In the event of an accident, all such material can become the subject of extensive investigation.

This means that everyone involved is strongly motivated to produce extremely accurate and highly faithful prototypes, user test materials, etc. But that’s exactly what we don't want! It's a real CYA problem, which should be addressed like any other step in the process--by pointing to the underlying experimental and technical reasons for the 'rough sketches' decision.

It strikes me that decent quantitative studies would very much help cost-justify the rough-sketch approach in selling usability in the corporate context. I hope we can do more in this area.

John Murray
Managing Engineer, Human Factors
Exponent Failure Analysis Associates, Menlo Park, CA
jxm@exponent.com

John has a rare attitude, doubtlessly because his company deals in failure—other people’s—so he tends to be perhaps overly sensitive about quality issues. Most software companies--yours and mine excepted, of course--seem to have figured out that software is good enough to release once the bug count gets down below, oh, perhaps 1000 or so.

This same issue of prototype fit-and-finish arises, however, regardless of the quality attitude of a company.

Jumping into complex, finely-tuned prototypes is perhaps the worst mistake a team can make. John points out the advantages of users (and clients) being freer to express contrary views if models look less than perfected. But there’s another side, too: designers/developers are more willing to listen to dissent if they haven’t lavished ultimate care on what should have been a storyboard or quick-and-dirty prototype.

Seymour Papert, the father of the Logo language for children (as well as particularly intelligent adults) once talked of being at a conference in Africa which instantly devolved into chaos, with many of the African participants walking out. When Seymour asked one of the Africans what had so troubled him, he replied, "You Americans say what you think."

It appears their culture was less adversarial than our own. People did not start out by taking strong positions, only to argue their way into eventual grudging compromise. Instead, they began by talking around the farthest edges of the problem, discussing issues on which they could agree. Only then did they begin spiraling inward, gaining unanimity on every point, until they arrived at a central, entirely agreeable whole. Any other approach was considered rude and insulting.

Having tried both methods in dealing with users and clients, I fully subscribe to the African approach. Make your users and clients a part of the earliest design process. Don’t expect them to hand you the answers, but do expect them to properly frame the question. Then share your earliest thoughts. At the worst, you will achieve buy-in. More often than not, you will all end up rolling up your sleeves and working together to achieve a solution none of you could have achieved on your own.

The only time I’ve ever had a problem with showing rough sketches was when the client was expecting a polished prototype. If you set your users’ and clients’ expectations early, they will accept the roughest of sketches. (Do a good sell job on the concept and they won’t just accept them, they’ll demand them.)

Even when moving on to the next stage, keep it simple. Build what appears to be a full cut across the entire system—essentially a false front—with no real depth. Then develop a few "deep dives" into critical or typical areas. Learn all you can from these few areas through user testing and client meetings. Then take that knowledge and apply it to the rest of your developing design.

I’ve tried it the other way, developing and "successfully" defending the full design. It doesn’t work. Just so you don’t have to repeat the experiment, I’ll let you in on how it turns out: Just before release, the client sits down in your office and explains why the entire project has to be changed in the next two weeks. No, I take that back. The client won’t be sitting in your office. She'll be sitting in your boss’ office.

John wants some research done on this whole area. Next year will be a whole new SIGCHI convention, and they are desperate for papers. Any takers?

And for those who would like to explore prototyping and testing issues in more depth, I recommend Jakob Nielsen’s excellent book, Usability Engineering.

Until next week, I remain,

tog