As Jim points out there is a huge difference between "focus groups" and
usability testing. Focus groups typically involve groups of people
responding to topics put forth by a skilled moderator. They are good at
deriving marketing messages, product themes, user scenarios and needs,
etc. We used "focus groups" to figure out the "hot features" in a
product cycle, naming of features, and the "value" of a feature (how
much they're willing to pay if at all). In planning for a next release,
we used "contextual inquiry" and "focus groups" to understand the top
user scenarios our products needed to address. Contextual inquiry is
where you go to a user site and watch them work. A lot of good and bad
features in Office 95 were "discovered" this way. Focus groups helped
us figure out what people liked and hated with the current release.
Usability testing is really an art. As Jim points out it's very hard to
find talent usability staff. It's also very hard to get semi-arrogant
product designers to watch people use the features they designed. I've
spent probably 20+ hours watching people use mail merge when I
redesigned it in 92-94. Usability testing is an invaluable tool when
used correctly. When used incorrectly it can merely validate bad ideas
and features. The difference is in the skill of the usability manager
and the script used to watch the user through the tests.
I would argue that Microsoft's most valuable usability lesson was
Microsoft Bob. I watched a couple of the Bob usability tests and they
went great for novice users. They were able to quickly accomplish a set
of predefined tasks. What Microsoft learned with Bob is that you need
to do longitudinal usability studies (bring the users back after a
while). Novice users quickly outgrew the help Bob provided and soon
found it annoying. Microsoft character is another example of how
usability crutches over time become annoying as the user gets more
familiar with the software.
So, I'm a big fan of usability. As I watched users play with mail merge
I found myself being totally depressed and cheering all in one session.
Usability can be as simple as bringing users in and watch them use the
product over their shoulder to dedicated, sophisticated usability labs
with one way windows. The later is slightly more effective because
users don't feel subconsicous about having someone peering over their
shoulder.
-----Original Message-----
From: Jim Whitehead [mailto:ejw@cse.ucsc.edu]
Sent: Monday, April 23, 2001 9:47 AM
To: FoRK@XeNT.CoM
Subject: RE: [NYT] Microsoft Relies Again on Inner Circle
Actually, it is my understanding that Microsoft has had a usability
group
since the early 90's. This group does stick actual users in front of MS
Software and paper prototypes to gather UI feedback.
One problem with this approach is that it's expensive and time
consuming,
and requires people with a fairly unusual background (combination of CS
and
Cog Sci. knowledge) to perform the work. Thus, it's hard to staff such
groups.
Note that there is a difference between "focus groups", which are more
of a
marketing technique, and usability testing, which is typically only one
or
two people interacting with the software specifically to analyze how
well
they can use it. It assumes the marketing work of determining whether
the
feature belongs at all has already been done.
- Jim
> Adam Rifkin wrote:
> ...
> > Not long ago, the company videotaped ordinary users as they
> tried to set up
> > their new computer systems. Mr. Allchin watched the videos, and
recalled
> > being alarmed and embarrassed by some of the confusing
> situations in which
> > nontechnical owners of PC's found themselves.
> >
> > "I wanted to crawl underneath my desk," he said.
>
> Just as I suspected: It's obvious MS hasn't done any, or at least
enough
> focus groups. Gee, they finally decided to try it.
>
> AOL has always lived on focus groups.
>
This archive was generated by hypermail 2b29 : Sun Apr 29 2001 - 20:26:06 PDT