18
Figure 4 - Instantiation of the Delta Design Window in IDRAK
Evaluating the Usability of IDRAK
Usability is synonymous with “the effectiveness, efficiency, and satisfaction with which specified
users can achieve goals in particular environments” (ISO 1998). Usability cannot be directly
measured, but can be operationalized into objective and subjective criteria (Folmer and Bosch
2004, Hornbæk 2006). The objective criteria capture aspects of the interaction that are not
dependant on users’ perceptions, including performance attributes such as efficiency and
learnability (Hornbæk 2006). Subjective criteria capture users’ perceptions of, or attitudes
towards, the interface, the interaction, and the outcome, including satisfaction and attractiveness
(Hornbæk 2006). We used various instruments to collect objective and subjective data about the
three dimensions of usability, including: analysis of logging data (chat-based dialogues, final
19
design concept); interviews with participants; retrospective reports about the experiments; and
data collected through a standard questionnaire that participants filled at the end of each
experiment (see summary in Table 1).
Table 1- Sources of Objective and Subjective Data about the Usability of IDRAK
Data sources
Usability
Attribute
Actual use logged in
Interviews, survey, reports
Satisfaction
(focuses on
user)
Does not apply
PSSUQ standard survey measuring:
system usefulness; information quality;
interface quality
Perceptions on satisfaction articulated in
interviews and reports
Efficiency
(focuses on
interaction)
Content coding of chat-based dialogues
in terms of:
work talk, work coordination, social
talk, IDRAK related
Perceptions (e.g. , on usage patterns,
frequency of use, access to information)
articulated in the interviews and
retrospective reports
Effectiveness
(focuses on
outcome)
Comparison of digital-based design
solutions against board-based solutions
Perceptions (e.g., ability to complete
tasks) articulated in the interviews and
retrospective reports
Satisfaction
We considered satisfaction in terms of “the freedom from discomfort, and positive attitudes
towards the use of a product” (ISO 1998). Metrics of satisfaction include: measures of average
satisfaction (Lewis 2002, Sauro and Kindlund 2005); perceived usefulness and ease-of-use (Davis
1989,
Hornbæk
2006,
Lewis
2002); preferences,
users’ attitudes, perception
of
outcomes/interaction, and satisfaction with the interface (Hornbæk 2006, Lewis 2002). Surveys
are typically used for measuring satisfaction (Folmer and Bosch 2004). Our survey instrument for
learning about users’ satisfaction with IDRAK was the PSSUQ standard questionnaire, which
assesses three dependent variables: system usefulness, information quality, and interface quality
(Lewis 2002). Each variable is an aggregate construct of an independent set of statements;
satisfaction, in turn, is an aggregate construct of the three dependent variables (ibid.) (See
Appendix II for the structure of the questionnaire and results). We triangulated the survey results
20
with qualitative data that we collected from carrying on debriefing sessions after the experiments,
and from retrospective reports written by the students who participated in the experiments.
4
Efficiency
We considered efficiency in terms of “the resources expended in relation to the accuracy and
completeness with which users achieve specified goals” (ISO 1998). Metrics of efficiency include
measures of time, input rate, mental effort, usage pattern, communication effort, and learning (e.g.,
Sauro and Kindlund 2005, Hornbæk 2006). We measured efficiency primarily by examining usage
patterns and the communication effort. Usage pattern manifests how users utilize a digital
interface to solve tasks, including the frequency in the use of the available functionalities and the
amount of information that users access or employ (Hornbæk 2006). Communication effort
measures the resources that users expend in communication, including ‘turn size’ (the number of
words included in each conversational turn within a chat environment), ‘number of turns’ (the
number of conversational turns included in each chat session), and frequency and duration of
conversations (Isaacs et al. 2002, Halverson et al. 2003,). We analyzed the content of the chat-
based dialogues using coding schemes developed to validate research carried out on Instant
Messaging and persistent chat (Nardi et al. 2000, Isaacs et al. 2002, Halverson et al. 2003, Haynes
et al. 2004). Further, we observed teams using IDRAK, performed debriefing sessions, and asked
students to discuss efficiency issues in retrospective reports.
Effectiveness
We considered effectiveness in terms of “the accuracy and completeness with which users achieve
specified goals” (ISO 1998). Metrics of effectiveness include measures of task completion,
4
We made this report count for the final mark, which ensured a 100 per cent turn out of written feedback