21
number of errors, accuracy, completeness, and quality of outcome (Sauro and Kindlund 2005,
Hornbæk 2006). We used task completion and quality of outcome as the two primary criteria to
measure the effectiveness of IDRAK. Accordingly, we collected the outcomes of the experiments,
evaluated their quality against the design requirements, and statistically compared those results
against unpublished results achieved by teams playing the board-based exercise.
Analysis of the Experiment Results
We performed twelve experiments in the computer laboratory over six month’s elapsed time. Four
experiments involved students enrolled in a graduate-standing programme in operations
management; seven experiments involved MBA students enrolled in a project management
course; and one experiment involved practitioners from ARUP, a world-leading engineering
design firm. The language of the exercise was English, but participants came from at least seven
different countries. Each team received an instruction manual about IDRAK (El-Tayeh 2007) one
week in advance. Right before the start of each experiment, participants were reminded that they
could save in IDRAK’s Repository – and retrieve from – chat-based dialogues and other
observations. Further, participants were instructed to spend the last 15 minutes of each session
answering the on-line questionnaire and voluntary contributing to IDRAK’s Repository. Hence,
later teams had the option – in theory – of searching know-how documented by earlier teams.
Insights on Satisfaction
The 7-point Likert scale questionnaire yielded positive ratings for the three dependent variables
and a positive ‘Overall Satisfaction’ rating (3.06 ± 0.93) (Appendix II). We cross-checked the
questionnaire results with qualitative data from interviews, debriefing sessions, and introspective
reports (see summary in Table 2). One user observed on the extent she was satisfied with the
22
system: “In the beginning it requires some attention but as you go on, it [using IDRAK] gets better
and better (…) I especially liked the ability to scroll up and down in the dialogue box because I
could easily check something that someone had said 15 or 20 minutes before.” The positive rating
in terms of interface quality suggests that IDRAK was easy to use, especially considering that
users only received a 15-min introduction before the experiment. One user remarked about the
social proxy: “In a couple of cases I found it useful. I was reading the brief and periodically
glanced at the proxy to quickly check whether the team was still engaged in conversation.”
Another user described how IDRAK helped their team do the work: “At the beginning I thought
that the tool would not enable us to do the job as we were struggling to reach decisions, but as we
started to talk more and engage in discussions I was amazed how it enabled us to achieve the task
at hand.” Users also highlighted that chat-based communication helped them overcome the
difficulties in understanding colleagues with strong English accents.
Table 2 – Data exemplars on (dis)satisfaction with IDRAK
Satisfaction
Dimension
Data Exemplar
Leveling of
status
“The chat-based communication is a great leveling mechanism,
yielding equal status to all team members …a player who would
usually be quiet and shy in a physical confrontation can be open and
expressive” (thermal engineer)
Ease of
communicatio
n
“When there are members of the team whose first language is not
English, it is easier to communicate with the written word as we do
not have to contend with fast speech or accents that we have not
heard before” (architect)
Accountability “Keeping track of communications increases accountability because
nobody can deny what they have stated during the process” (project
manager)
Pros
Traceability
“The use of written messages brought some benefits in terms of
clarity. It was possible to examine the messages’ history and follow
the threads to recall information when needed” (structural engineer)
Lack of
authority
“In some instances where exercise of authority may have benefited
the team, it was difficult for one to do so; we need to consider ways
of giving control back to the project manager” (project manager)
Cons
Information
overload
“Communications were overloaded from the beginning and I had to
keep an eye on the message board in order not to miss important
information; there is a considerable amount of clutter and identifying
the meaningful requires attention” (structural engineer)
23
Excessive
Visibility
“There was not a clear distinction between one-to-one and one-to-
many messages. Communication would have been easier with
software that allowed multiple conversations in different private
rooms” (project manager)
Loss
of
nuances
“Understanding between individuals was a little harder to gauge as
communication nuances such as tone and expression are lost in this
medium” (architect)
We received some suggestions to improve IDRAK from a user satisfaction perspective. First,
users suggested that IDRAK should help them to target specific people, whether for a
conversation or to attract a colleague’s attention (currently, IDRAK only allows users to ring the
whole group at the same time). Second, some users suggested complementing the current social
proxy with audio awareness cues; as one user put it: “It is sometimes easy to miss people talking
while you are working on other things; it would be helpful for messages arriving on the dialogue
to be accompanied by some kind of noise.” Literature in CSCW recommends, however, prudence
in providing audio-awareness cues because they can be annoying (Isaacs et al. 2002). Third, some
users expressed interest in using audio alongside the chat-based medium to enhance
communication: “It will be easier if members can talk to each other [and] not only type stuff [in].”
One ARUP practitioner observed, however, that voice/video communication is not always viable
if teams work from opposite time zones (e.g., Australia and UK have a maximum of 2 hours to
engage in real time collaboration). Fourth, various ‘project managers’ noted that, while they
appreciated IDRAK’s ability to level the status of participants, they wished they could have a
mechanism to help them better assert authority.
Insights on Efficiency
To learn about the efficiency of IDRAK, we recorded and analyzed a total of 1326 conversational
turns with a total of 12,925 words, which averages 9.74 words per turn. The brevity of the
conversations is in line with previous research on the length of informal interactions in the
workplace (Isaacs et al. 2002, Halverson et al. 2003). We used the following categories to code the
Dostları ilə paylaş: |