24
conversational data (Nardi et al. 2000, Isaacs et al. 2002, Halverson et al. 2003): (1) Work Talk: It
mainly includes turns focused on problem-solving, e.g., quick questions and answer pairs; (2)
Work Coordination: It includes turns that are directly tied with the work practice and are needed to
coordinate tasks; (3) Social Talk: It includes turns such as greetings, good-byes, discussion of the
weather, humor, and personal types of social talk; and (4) IDRAK Related: It includes turns that
involve questions and discussions related to the use of the tool. Figure 5 illustrates the application
of the coding criteria on a fragment of a recorded conversation, and Figure 6 shows how the
dialogues progressed throughout six experiments.
Thermal Eng.: it’s the avg temp thats a bit of a problem
Architect: So we just need to increase internal temp?
Structural Eng.: Project Manager, I don’t know what to do, but robustness is decreasing
at the moment
Architect: Structural Eng. – it was fine before – what changed to make it worse?
Structural Eng.: did you change any delta? position?
Architect: Watch my mouse!
Structural Eng.: where is your mouse?
Architect:: Turn on yours maybe that helps
Thermal Eng.: Any changes guys?
Architect: Thermal Eng. – I’ll try to make smoother
Figure 5 - Example of Content Coding on a Fragment of a Recorded Conversation
Work talk
IDRAK
related
Work
coordination
25
Figure 6 - Progression of the Content Categories throughout the Dialogues for Six Test-runs
The majority of the experiments (e.g., numbers 1, 3, 4, 5 and 6 in Figure 6) generated a usage
pattern manifesting a predominance of ‘work talk’ and ‘work coordination’, measured by total
number of words. This finding lessens hypothetical concerns that IDRAK would lend itself to
support a disproportional amount of time spent in ‘social talk’ at the expense of time spent on
work related activities. Further, the low response incidence for the ‘IDRAK related’ category
suggests that users encountered few technical problems, corroborating the positive rating in terms
of users’ satisfaction. One exception to this pattern was experiment (2) where a technical glitch
prevented the ‘architect’ from logging in for approximately the first 30 minutes. While the other
26
three players waited for the IT administrator to sort out the problem, they used IDRAK to build
social bonding, chatting about holiday destinations, leisure activities, and the weather.
We also noted that team (1) predominantly discussed work coordination issues (e.g., how do
we agree when to make design changes versus hold on to make calculations; what do you know
that I need to know and vice-versa) in the initial 30 minutes before engaging in work talk. In
contrast, teams (3), (4), and (6) iterated between discussing work-coordination and work-talk
issues throughout the experiment; and teams (2) and (5) hardly discussed any work coordination
issues. This finding corroborates theory about how users appropriate the same digital tools in
different ways unless a usage protocol is enforced (Orlikowski 2000). One participant noted:
“I made note of the information that everyone else provided, but I did not take a moment to
step back and think about how each person’s tasks could fit in the overall picture…I believe if
we had set a point where we had to freeze the design, the process wouldn’t have been so
chaotic.” (Structural engineer)
We conjecture that failure to agree upfront a protocol about using, and avoiding misuse of,
IDRAK can undermine its usability to efficiently support virtual teams.
Insights on Effectiveness
We evaluated the effectiveness of IDRAK by statistically comparing the outcomes of the 12
experiments (see sample in Figure 7) against data from 28 board experiments. We understood
from the outset that it would be unlikely that the teams could reach solutions that would ‘satisfy’
(Simon 1962) all design criteria (Bucciarelli 1999). This expectation was corroborated by the
unpublished results accomplished by the 28 groups of engineering graduates and MBA students
who spent 90 minutes playing the board exercise (Beckman and Agogino 2002).
27
Figure 7 – Sample of Six Delta design results using IDRAK (legend in Table 3)
A series of independent samples T-tests was conducted to assess the comparative performance of
‘virtual’ and co-located engineering design teams. Relevant data plots indicated that underlying
distributions conformed to assumptions of normality (Anderson et al. 2007). Test results are
summarized in Table 3.