Article Text

Download PDFPDF

How to turn a team of experts into an expert medical team: guidance from the aviation and military communities
  1. C S Burke,
  2. E Salas,
  3. K Wilson-Donnelly,
  4. H Priest
  1. Department of Psychology, and Institute for Simulation & Training, University of Central Florida, Orlando, Florida, USA
  1. Correspondence to:
 C Shawn Burke PhD
 Institute for Simulation and Training, University of Central Florida, 3280 Progress Drive, Orlando, FL 32826 USA; sburkeist.ucf.edu

Abstract

There is no question that interdisciplinary teams are becoming ubiquitous in healthcare. It is also true that experts do not necessarily combine to make an expert team. However when teams work well they can serve as adaptive systems that allow organisations to mitigate errors within complex domains, thereby increasing safety. The medical community has begun to recognise the importance of teams and as such has begun to implement team training interventions. Over the past 20 years the military and aviation communities have made a large investment in understanding teams and their requisite training requirements. There are many lessons that can be learned from these communities to accelerate the impact of team training within the medical community. Therefore, the purpose of the current paper is to begin to translate some of the lessons learned from the military and aviation communities into practical guidance that can be used by the medical community.

  • AO, advance organisers
  • CRM, crew resource management
  • KSA, knowledge, skills and attitudes
  • SA, situation awareness
  • teams
  • team training
  • team development
  • medical teams
  • AO, advance organisers
  • CRM, crew resource management
  • KSA, knowledge, skills and attitudes
  • SA, situation awareness
  • teams
  • team training
  • team development
  • medical teams

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

On a weekend in a large tertiary care hospital the anaesthesiology team was called to perform anaesthetics for several clinical conditions deemed emergencies. The anaesthesiology team was already overloaded. To make matters more complicated a demanding nurse was insisting that her case be done first, surgeons were complaining that their cases were increasing in urgency, and within the primary operating suites the staff was only able to run two operating rooms simultaneously. The anaesthesiologist in charge was under pressure to attempt to overlap portions of the procedures by starting one case as another was finishing.1

The above example illustrates not only the dynamic and complex nature of many medical environments, but that the need for teamwork is ubiquitous within this complex environment. The medical personnel described in the above vignette operated primarily as separate individuals rather than a team whose goals were to ensure patient safety and well being. Instead of competing against one another the nurse, surgeons, anaesthesiologists, and operating room staff could have worked together in a coordinated fashion to make decisions regarding patient prioritisation and the best manner in which to deal with a stressful situation given the current constraints. Given the complexity, criticality of conditions, and lack of teamwork the propensity for errors was high in the example cited above. The cost of any error in teamwork could easily have been life threatening.

Recognising this, as well as the fact that effective teamwork is not automatic, the medical community has begun to implement team training interventions. The most popular of these interventions have been adaptations of crew resource management (CRM) training. While being commended for recognising the importance of teams in error prevention, a recent review of CRM, as applied in the medical community (for example, crisis resource management), revealed that such training has not yet proven to be effective (Wilson-Donnelly et al. Does CRM training work? An update, extension, and some critical needs (under review)). Results indicated that, in general, trainees reacted favourably to the training, but the transfer of the learned behaviours to the job or simulated environment was not consistent. In cases where transfer was observed, it was based on a particular simulated scenario2 or the experience of trainees.3

The review cited above indicates that although the medical community is making strides in recognising the importance of team training there is still a long road ahead. In this vein, the military and aviation communities have invested a great deal of time, money, and effort over the last few decades to better understand team functioning and training. As a result, there are many lessons that can be extracted and applied to those beginning to implement medical team training with the purpose of increasing the likelihood of effective team training interventions. Therefore, the purpose of this paper is to translate a subset of the lessons learned from the military and aviation communities into practical guidelines. We hope these can serve as guidance to those responsible for designing, implementing, and evaluating medical team training.

HOW CAN THE SCIENCE OF TRAINING HELP?

Training can be defined as the systematic acquisition of knowledge (what we think), skills (what we do), and attitudes (what we feel) (KSAs) that lead to improved performance in a particular environment.4 The medical community has typically focused on training task work skills, that is, the technical aspects of the job, but as the use of interdisciplinary health care teams increases, training task work skills will no longer be sufficient. There is currently over two decades of research evidence coming primarily out of the military and aviation communities which suggests that while task work skills provide the foundation for effective team performance they are a necessary, but not sufficient condition … teamwork skills are also needed.6–9 In this work, task work skills are those skills that members must understand and acquire for individual task performance, while teamwork skills are the cognitive, behavioural, and attitudinal actions that members need to function effectively as part of an interdependent team.10

Key messages

  • A team of experts does not make an expert team.5

  • There is much known about the design, implementation, and evaluation of team training that the medical community can leverage.

  • The creation and maintenance of expert teams takes planning and preparation—you must invest.

  • Factors before, during, and after training may impact the effectiveness of team training.

  • Expert teams have been trained in both task work and teamwork skills.

  • Tools exist to assist in the creation, implementation, and evaluation of team training.

  • Teams are complex and there is still much to learn.

Team training provides trainees with the necessary competencies at both the individual and team levels to complete their assigned tasks safely and effectively.11 At the individual level, team members must possess the KSAs needed to perform their specific task roles within the framework of the team. These individual competencies are incorporated into an interdependent, coordinated unit at the team level. In an effort to aid those responsible for developing team training, Salas and colleagues12,13 reviewed over 20 years of research and devised a heuristic framework to help conceptualise the basic components of team training (figure 1). This heuristic depicts team training as a set of interventions that are designed and developed with four main elements: a set of tools, delivery methods, instructional strategies, and content.

Figure 1

 Anatomy of team training.13

Tools—for example, team task analysis,14 feedback strategies, task simulation (Salas et al. Improving patient safety through simulation based training: what does it take? (unpublished work))—are used to diagnose, assess, and remediate team performance before, during, and after training. Based on the information gathered through the tools, team training objectives are created and the KSAs needed to complete each objective are identified. This information drives the type of methods used to deliver the training. The three most common methods used to present material to trainees are information based (for example, lecture), demonstration based (for example, behavioural modelling), and guided practice (for example, role play, hands on practice, simulation). These methods come together to create a team training strategy. It should be noted that while there are several team training strategies the medical community has adopted only one—CRM. Table 1 illustrates additional proven strategies.12,15

Table 1

 Team instructional strategies

While figure 1 depicts how tools, methods, and content combine to yield team based instructional strategies, it does not provide direct practical guidance on how to design, implement, and evaluate team training. The next section offers some practical guidance.

PRACTICAL GUIDANCE

It has recently been suggested that it is not enough to use a systematic approach to design training, attention also needs to be paid to factors outside the training that may impact its effectiveness.26,27 With this in mind, the practical guidance that follows is organised around three themes: (a) setting the stage/pre-team training factors, (b) design/implementation, and (c) post team training factors. The guidelines presented here are those where we have evidence that applying such guidance will lead to a more effective team training programme (table 2).

Table 2

 Guidelines for effective team training

Setting the stage

Guideline 1: the design and development of team training must be guided by the results of a team task analysis

The foundation of effective team training programmes lies within the a priori determination of the KSAs needed for effective performance on the job in question.28 One of the most common and theoretically grounded methods for determining the requisite KSAs is task analysis. While a variety of methods have been developed for analysing team tasks most do not typically capture the requisite cues, conditions, and standards that provide the basis for team tasks.28,29 For example, researchers have found that indices of importance and task criticality used for individual tasks do not translate well for team tasks.30 In light of this need, several researchers have been working over the past decade to develop and refine a procedure known as team task analysis.13,30–32 This procedure not only allows researchers and practitioners to identify the operational skills—that is, task work—needed within team tasks, but also identifies the skills needed for smooth coordination among team members—that is, teamwork. Finally, this procedure provides a mechanism by which training objectives can be linked to the KSAs needed to complete the team tasks in question.

The implementation of a team based task analysis is important, for without one, organisations run the risk of only training one of the two skill sets that empirical research has identified as being required for effective team performance. This state of affairs is evident within the medical community as well as several other communities. For example, a recent review of “team” training programmes within the medical community found that such programmes often teach task work skills without acknowledging teamwork skills (Wilson-Donnelly et al, Does CRM training work? An update, extension, and some critical needs (under review)).

Despite the argued importance of using team based task analysis there remains a lack of prescriptive integrated guidance pertaining to its implementation. In an effort to remedy this state of affairs, researchers14 extracted a series of steps practitioners can use in conducting a team task analysis (table 3).

Table 3

 Guidance for conducting team task analysis

As team task analysis is a general technique whose foundation lies within a long history of individual task analysis procedures, it can be applied to a variety of domains and can be readily implemented with minimal modification to the medical community.

Guideline 2: when training time is scarce, pre-practice tools can be used to maximise the impact of training

Recently, it has been argued that a number of factors prior to the actual training and practice session(s) can impact the utility of practice opportunities during training.15 As training time is often limited, training developers should not neglect the impact that tools, such as: attentional advice, preparatory information, advance organisers (for example, outlines, diagrams, graphic organisers), and pre-practice briefs may have on maximising practice opportunities. Each of the above pre-practice tools have been argued to maximise practice time, allowing trainees to learn more in shorter amounts of time, by aiding trainees in identifying critical aspects of training. Attentional advice directs trainee attention towards particular factors in the practice environment thereby improving the utility of practice.34,35 For example, in teaching medical teams how to assess and perform initial treatment on trauma alert cases attentional advice may be given prior to using a simulation in terms of those aspects that are the most difficult or important.

Preparatory information serves a similar function in that it can be used prior to practice to set trainee expectations about the events likely to occur and their consequences. The use of information of this type has effectively been used within the medical community to reduce anxiety and reactions to stress and is now extending to other domains.23,24 Although the exact content contained within preparatory information may differ according to domain, it tends to focus on an explanation of the physiological reactions that are likely to occur regarding the stressful event. Preparatory information typically addresses how the person is likely to feel, describes the events that are likely to occur, and how to mitigate these undesirable consequences.23

Advance organisers (AO) are another pre-practice tool that helps trainees develop a basic structure—that is, their expectations—for the information that is to be provided during training. This, in turn, aids in the internalisation of information learned, easing the integration of new knowledge with existing knowledge. Advanced organisers come in many forms and researchers have offered guidance in terms of the steps required to develop these tools. Steps include:

  1. inform trainees of advance organiser purpose,

  2. identify topics of tasks,

  3. provide organising framework,

  4. clarify action to be taken,

  5. provide background information,

  6. state concepts to be learned,

  7. clarify concepts,

  8. introduce vocabulary, and

  9. state the general outcome/goal desired.36

Advanced organisers within the medical community have been shown to be effective37 and may come in the form of course outlines or graphical/pictorial diagrams handed out prior to training which illustrate key components to be included in training. In keeping with the earlier trauma example, an AO might be a graphical representation related to the cascading symptoms which indicate such a call.

In addition to assisting individual performance in training these AOs have also been argued to facilitate team training in that they provide a foundation for the shared knowledge that drives teamwork and the requisite coordination requirements. For example, pre-practice briefs can be used prior to training to clarify team performance expectations and set team member roles and responsibilities prior to practice. In turn, this serves to develop shared knowledge and increase coordination. Such briefs are commonly used within the aviation, military, and medical communities, but often not in a training environment.38,39

Guideline 3: ensure latent organisational messages about the importance of team training match those that are spoken

Organisations spend up to $100 billion on training and development, but less than 10% of these expenditures actually result in a positive training experience where skills are transferred back on the job.40 Research has shown that characteristics of the work environment such as supervisor and peer support and opportunities to practice skills back on the job impact the effectiveness of training.41–43 Messages sent by the organisation and supervisor regarding the importance of training have an impact. For example, research indicates that leaders can significantly influence the effectiveness of training (that is, the likelihood of transfer back to job) simply through informal reinforcement of trained behaviours.32 Research has shown that teams containing a leader that was perceived as being supportive exhibited 42–52% more behaviours consistent with the trained skill than those with non-supportive leaders.43

In addition, how the organisation frames the training experience in terms of attendance policies, where training is placed (within the broader curriculum), and how behaviour learned in training is promoted back on the job all speak to the organisation’s view of training’s importance. For example, evidence indicates that when training is framed as remedial it reduces trainee motivation and learning; while framing it as advanced contributes positively to motivation and learning.44,45 In addition, attendance policies (that is, voluntary v mandatory) are also believed to influence the outcomes of training such that the outcome is better when attendance is framed as voluntary.46

Design/implementation

Guideline 4: team training must emphasise key teamwork components

The past 20 years has witnessed many advances in what is known about teamwork.47 As a result many models have been developed that not only illustrate the dynamic and multi-dimensional nature of teamwork, but also show the importance of process variables in determining team performance.47,48 Recently researchers49 reviewed the team literature and argued that at its simplest teamwork consists of a set of five competencies—adaptability, leadership, back up behaviour, mutual performance monitoring, and team orientation (table 4)—which are bound together through communication processes that result in shared mental models pertaining to the task, team, and situation. While the above constructs provide a generalisable set of teamwork competencies which have clearly shown their importance through empirical work (Wilson-Donnelly et al, Does CRM training work? An update, extension, and some critical needs (under review)),10,15 there may be other competencies that are additionally relevant dependent on the domain. Note that while the competencies cited below serve as an initial starting point, a team task analysis (see guideline 1) should be conducted to ensure the competencies most relevant for the job/task being trained are identified.

Table 4

 The “big five” of teamwork38

Guideline 5: design team training to facilitate shared understanding among team members

When operating within complex and dynamic domains (for example, the medical arena), stressors such as time pressure, fatigue, and workload tend to narrow attention increasing the possibility of errors. Due to cognitive limitations it is not possible for one person to process all the relevant information. Through the promotion of shared cognition in which all members are continually monitoring, assessing, and communicating key environmental cues, performance is facilitated.

Three factors which contribute to this state are: shared mental models, situational awareness (SA), and metacogniton. Shared mental models describe the degree to which long term memory structures held by team members regarding aspects of the task and team are aligned such that substantial agreement exists.55 This shared knowledge serves to guide coordinated action and allows members to quickly determine when something is “out of place”. For example, in deciding whether a trauma alert call should be placed to the hospital team, members of the emergency medical services first response team whom have a shared conceptualisation of each other’s roles can maximise the timeliness and quality of the call through better coordination and decision making. When team members have this shared knowledge it also provides them with the foundation and ability to offload the work of members whom are overloaded. Recently a wealth of empirical evidence is beginning to emerge to support the theoretical arguments concerning the importance of shared mental models to team performance (Burke CS. Examination of the cognitive mechanisms through which team leaders promote effective team processes and adaptive team performance. Unpublished doctoral dissertation. Virginia, USA: George Mason University, 1999).16,56–58 Furthermore many of the training strategies shown in table 1 have been shown to facilitate the development of shared mental models as can many of the pre-practice tools identified earlier.

Situation awareness refers to the “perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status into the near future” (Endsley, p. 36).59 It’s the result of members scanning of the environment and the perceiving of cues and patterns in a dynamic context. Information gained is then communicated and integrated into existing knowledge structures serving to update members’ shared mental models.60 Empirical work, conducted primarily with “real teams” within the military and aviation communities, has consistently found that effective teams hold higher levels of SA than low performing teams.61–64 Within our trauma alert example, team members would need to be aware of key components of the situation which might impact the interpretation of patient symptoms. Within a team setting this information/assessment needs to be shared so everyone is operating from a similar picture in making the decision as to call an alert or not.

In term of developing SA the most common tack that has been taken is to design systems such that SA is facilitated. However, recently researchers have begun to develop and implement training programmes which assist in the development and maintenance of SA. Research on pilots has indicated several target areas which may improve pilot SA, these are: task management, development of comprehension, projection and planning, information seeking, and self checking activities.65 Working with this same population, researchers have begun to define basic training approaches, including: higher order cognitive skills training, intensive pre-flight briefings, schema and mental model development, and structured feedback.66 Cue recognition training is another method which has successfully been used to facilitate the development of SA.60 It is expected that while these approaches would apply to the medical community they might need to be modified due to the structure of teams within medicine as well as the level of distraction.

Simultaneous to this assessment process is the requirement to regulate one’s own monitoring and interpretation of input by engaging in metacognitive processes. Metacognitive processes allow the learners to gain awareness of the current state of knowledge and the effectiveness of their learning strategies. Within the trauma case metacognitive activities might serve to alert a member performing a complex procedure that he/she is doing something slightly off, perhaps due to the stress, and allow that person to revise the procedure without being corrected by a fellow team member. Metacognitive processes also allow active mental models to be modified such that they provide a context sensitive assessment of the current situation. Within the team, this assessment needs to be shared not only to promote a common awareness of the situation, but it also mitigates the impact of an individual’s attention narrowing. While evidence has clearly shown the importance of metacognitive processes to individual performance its application to teams is a fairly recent development with their being few explicit strategies used at the team level to train these processes.

Guideline 6: ensure team training facilitates adaptive behaviours

The propensity to be adaptive is one of the hallmarks of an effective team and the factor which serves to allow teams to perform as more than the sum of their parts. Within the health care community there are a preponderance of redundant systems in terms of machines, however fellow team members can also be thought of as redundant systems. Teamwork encourages resilience to errors in that members may serve as adaptive systems by providing back up and monitoring behaviours. Empirical evidence has shown they can reallocate resources and recombine strategies as they perform.15 Back up behaviours and monitoring provide teams with an effective means to manage errors that occur by avoiding, trapping, or mitigating the consequences of errors.67 In order to manage errors within teams one of three things must happen: (a) team members ask for help when overloaded, (b) team members monitor each others performance to notice any performance decreases (mutual performance monitoring), or (c) team members take an active role in assisting other team members who are in need of help (backup behaviour). An essential component to the above actions happening is trust among team members.

While breakdowns in monitoring and backup behaviour have been implicated in several accidents involving aviation,68 these behaviours have also been implicated in near misses within the medical community. For example, a recent report indicated that while many operating room teams are required to conduct a “count” procedure of the tools that have been used during surgery, items such as the tissue retrieval bag are not included in this count. Within the above report, a nurse indicated that recently a tissue retrieval bag was almost left in a patient’s wound, but luckily a team member noticed the bag before the procedure was completed. This example illustrates monitoring in that someone noticed the bag was left in and backup behaviour in that the person spoke up so that it was taken out prior to closing.

In addition to the above behaviours shared mental models serve as the foundation for a team’s ability to be adaptive in the sense that these cognitive structures contain the information needed for team members to be able to identify when things are “out of place” as well as how members may be supported. Therefore, one way to facilitate adaptive team behaviours is to ensure the possession of shared mental models. In addition to this there are other design features which can be built into team training to facilitate the team’s adaptive capacity. For example, based on work conducted on transfer of training, it is known that training which presents multiple examples of varying complexity and situations serves to broaden and deepen trainee cognitive structures. Varying the nature of examples and practice opportunities serves to provide trainees with a broader response repertoire that can be pulled from, thereby increasing the potential for adaptability. This type of intervention should be fairly easy for any community to work into training in the design phase, especially if training is simulation based (see below).

Guideline 7: team training must promote attitudes and behaviours that are indicative of a learning climate

Within environments where errors are relatively infrequent, but high cost, team members must treat every potential opportunity as a learning event. In order to promote this type of “learning climate” team members must not only be willing to speak up, but must also trust that what they say or do will not be held against them. While actions such as mutual performance monitoring can be used as feedback for learning, it does no good if team members cross check and monitor one another’s actions if when something out of the ordinary is recognised no one speaks up due to fear of reprimand. The ability to speak up in a non-threatening and respectful manner (deference to expertise) is a hallmark of learning organisations and the teams within them.

Assertiveness training is one mechanism that can foster the propensity for junior team members to speak up when a concern exists. Assertiveness training involves teaching individuals to clearly and directly communicate their concerns, ideas, feelings, and needs to others.21 Assertiveness is trained not only so that junior members feel comfortable offering their perceptions to higher status members, but also so that communication is delivered in a manner that does not demean others or infringe upon their rights. In turn, this allows teams to take full advantage of the potential synergy available within the team by deferring to expertise in any given situation, regardless of rank. A recent review of CRM within both commercial and military environments has shown that promoting assertiveness has been a challenge for these communities (Wilson-Donnelly KA, Salas E, Burke CS. Crew resource management training in the military and beyond: a review and lessons learned. (Unpublished working document)). Given the similarities to aviation, in terms of power differences within teams, we would predict this to be a potential roadblock to effective team performance within the medical community. The reader is referred to the references in table 1 for more information on how to train this skill.

Guideline 8: ensure team members apply closed loop communication

Researchers69 have argued that communication breakdowns are the second most frequently cited cause of teamwork failures and accidents. In order to combat this the military has taught their teams and crews a particular type of communication strategy—closed looped communication.6,70 It is expected that this communication strategy will be especially important for teamwork within the medical community as many of the teams are interdisciplinary. While interdisciplinary (for example, nurse, surgeon, anaesthesiologist) teams are often necessary in complex environments the propensity for miscommunication increases in these teams as each discipline has their own “jargon”. Closed loop communication is one strategy that has been successfully implemented within the military to aid in the above situation. Closed looped communication is built upon a strategy of verification that ensures that the message sent was received and interpreted as intended. Specifically, closed loop communication involves (1) the sender initiating a message, (2) the receiver receiving the message, interpreting it, and acknowledging its receipt, and (3) the sender following up to insure the intended message was received.6 An example of the use of this strategy within the trauma call might be witnessed on the actual call from the field to the hospital. The field unit may report that they have a trauma patient with injury of type x and vitals of type y. The person receiving the message on the other end of the transmission would acknowledge the transmission and then repeat or read back the key information that should have been extracted. Finally, the field unit responds with an acknowledgment, unless there is something missing from transmission.

Closed loop communication is often trained or promoted through the use of standardised terminology and procedural type communication. This is combined with an agreement or norm that when possible communication transmission will follow the three step sequence laid out above.

Guideline 9: design team training to create systematic opportunities for practice of requisite team competencies

Everyone has heard the adage, “practice makes perfect”.71 While it is known that practice is needed to acquire new skills, it is often falsely believed that task exposure alone will lead to learning new behaviours. Task exposure, while beneficial, does not equal learning. Practice needs to be guided and requires measurement and feedback.72 By guiding practice and ensuring that participants practice targeted behaviours and attitudes not only is training time maximised, but trainees develop appropriate mental representations of the task73 (as opposed to inaccurate knowledge structures that are difficult to correct once formed).

Within the aviation and military communities there are primarily two types of guided practice that have been used successfully and would be expected to transfer to the medical community: a priori scripted scenarios and role plays. A priori defined scenarios have provided a foundation for much of the training work done within the military.74,75 In using a scenario based approach to practice, subject matter experts help in creating scenarios that contain embedded cues/triggers. These cues, based on training objectives, represent structured opportunities for team members to practice the competencies targeted in training. Using this method of practice not only ensures that opportunities to practice targeted behaviours are present, but also eases observation as observers have an idea of when key events should occur (for more detail see Salas E, et al. Improving patient safety through simulation-based training: what does it take? (unpublished)).

Role plays are another method which has been used to provide guided practice within the aviation community.76 While varying in the degree to which they are structured, role plays generally proceed as follows. Participants are given a description of a scenario and their role in it. In most cases they are told to assume they are in the scripted scenario and should respond to both the situation and the actions of other players. Upon completion guided discussion revolving around key learning points occurs. Role plays are a common technique used across a wide variety of training environments and have proven to be successful provided they are structured according to learning objectives and include guided discussion. Within the medical community the use of standardised patients and objective structured clinical exams for medical students is an example of a role play which could be structured according to learning objectives (for example, teamwork skills, clinical diagnosis).

Post training guidance

Guideline 10: evaluate team training and collect a variety of evidence as its impact

Training evaluation has been defined as “the systematic collection of descriptive and judgmental information necessary to make effective training decisions related to the selection, adoption, value, and modification of various instructional activities” (Goldstein, p. 147).28 Evaluation provides insight into whether the content and methods utilised in training were appropriate, how to maximise transfer, and may serve as the content of feedback delivery for programme participants. The bottom line is that without systematic evaluation the organisation will not know if the money spent on training is worthwhile or if it is training the correct skills effectively, which in turn may open organisations to litigation.

While very important systematic evaluation of training within a real world context (outside the laboratory) is not easy for the following reasons: (a) often not a well defined measurable, “ultimate” criterion, (b) resource intensive, (c) organisations want to see return on investment (ROI) and this requires a longitudinal approach, and (d) many don’t know how to make a case for systematic evaluation. Given these constraints and the absence of an ultimate criterion it becomes important to assess training at multiple levels. In this vein, Kirkpatrick argued for a multi-level approach to training evaluation consisting of four levels of evaluation: (1) reactions (utility, affective), (2) learning, (3) behaviour (that is, extent of performance change), and (4) results (that is, degree of impact on organisational effectiveness or mission success). Within recent years, this typology has been expanded by several researchers.77 For example, researchers have argued that learning is multi-dimensional and results in cognitive, affective, and skill based learning outcomes.77 In addition, collecting data at multiple levels is important not only in that each additional source of data serves to increase confidence in the overall evaluation,78 but because there is not an automatic link between the levels. For example, although reaction data can indicate whether the trainee felt the programme was worthwhile, it has little if any relation to whether the participant learned the material. Similarly, just because a trainee learned the knowledge during training does not guarantee that he/she can translate this knowledge into effective behaviour, nor does it guarantee that, if applied, the behaviour will have an effect on organisational outcomes.

The importance of systematic evaluation efforts can be seen within the aviation community and its application of CRM training, which has been touted as a success anecdotally. A recent review of published CRM training studies indicated that while CRM is generally effective in producing some level of change in participants (for example, reaction, learning, behaviour), a lack of multi-level evaluation efforts makes it difficult to answer whether CRM is truly effective.17 For further examples of how information is gathered on each of these levels the reader is referred to work by Salas and colleagues.17

APPLYING THE GUIDANCE

Throughout this paper we have provided a series of theoretically based and empirically validated guidelines that can assist the medical community in transforming teams of experts into expert teams. Couched within many of the guidelines are tools which can assist practitioners in their implementation. While the presented guidelines represent a start they only touch the tip of the iceberg in terms of what is known about the creation and maintenance of effective teams. Much of what has been presented is based on evidence culminating from the aviation and military communities and their large investment in teams over the last 20 years. If there is one lesson that the medical community needs to learn from those who have come before, it is that the creation and maintenance of effective teams takes time and effort. In order for organisations to capitalise on the synergy present within teams they must INVEST. The guidelines presented within this paper are only the first step.

CONCLUSION

Within dynamic and complex environments—for example, the medical community—teams can be used as an intervention by which to promote safety and reduce errors. However, if not properly trained in the requisite teamwork competencies teams can actually become a liability to the organisation. The medical community has begun to recognise this fact and has turned to the military and aviation communities to look for interventions that could be adapted for use within a medical domain. This has resulted in the medical community beginning to implement a form of CRM training. However while adopting team interventions from communities with more experience there has been a lack of practical guidance for those in the medical community pertaining to the development, implementation, and evaluation of such programmes. It is our hope that this article serves as an initial step in providing that guidance along with promoting awareness within the medical community that CRM training is only one of many team training interventions.

REFERENCES

Footnotes

  • Competing interests: None.

  • Copies of the unpublished articles cited in this paper are available on request from the corresponding author.