Inter-Rater Agreement of Auscultation, Palpable Fremitus, and Ventilator Waveform Sawtooth Patterns Between Clinicians
======================================================================================================================

* Marc P Berry
* Joan-Daniel Martí
* George Ntoumenopoulos

## Abstract

**BACKGROUND:** Clinicians often use numerous bedside assessments for secretion retention in participants who are receiving invasive mechanical ventilation. This study aimed to evaluate inter-rater agreement between clinicians when using standard clinical assessments of secretion retention and whether differences in clinician experience influenced inter-rater agreement.

**METHODS:** Seventy-one mechanically ventilated participants were assessed by a research clinician and by one of 13 ICU clinicians. Each clinician conducted a standardized assessment of lung auscultation, palpation for chest-wall (rhonchal) fremitus, and ventilator inspiratory/expiratory flow-time waveforms for the sawtooth pattern.

**RESULTS:** On the presence of breath sounds, agreement ranged from absolute to moderate in the upper zones and the lower zones, respectively. Kappa values for abnormal and adventitious lung sounds achieved moderate agreement in the upper zones, less than chance agreement to substantial agreement in the middle zones, and moderate agreement to almost perfect agreement in the lower zones. Moderate to almost perfect agreement was established for palpable fremitus in the upper zones, moderate to substantial agreement in the middle zones, and less than chance to moderate agreement in the lower zones. Inter-rater agreement on the presence of expiratory sawtooth pattern identification showed moderate agreement. The level of percentage agreement between the research and ICU clinicians for each respiratory assessment studied did not relate directly to level of clinical experience.

**CONCLUSIONS:** Inter-rater agreement for all assessments showed variability between lung regions but maintained reasonable percentage agreement in mechanically ventilated participants. The level of percentage agreement achieved between clinicians did not directly relate to clinical experience for all respiratory assessments. Therefore, these respiratory assessments should not necessarily be viewed in isolation but interpreted within the context of a full clinical assessment.

*   auscultation
*   palpation
*   ventilators
*   mechanical
*   observer variation
*   respiratory sounds
*   critical care
*   humans

## Introduction

A bedside respiratory assessment of patients receiving mechanical ventilation in the ICU is considered a fundamental part of clinical practice.1 Respiratory complications frequently occur in patients who are mechanically ventilated due to the presence of artificial airways2,3 and sedation,4 inadequate airway humidification,5,6 impaired mucociliary clearance,7 diminished cough,8 and variable airway pressures.9 All these factors have the potential to cause respiratory impairment8,10 and exacerbate pre-morbid respiratory injury or disease.10 Imaging techniques such as radiography or computed tomography are considered more accurate modalities than most bedside assessments (such as auscultation) at providing information on pulmonary dysfunction11 but lack the ability to provide temporal changes in pulmonary status, especially at the point of a clinical intervention or deterioration.12 This therefore warrants the need for other timely bedside assessment to support accurate monitoring and diagnoses of respiratory dysfunction and, in turn, supports clinical decision making in real time.13,14

Bedside respiratory assessments are most commonly performed by doctors, nursing staff,15 physical therapists,13 and respiratory therapists16 to differentiate between a range of pulmonary disorders, including pulmonary edema, consolidation, atelectasis, interstitial lung disease, bronchospasm, and retained secretions.1 One of the most common bedside respiratory assessments performed in clinical practice is lung auscultation,1,17 which is considered an integral part of the clinical reasoning process for qualified and training health-care professionals.1,18 This is because auscultation is a cost-effective, easily applied respiratory tool that can be used to assess changes in lung sounds that may be associated with certain respiratory pathologies or dysfunction.19 Auscultation works on the concept that sound vibration is created and amplified by turbulent air flow in the major airways.20,21 This sound energy is then transmitted to the stethoscope via the chest wall but is filtered and altered by structures and fluid within the thorax and further altered by lung disorders.1 Breath sounds, both in the absence and presence of lung disorders, have been shown (via respiratory acoustic analysis) to have specific sound signatures that reflect the underlying dysfunction.1,22 However, the ability to recognize, differentiate between, and associate these acoustic signatures with specific lung disorders via auscultation relies heavily on the experience of the clinician, the acuity of the clinician's hearing, and his or her personal interpretation of what is heard.21 Furthermore, differences in nomenclature used for the lung sounds heard have led to discrepancies between clinicians and their clinical interpretations of auscultation.1,22 As such, the literature surrounding lung auscultation suggests that inter-rater agreement for this assessment modality is poor to moderate, irrespective of the level of clinical experience.23 However, the inter-rater agreement of auscultation in patients who are ventilator-dependent has not been established, despite its frequency of use in the ICU population. Moreover, it is not known whether the level of clinical experience affects inter-rater agreement in this ICU population.

In addition to auscultation, tactile respiratory assessments are often used to assess for pulmonary dysfunction.24,25 Tactile (vocal) fremitus assessments utilize chest wall palpation to detect the changes in the intensity of vibrations created with specific spoken words to indicate toward certain lung pathologies. Research surrounding inter-rater agreement has shown variability, with some studies showing poor inter-rater agreement26 and others showing excellent inter-rater agreement.25 However, this technique is not possible during mechanical ventilation due to the presence of an artificial airway preventing phonation. A type of fremitus that is possible to assess for during mechanical ventilation is rhonchal fremitus. This type of fremitus (also known as palpable fremitus24 in clinical practice) is commonly related by clinicians to the presence of retained airway secretions. It is often used as an adjunct to a respiratory assessment to support the need for or the effectiveness of a therapeutic intervention for the removal of retained airway secretions. In fact, palpable fremitus was considered a key indicator for warranting chest physiotherapy for excessive secretions by an expert Delphi panel in a study by Hanekom et al.24 As with auscultation, the level of inter-rater agreement for palpable fremitus and the extent to which clinical experience influences inter-rater agreement is unknown for patients who are dependent on mechanical ventilation.

With advances in mechanical ventilation has come real-time graphical representation of airway characteristics.27 As with lung sounds, ventilator waveforms also have specific signature patterns that represent certain lung disorders, such as obstructive lung disease and retained airway secretions.27 Studies by Jubran and Tobin28 and Guglielminotti et al29 found that a sawtooth pattern on a flow-volume loop had a high likelihood ratio of retained secretions in ventilator-dependent subjects. They also found excellent inter-rater agreement when assessing for the sawtooth pattern. The use of flow-time waveform to detect the sawtooth pattern on expiration has also been explored.30

Because the presence of a sawtooth pattern on the ventilation waveform is currently considered the most valid indicator of retained secretions (coupled with auscultation of harsh breath sound over the trachea),31 it is important to ensure that inter-rater agreement is consistent between a range of clinicians of varying abilities. This would assist in accurate clinical decision making when considering therapeutic intervention for retained secretions.

The primary aim of this study was to evaluate the inter-rater agreement between a single research clinician and an ICU clinician for 3 standard clinical assessments, namely lung auscultation, chest-wall palpation for palpable fremitus, and expiratory flow-time waveform analysis for the sawtooth pattern, in participants who are mechanically ventilated and at risk of secretion retention. A secondary aim was to explore whether the level of clinical experience was a factor in inter-rater agreement.

### QUICK LOOK

#### Current knowledge

Clinicians often use auscultation, ventilator flow waveform analysis, and chest wall palpation as part of respiratory assessments in the ICU. Inter-rater agreement for lung auscultation in spontaneously breathing patients is poor to moderate, irrespective of clinical experience. Ventilator flow waveform analysis for the presence of a sawtooth pattern has shown excellent inter-rater agreement. Palpable chest wall fremitus has been considered a key indicator for warranting intervention for excessive secretion retention.

#### What this paper contributes to our knowledge

Inter-rater agreement for auscultation, palpable chest wall fremitus, and flow waveform sawtooth patterns showed variability between lung regions but maintained reasonable percentage agreement in mechanically ventilated subjects. The level of percentage agreement between clinicians did not directly relate to clinical experience for all respiratory assessments. Therefore, these respiratory assessments should not necessarily be viewed in isolation but interpreted within the context of a full clinical assessment.

## Methods

### Participants

This study was approved by the Research and Development Department at St Thomas' Hospital, London, and ethical approval was obtained from the South East London Ethics Committee.

All mechanically ventilated patients were prospectively screened for study eligibility and enrollment during their admission to a 30-bed ICU between October 2011 and May 2012. Study inclusion criteria were participants ≥18 y old; invasively mechanically ventilated via endotracheal tube or tracheostomy; and clinical indication for chest physiotherapy, based on local ICU physiotherapy treatment guidelines for retained secretions (ie, loss of lung volumes, gas exchange mismatch, and/or respiratory pump failure). Exclusion criteria were patients with active pulmonary tuberculosis, active H1N1 influenza A, or active pulmonary bleeding and those who were unable to be disconnected from mechanical ventilation (PEEP >10 cm H2O, FIO2 >0.6 or participants receiving high-frequency oscillatory ventilation). These exclusion criteria were such due to risk associated with ventilator disconnection to clear condensate from the circuit or while establishing participants on a standard ventilator for the study (because a number of different ventilators were in use in the ICU). Informed consent was gained from the participants' next of kin to enroll in the study as per the request of the ethics committee. Participants were only studied on one occasion.

### Study Design

For each of the 71 participants enrolled in this prospective observational study, an independent and simultaneous assessment was carried out by 2 observers: the research clinician (band 7 clinician), who remained constant, and one of 13 different ICU physiotherapy clinicians (four band 5, five band 6, two band 7, one band 8a, and one band 8c), who varied between participants and in terms of clinical experience (Table 1). Band 5 clinicians were junior staff members with non-specialist respiratory skills, band 6 therapists were clinicians at the beginning of specializing in respiratory care, and bands 7–8c had increasing clinical respiratory care experience, with band 8c being considered the most expert band of clinicians. Before assessment, condensate from the ventilator circuit was systematically removed, oral suctioning was performed, and airway sub-glottic ports were aspirated.

View this table:
[Table 1.](http://rc.rcjournal.com/content/61/10/1374/T1)

Table 1. 
Levels of Experience According to the National Health Service Agenda for Change Pay Grades

The research clinician and the ICU clinician then conducted their standardized assessment, which included lung auscultation, bilateral chest palpation for palpable chest wall fremitus, and visual inspection of ventilator expiratory flow-time waveforms for the sawtooth pattern. Of note, no discussion or communication between the 2 clinicians was permitted.

### Lung Auscultation

Lung auscultation was undertaken using a dual-limb stethoscope (Classic II SE Teaching Stethoscope, 3M Littmann, St. Paul, Minnesota). Six standard antero-lateral lung areas (right and left upper, middle, and lower zones) were assessed with the subject in a supine, head-up 30° position. The ICU and research clinicians documented their auscultation findings separately on standardized data sheets, using standardized terminology for each lung area (Table 2). They included information on the presence or absence of breath sounds, type of additional abnormal (ie, bronchial breathing), and adventitious lung sounds (ie, crackles, wheezes, or pleural rub), including the phase of respiration in which these sounds occurred (Table 2).

View this table:
[Table 2.](http://rc.rcjournal.com/content/61/10/1374/T2)

Table 2. 
Standardized Terminology for Breath Sounds, Abnormal and Adventitious Lung Sounds Used by the Research and ICU Clinician During Their Assessment of the 6 Lung Zones

### Chest Wall Palpation

Bilateral chest wall palpation of the 6 lung regions was undertaken by the ICU and research clinicians to assess the presence of palpable fremitus. Anatomical areas were not marked out; hence, hand positioning was approximated, as per clinical practice. The findings of the ICU and research clinicians were recorded, again independently of each other.

### Flow-Time Ventilator Waveforms

Participants were established on an Avea ventilator (CareFusion, San Diego, California). Flow-time waveforms and ventilator parameters were recorded in real time for 30 s, using a screen capture device (Epiphan Screen Capture Tools, Epiphan Systems, Ottawa, Canada). The 30-s screen capture recording of ventilator flow-time waveforms was visually inspected for the sawtooth pattern by both the ICU and research clinician. Once again, their assessment findings were documented independently, maintaining blinding.

### Statistical Analysis

All statistical analyses were conducted using Stata 12 (StataCorp, College Station, Texas) and Microsoft Office Excel 2003 (Microsoft Corporation, Redmond, Washington). Descriptive statistics are presented as the mean ± SD. Data were assessed for normality using the Kolmogorov-Smirnov test. The dependent variable was tested using Cohen's kappa coefficient, as a measure of inter-rater agreement, assuming equal weighting. The standardized kappa agreement was rated as follows: <0 = less than chance agreement; 0.01–0.20 = slight agreement; 0.21–0.40 = fair agreement; 0.41–0.60 = moderate agreement; 0.61–0.80 = substantial agreement; 0.81–0.99 = almost perfect agreement.32 Data analysis included comparisons of the research and ICU clinicians' findings for all regions of the lung assessed. The level of experience of each clinician related to level of agreement was also explored. A value of *P* < .05 was considered significant for all tests.

This study is part of a larger study exploring novel markers of secretion retention, of which the 3 clinical bedside assessments were used, in conjunction with other novel markers and assessment devices. One such device was vibration response imaging, an acoustic lung monitoring device which visually displays lung sound amplitude. Based on a previous pilot trial,33 the power calculation for this study required a sample size of 71 subjects to provide 90% power to detect a difference of 15 in the sound amplitude (arbitrary units) of the vibration response imaging acoustic lung sound imaging, using a 2-sided test and a 5% significance level.

## Results

### Inter-Rater Agreement

Of the enrolled participants, the majority were intubated males receiving Continuous Spontaneous Ventilation (CPAP or CPAP/PSV), which were heated and humidified. Table 3 presents descriptive data of the included participants.

View this table:
[Table 3.](http://rc.rcjournal.com/content/61/10/1374/T3)

Table 3. 
Participant Characteristics, Airway Characteristics, and Ventilator Setup Summary Data

Inter-rater agreement between the research and ICU clinicians on the detection of the presence of breath sounds (Table 4) reached absolute agreement in the upper zone (no κ value was calculated because both observers' responses were constant), whereas substantial to almost perfect agreement in the middle zones (κ = 0.66–1.0) was found. However, the right and left lower zones both only achieved moderate agreement (κ = 0.45 and κ = 0.42, respectively) and therefore achieved the least percentage agreement (Table 4) of all of the zones.

View this table:
[Table 4.](http://rc.rcjournal.com/content/61/10/1374/T4)

Table 4. 
Levels of Agreement on Auscultation of Breath Sounds in All Lung Zones

In the upper zones, inter-rater agreement for abnormal and adventitious sounds (Table 5) were moderate (κ = 0.47–0.60) with only inspiratory crackles in the right upper zone and expiratory crackles in the left upper zones achieving substantial agreement (κ = 0.61; κ = 0.70, respectively). The highest percentage agreement between clinicians occurred with bronchial breathing and pleural rub, where nearly perfect agreement on the absence of these abnormal and adventitious sounds occurred (see Table 1 in the supplementary materials at [http://www.rcjournal.com](http://www.rcjournal.com)). However, a kappa agreement on the presence of these adventitious sounds was unable to be calculated because at least one variable in each 2-way table upon which measures of association are computed was a constant.

View this table:
[Table 5.](http://rc.rcjournal.com/content/61/10/1374/T5)

Table 5. 
Levels of Agreement on Auscultation of Abnormal and Adventitious Sounds in All Lung Zones

Inter-rater agreement in the middle zones (see Table 5) saw great variability in kappa values for specific abnormal and adventitious sounds, ranging from less than chance agreement (κ = −0.01) to substantial agreement (κ = 0.72) on the left and less than chance agreement (κ = −0.1) to moderate agreement (κ = 0.58) on the right.

Within the left lower zone, all kappa values for abnormal and adventitious sounds (see Table 5) ranged from substantial to almost perfect agreement (κ = 0.63–1.0), apart from inspiratory crackles (κ = 0.59). The right lower zone saw half of the abnormal and adventitious sounds achieving fair to moderate agreement (κ = 0.36–0.57) and half reaching substantial agreement (κ = 0.63–0.67).

As with auscultation, the inter-rater agreement on the presence of palpable fremitus within the 6 lung regions varied considerably (Table 6). Moderate agreement was established for fremitus in the left upper zone (κ = 0.53). Almost perfect agreement (κ = 0.84) was found in the right upper zone. Middle zones showed substantial agreement (κ = 0.63–0.66). Lower zones presented with the lowest level of inter-rater agreement, only achieving fair agreement to moderate agreement (κ = 0.36–0.41).

View this table:
[Table 6.](http://rc.rcjournal.com/content/61/10/1374/T6)

Table 6. 
Levels of Agreement on Palpable Fremitus in All Lung Zones

Inter-rater agreement on the presence of expiratory sawtooth pattern identification (Table 7) showed moderate agreement (κ = 0.59), with 80.3% agreement. Inspiratory sawtooth patterns identified fair agreement (κ = 0.30), with 83.1% agreement.

View this table:
[Table 7.](http://rc.rcjournal.com/content/61/10/1374/T7)

Table 7. 
Levels of Agreement on Inspiratory and Expiratory Ventilator Flow-Time Sawtooth Patterns

### Experience of Clinician

For auscultation, the experience of the ICU clinician did not influence the extent to which he or she agreed on the presence or absence of breath sounds with the research clinician (Fig. 1). High levels of percentage agreement between the research clinician and all bands of ICU clinicians in the upper and middle zones for the presence or absence of breath sounds was seen. However, percentage agreement between the research clinician and all levels of experience of ICU clinicians showed discrepancies in the lower zones (ranging from 80 to 100%); with lowest percentage agreements occurring in the band 6 group (see Fig. 1).

![Fig. 1.](http://rc.rcjournal.com/http://rc.rcjournal.com/content/respcare/61/10/1374/F1.medium.gif)

[Fig. 1.](http://rc.rcjournal.com/content/61/10/1374/F1)

Fig. 1. 
Percentage agreements between the research clinician and ICU clinicians, of variable experience, on the presence or absence of breath sounds during auscultation in all lung zones.

Generally, the percentage agreement between the research clinician and all bands of ICU clinicians for abnormal and adventitious sounds varied greatly (ranging from 70 to 100%), depending on the sound heard and the zone of the lung (Fig. 2). The highest percentage agreements tended to occur in the band 8 (most experienced) and band 6 groups, but only in relation to specific abnormal or adventitious sounds, because these groups also saw low levels of percentage agreement for other sounds auscultated. The band 5 clinicians (least experienced) saw low percentage agreements (≤80%) most frequently, but again this was zone, abnormal sound and adventitious sound specific (see Fig. 2).

![Fig. 2.](http://rc.rcjournal.com/http://rc.rcjournal.com/content/respcare/61/10/1374/F2.medium.gif)

[Fig. 2.](http://rc.rcjournal.com/content/61/10/1374/F2)

Fig. 2. 
Percentage agreements between the research clinician and ICU clinicians, of variable experience, on the presence or absence of abnormal and adventitious sounds during auscultation in all lung zones. Br = bronchial breath sounds, Icr = inspiratory crackles, Iwh = inspiratory wheeze, Ecr = expiratory crackles, Ewh = expiratory wheeze, Pr = pleural rub.

Percentage agreement for the sawtooth pattern on the flow-time waveforms (Fig. 3) was highest in the band 7 group (100%) and lowest in the band 8 cohort (77%) for expiratory flow. In contrast, for inspiratory flow, the greatest percentage agreement occurred in the band 5 group (100%) and the lowest in the band 6 group (73%). Indeed, there was not a consistent percentage agreement for any of the 4 bands when assessing flow-time waveforms.

![Fig. 3.](http://rc.rcjournal.com/http://rc.rcjournal.com/content/respcare/61/10/1374/F3.medium.gif)

[Fig. 3.](http://rc.rcjournal.com/content/61/10/1374/F3)

Fig. 3. 
Percentage agreements between the research clinician and ICU clinicians, of variable experience, on the presence or absence of ventilator flow-time sawtooth pattern during inspiration and expiration.

When assessing for palpable fremitus, the research and ICU clinicians showed no consistent percentage agreement in relation to a specific band; again, the percentage agreement for each band varied greatly between lungs and zones of assessment (Fig. 4).

![Fig. 4.](http://rc.rcjournal.com/http://rc.rcjournal.com/content/respcare/61/10/1374/F4.medium.gif)

[Fig. 4.](http://rc.rcjournal.com/content/61/10/1374/F4)

Fig. 4. 
Percentage agreements between the research clinician and ICU clinicians, of variable experience, on the presence or absence of palpable fremitus in all lung zones.

## Discussion

This study demonstrates that auscultation of breath sounds in critically ill subjects receiving invasive mechanical ventilation exhibits good inter-rater agreement compared with that of abnormal or adventitious sounds, although inter-rater agreement worsens in the lower zones of the lungs. We also found that inter-rater agreement during palpation for chest-wall fremitus decreases from the upper to lower zones assessed and that identification of a sawtooth pattern on the ventilator flow-time waveforms may have moderate to fair agreement, particularly on inspiratory flow. Finally, although level of experience was not a key focus of this study, we found high levels of variability in percentage agreement between ICU and research clinicians, irrespective of level of experience.

In our study, agreement during auscultation of breath sounds varied from absolute in the upper zones of the lungs to moderate in the lower zones. Moreover, inter-rater agreement differed considerably between all abnormal and adventitious sounds, irrespective of zone. Despite some studies showing excellent inter-rater agreement when detecting breath sounds on auscultation (95.80% agreement; κ = 0.89),25 inter-rater agreement has tended to be reported as poor.23 Regarding abnormal and adventitious sounds, similar results of our study were reported by Kalantri et al,25 who reported less than chance agreement (κ = −0.2; CI −0.57 to 0.78) on auscultating pleural rub but a 94.27% agreement between clinicians. To some extent, our results were to be expected, due to natural variance in lung sound amplitude across different lung regions,22 which may be further altered by lung disease (eg, emphysema)34 or ventilator settings (eg, increase in PEEP).35 Moreover, auscultation relies on the clinician's acuity of hearing21; therefore, variances in inter-rater agreement regarding the presence of breath sounds in the lower regions of the lungs may occur if its presence is subtle. Another explanation for this variability is the therapist's interpretation of the nomenclature surrounding the abnormal and adventitious sounds, which may have inaccuracies.1 Additionally, despite high percentage levels of agreement with some adventitious sounds being coupled with lower kappa values, the kappa values should be deemed representative of inter-rater agreement because this considers the influence of chance, whereas percentage agreement does not.36 However, because kappa values are influenced by the prevalence of the factor being investigated, infrequent occurrences of clinical factors (eg, bronchial breathing) may generate low kappa values but not necessarily be representative of low overall agreement.37

Despite scant evidence for its clinical benefit, palpable chest-wall fremitus is commonly used by practitioners38 and has been established as a key clinical indicator of retained secretions throughout management of pulmonary dysfunction in adults receiving mechanical ventilation.24 Overall, we found moderate therapist agreement on detecting palpable fremitus, but this was inconsistent among different regions of the lungs. In fact, the upper zones of the lungs exhibit moderate to almost perfect agreement, whereas less than chance to moderate agreement was found in the basal zones. In previous studies, tactile (vocal) fremitus when conducting a pulmonary assessment showed both poor (κ = 0.25)26 and excellent (κ = 0.86)25 inter-rater agreement. Although these studies looked at vocal rather than a secretion-related resonance, they corroborate that tactile assessments may vary in their accuracy between clinicians. A reasonable explanation for these results is that chest-wall resonance may be transmitted wider than its origin (ie, area of retained secretions), and tactile feedback may transcend the boundaries of the assessed lung zones. Therefore, as stated in previous studies,6,21 palpable chest-wall fremitus should be always accompanied by other signs of retained secretions to increase the robustness of clinical decisions.

Several studies7,8 in intubated subjects have associated the presence of a sawtooth pattern on a ventilator flow-volume loop with retained airway secretions, with excellent inter-rater agreement between clinicians. Conversely, in our study, identification of a sawtooth pattern on expiratory and inspiratory flow-time waveforms demonstrated moderate and fair agreement, respectively. Jubran and Tobin28 recorded ventilator flow-volume loops via an external recording device connected to the endotracheal tube during spontaneous breathing off of mechanical ventilation. Therefore, waveforms may not have been subjected to the same airway variability during mechanical ventilation as in the current study (ie, flows and airway pressures) or to residual condensate in the ventilator tubing, which has the potential to add ambiguity to sawtooth interpretation. In addition, the authors used flow-volume loops, whereas the current study analyzed flow-time waveforms, which have been theorized to be less sensitive at detecting sawtooth patterns.30 However, flow-time waveforms are more commonly used in clinical practice than flow-volume loops30; thus, their use may make our results more applicable to current ICU practice. Finally, a recent study by Sole et al30 clearly distinguished between oscillation of a sawtooth pattern due to secretions, and a wavy flow-time waveform due to condensate. Therefore, it is possible that clinicians in our study may have misinterpreted wavy waveforms as a sawtooth pattern. These contradictory results highlight the need for standardization of ventilator waveform analysis through education, particularly because the sawtooth pattern is currently considered one of the 2 best objective markers indicating the presence of retained airway secretions in patients who are mechanically ventilated.28–31

We also found that the level of training of the research and ICU clinician did not influence the consistency of percentage agreement for all assessment techniques. This is in accordance with a study by Brooks and Thomas23 in which inter-rater agreement varied depending on the breath sound heard but was not affected by level of clinical experience. Mangione and Nieman39 reported inaccuracies in pulmonary auscultation skills between pulmonary fellows, family practitioners, internal medicine residents, and medical students when assessing 10 common pulmonary events. However, they suggested that experience did not influence the ability of a clinician to correctly identify a pulmonary event via auscultation and that education would be the key to improving a clinician's auscultation skills.39

Our study may have a number of drawbacks that merit discussion. First, the nomenclature of the abnormal and adventitious sounds was not explicitly outlined for the researcher or the clinician before the study; nor was the definition of a sawtooth pattern or of palpable fremitus. Moreover, the abnormal and adventitious sounds were grossly grouped by location, type of sound, and phase of breath. Probably, these groupings did not allow the clinician to discretely differentiate between the nature of each sound heard (eg, between coarse and fine crackles) or consider the temporal variance of the sound during each phase of breathing, resulting in some variance in the inter-rater agreement. Second, clinicians did not auscultate and palpate the posterior regions of the participant's chest wall because of the difficulty in standardizing patient positions and assessment of the posterior regions. Thus, the inter-rater agreement of auscultation and palpation reported in this study cannot be extrapolated to assessments of the posterior chest wall. In clinical practice, clinicians would be expected to auscultate and palpate all regions as a comprehensive respiratory assessment, but our study still highlights the extensive variability in the inter-rater agreement of all the respiratory assessments. Finally, the clinician's understanding of techniques and assessments used in the study may have varied, since their training was not standardized and occurred at different periods of time, was delivered by a number of different ICU practitioners, and varied in depth of explanation. This suggests that the reliability of auscultation, palpation, and flow-time sawtooth pattern identification is not inherent within the assessment but lies within the level of education, skill, and understanding of the practitioner conducting the assessment.

## Conclusion

Inter-rater agreement during auscultation for the presence of breath sounds and chest wall palpation for fremitus was almost perfect for the upper regions of the lungs, mainly substantial in the middle zones, and moderate in the lower zones. Percentage agreement between the ICU and research clinicians was high for breath sounds throughout all lung zones, as was percentage agreement for palpable fremitus. Inter-rater agreement for all abnormal and adventitious breath sounds varied greatly throughout the upper, middle, and lower zones, particularly where more adventitious sounds were present. Despite these variances, the percentage agreement between the ICU and research clinicians remained reasonably high. Inter-rater agreement was moderate during ventilator waveform analysis for the presence of a sawtooth pattern. Overall, the experience of clinicians did not influence the level of percentage agreement achieved for all respiratory assessments studied. Interpretations of the bedside respiratory assessments explored in this study have the potential to vary between clinicians, irrespective of experience, and between the lung zones assessed. Therefore, these respiratory assessments should not necessarily be viewed in isolation but should be interpreted within the context of a full clinical assessment. Further studies are warranted to assess inter-rater agreement between clinicians for bedside respiratory assessments, in particular exploring the influence of standardized education and training regimes. Furthermore, future research should look at the diagnostic accuracy and intra-rater reliability of bedside respiratory assessments in patients who are mechanically ventilated.

## Footnotes

*   Correspondence: Marc P Berry, Physiotherapy Department, Hampshire Hospitals NHS Foundation Trust, Basingstoke and North Hampshire Hospital, Aldermaston Road, Basingstoke, Hampshire RG24 9NA, United Kingdom. E-mail: marc.berry{at}hhft.nhs.uk.
*   Supplementary material related to this paper is available at [http://www.rcjournal.com](http://www.rcjournal.com).

*   This work was supported by Guy's and St Thomas' Charity. The authors have disclosed no conflicts of interest.

*   Copyright © 2016 by Daedalus Enterprises

## References

1.  1.Bohadana A, Izbicki G, Kraman SS. Fundamentals of lung auscultation. N Engl J Med 2014;370(8):744–751.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1056/NEJMra1302901&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=24552321&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

2.  2.Keller C, Brimacombe J. Bronchial mucus transport velocity in paralyzed anesthetized patients: a comparison of the laryngeal mask airway and cuffed tracheal tube. Anesth Analg 1998;86(6):1280–1282.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1097/00000539-199806000-00028&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=9620520&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=000073882700027&link_type=ISI) 

3.  3.Sackner MA, Hirsch J, Epstein S. Effect of cuffed endotracheal tubes on tracheal mucous velocity. Chest 1975;68(6):774–777.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1378/chest.68.6.774&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=1192854&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=A1975AY71300007&link_type=ISI) 

4.  4.Reade MC, Finfer S. Sedation and delirium in the intensive care unit. N Engl J Med 2014;370(5):444–454.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1056/NEJMra1208705&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=24476433&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

5.  5.Lorente L, Lecuona M, Jiménez A, Mora ML, Sierra A. Ventilator-associated pneumonia using a heated humidifier or a heat and moisture exchanger: a randomized controlled trial [[ISRCTN88724583](http://rc.rcjournal.com/external-ref?link_type=ISRCTN&access_num=ISRCTN88724583)]. Crit Care 2006;10(4):R116.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1186/cc5009&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=16884530&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

6.  6.Seo H, Kim SH, Choi JH, Hong JY, Hwang JH. Effect of heated humidified ventilation on bronchial mucus transport velocity in general anaesthesia: a randomized trial. J Int Med Res 2014;42(6):1222–1231.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BpbXIiO3M6NToicmVzaWQiO3M6OToiNDIvNi8xMjIyIjtzOjQ6ImF0b20iO3M6MjU6Ii9yZXNwY2FyZS82MS8xMC8xMzc0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

7.  7.Konrad F, Schreiber T, Brecht-Kraus D, Georgieff M. Mucociliary transport in ICU patients. Chest 1994;105(1):237–241.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1378/chest.105.1.237&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=8275739&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=A1994MQ72100049&link_type=ISI) 

8.  8.P. Berry M, Marti J-D. Clinical management of secretion retention in critically ill patients who are intubated and mechanically ventilated. Curr Respir Med Rev 2014;10(3):163–175.
    
    

9.  9.Protti A, Votta E, Gattinoni L. Which is the most important strain in the pathogenesis of ventilator-induced lung injury: dynamic or static? Curr Opin Crit Care 2014;20(1):33–38.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=24247615&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

10. 10.1.  Pinsky MR, 
    2.  Brochard L, 
    3.  Hedenstierna G, 
    4.  Antonelli M
    
    Tremblay L, Slutsky A. Ventilator-induced lung injury: from the bench to the bedside. In: Pinsky MR, Brochard L, Hedenstierna G, Antonelli M, editors. Applied Physiology in Intensive Care Medicine 1. Berlin: Springer; 2012:343–352.
    
    

11. 11.Murphy RL. In defense of the stethoscope. Respir Care 2008;53(3):355–369.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoicmVzcGNhcmUiO3M6NToicmVzaWQiO3M6ODoiNTMvMy8zNTUiO3M6NDoiYXRvbSI7czoyNToiL3Jlc3BjYXJlLzYxLzEwLzEzNzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

12. 12.Victorino JA, Borges JB, Okamoto VN, Matos GF, Tucci MR, Caramez MP, et al. Imbalances in regional lung ventilation: a validation study on electrical impedance tomography. Am J Respir Crit Care Med 2004;169(7):791–800.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1164/rccm.200301-133OC&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=14693669&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=000220494100008&link_type=ISI) 

13. 13.Gosselink R, Bott J, Johnson M, Dean E, Nava S, Norrenberg M, et al. Physiotherapy for adult patients with critical illness: recommendations of the European Respiratory Society and European Society of Intensive Care Medicine Task Force on physiotherapy for critically ill patients. Intensive Care Med 2008;34(7):1188–1199.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1007/s00134-008-1026-7&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=18283429&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=000257123400005&link_type=ISI) 

14. 14.Marques A, Oliveira A, Jácome C. Computerized adventitious respiratory sounds as outcome measures for respiratory therapy: a systematic review. Respir Care 2014;59(5):765–776.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoicmVzcGNhcmUiO3M6NToicmVzaWQiO3M6ODoiNTkvNS83NjUiO3M6NDoiYXRvbSI7czoyNToiL3Jlc3BjYXJlLzYxLzEwLzEzNzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

15. 15.West SL. Physical assessment: whose role is it anyway? Nurs Crit Care 2006;11(4):161–167.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=16869522&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

16. 16.Kacmarek RM, Stoller JK, Heuer A. Egan's fundamentals of respiratory care, 10th edition. Amsterdam: Elsevier Health Sciences; 2013.
    
    

17. 17.Sovijärvi A, Vanderschoot J, Earis J. Standardization of computerized respiratory sound analysis. Eur Respir Rev 2000; 10(77):585.
    
    

18. 18.Higgs J, Jones M, Loftus S, Christensen N. Clinical reasoning in the health professions, 3rd edition. Amsterdam: Elsevier Health Sciences; 2008.
    
    

19. 19.Sole ML, Bennett M. Comparison of airway management practices between registered nurses and respiratory care practitioners. Am J Crit Care 2014;23(3):191–199.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpjYyI7czo1OiJyZXNpZCI7czo4OiIyMy8zLzE5MSI7czo0OiJhdG9tIjtzOjI1OiIvcmVzcGNhcmUvNjEvMTAvMTM3NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

20. 20.Welsby PD, Parry G, Smith D. The stethoscope: some preliminary investigations. Postgrad Med J 2003;79(938):695–698.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTI6InBvc3RncmFkbWVkaiI7czo1OiJyZXNpZCI7czoxMDoiNzkvOTM4LzY5NSI7czo0OiJhdG9tIjtzOjI1OiIvcmVzcGNhcmUvNjEvMTAvMTM3NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

21. 21.Alsmadi S, Kahya YP. Design of a DSP-based instrument for real-time classification of pulmonary sounds. Comput Biol Med 2008;38(1):53–61.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=17716642&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

22. 22.Pasterkamp H, Kraman SS, Wodicka GR. Respiratory sounds: advances beyond the stethoscope. Am J Respir Crit Care Med 1997;156(3 Pt 1):974–987.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1164/ajrccm.156.3.9701115&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=9310022&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=A1997XX17700044&link_type=ISI) 

23. 23.Brooks D, Thomas J. Interrater reliability of auscultation of breath sounds among physical therapists. Phys Ther 1995;75(12):1082–1088.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToicHRqb3VybmFsIjtzOjU6InJlc2lkIjtzOjEwOiI3NS8xMi8xMDgyIjtzOjQ6ImF0b20iO3M6MjU6Ii9yZXNwY2FyZS82MS8xMC8xMzc0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

24. 24.Hanekom S, Berney S, Morrow B, Ntoumenopoulos G, Paratz J, Patman S, Louw Q. The validation of a clinical algorithm for the prevention and management of pulmonary dysfunction in intubated adults: a synthesis of evidence and expert opinion. J Eval Clin Pract 2011;17(4):801–810.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=20630012&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

25. 25.Kalantri S, Joshi R, Lokhande T, Singh A, Morgan M, Colford JM Jr., Pai M. Accuracy and reliability of physical signs in the diagnosis of pleural effusion. Respir Med 2007;101(3):431–438.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1016/j.rmed.2006.07.014&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=16965906&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

26. 26.Spiteri M, Cook D, Clarke S. Reliability of eliciting physical signs in examination of the chest. Lancet 1988;1(8590):873–875.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1016/S0140-6736(88)91613-3&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=2895374&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=A1988M921600013&link_type=ISI) 

27. 27.Correger E, Murias G, Chacon E, Estruga A, Sales B, Lopez-Aguilar J, et al. [Interpretation of ventilator curves in patients with acute respiratory failure]. Med Intensiva 2012;36(4):294–306.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=22014424&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

28. 28.Jubran A, Tobin MJ. Use of flow-volume curves in detecting secretions in ventilator-dependent patients. Am J Respir Crit Care Med 1994;150(3):766–769.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1164/ajrccm.150.3.8087350&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=8087350&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=A1994PH35200027&link_type=ISI) 

29. 29.Guglielminotti J, Alzieu M, Maury E, Guidet B, Offenstadt G. Bedside detection of retained tracheobronchial secretions in patients receiving mechanical ventilation: is it time for tracheal suctioning? Chest 2000;118(4):1095–1099.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1378/chest.118.4.1095&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=11035683&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

30. 30.Sole ML, Bennett M, Ashworth S. Clinical indicators for endotracheal suctioning in adult patients receiving mechanical ventilation. Am J Crit Care 2015;24(4):318–324.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpjYyI7czo1OiJyZXNpZCI7czo4OiIyNC80LzMxOCI7czo0OiJhdG9tIjtzOjI1OiIvcmVzcGNhcmUvNjEvMTAvMTM3NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

31. 31.Paratz J, Ntoumenopoulos G. Detection of secretion retention in the ventilated patient. Curr Respir Med Rev 2014;10(3):151–157.
    
    

32. 32.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159–174.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.2307/2529310&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=843571&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=A1977CY39700012&link_type=ISI) 

33. 33.Ntoumenopoulos G, Glickman Y. Computerised lung sound monitoring to assess effectiveness of chest physiotherapy and secretion removal: a feasibility study. Physiotherapy 2012;98(3):250–255.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=22898583&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

34. 34.Schreur HJ, Sterk PJ, Vanderschoot J, van Klink HC, van Vollenhoven E, Dijkman JH. Lung sound intensity in patients with emphysema and in normal subjects at standardised airflows. Thorax 1992;47(9):674–679.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToidGhvcmF4am5sIjtzOjU6InJlc2lkIjtzOjg6IjQ3LzkvNjc0IjtzOjQ6ImF0b20iO3M6MjU6Ii9yZXNwY2FyZS82MS8xMC8xMzc0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

35. 35.Vena A, Perchiazzi G, Giuliani R, Fiore T, Hedenstierna G. Acoustic effects of positive end-expiratory pressure on normal lung sounds in mechanically ventilated pigs. Clin Physiol Funct Imaging 2006;26(1):45–53.
    
    [CrossRef](http://rc.rcjournal.com/lookup/external-ref?access_num=10.1111/j.1475-097X.2005.00652.x&link_type=DOI) 
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=16398670&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 

36. 36.Hunt RJ. Percent agreement, Pearson's correlation, and kappa as measures of inter-examiner reliability J Dent Res 1986;65(2):128–130.
    
    [Abstract/FREE Full Text](http://rc.rcjournal.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BqZHIiO3M6NToicmVzaWQiO3M6ODoiNjUvMi8xMjgiO3M6NDoiYXRvbSI7czoyNToiL3Jlc3BjYXJlLzYxLzEwLzEzNzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

37. 37.Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med 2005;37(5):360–363.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=15883903&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=000229008800015&link_type=ISI) 

38. 38.Ntoumenopoulos G. Editorial from Guest Editor (Thematic Issue: Secretion Retention in the Adult Intubated and Mechanically Ventilated Patient: The Definition, Detection, Prevention, Aggravation, Patient Outcomes, Interventions and Future Directions). Curr Respir Med Rev 2014;10(3):141–142.
    
    

39. 39.Mangione S, Nieman LZ. Pulmonary auscultatory skills during training in internal medicine and family practice. Am J Respir Crit Care Med 1999;159(4 Pt 1):1119–1124.
    
    [PubMed](http://rc.rcjournal.com/lookup/external-ref?access_num=10194155&link_type=MED&atom=%2Frespcare%2F61%2F10%2F1374.atom) 
    
    [Web of Science](http://rc.rcjournal.com/lookup/external-ref?access_num=000079638400015&link_type=ISI)