Abstract
BACKGROUND: With an increasing number of follow-up studies of acute respiratory failure survivors, there is need for a better understanding of participant retention and its reporting in this field of research. Hence, our objective was to synthesize participant retention data and associated reporting for this field.
METHODS: Two screeners independently searched for acute respiratory failure survivorship studies within a published scoping review to evaluate subject outcomes after hospital discharge in critical illness survivors.
RESULTS: There were 21 acute respiratory failure studies (n = 4,342 survivors) over 47 follow-up time points. Six-month follow-up (range: 2–60 months) was the most frequently reported time point, in 81% of studies. Only 1 study (5%) reported accounting for loss to follow-up in sample-size calculation. Retention rates could not be calculated for 5 (24%) studies. In 16 studies reporting on retention across all time points, retention ranged from 32% to 100%. Pooled retention rates at 3, 6, 12, and 24 months were 85%, 89%, 82%, and 88%, respectively. Retention rates did not significantly differ by publication year, participant mean age, or when comparing earlier (3 months) versus each later follow-up time point (6, 12, or 24 months).
CONCLUSIONS: Participant retention was generally high but varied greatly across individual studies and time points, with 24% of studies reporting inadequate data to calculate retention rate. High participant retention is possible, but resources for optimizing retention may help studies retain participants. Improved reporting guidelines with greater adherence would be beneficial.
Introduction
The number of patients surviving acute respiratory failure (ARF) is increasing with advances in critical care medicine.1,2 These survivors often experience long-term health impairments.3–7 Consequently, there has been an increasing number of follow-up studies focused on survivors' functional outcomes after hospital discharge.8
In studies completing longitudinal follow-up, it is important to assess as many participants of those eligible for follow-up (eg, alive) at each time point (ie, high retention rates). Achieving high participant retention rates is important for preserving statistical power and maintaining internal validity and generalizability.9 There is growing interest and associated publications related to identifying effective participant-retention strategies;10–12 however, there are little data specifically evaluating participant retention rates and related reporting of such data in longitudinal studies evaluating ARF survivors. The objective of this systematic review and meta-analysis is to synthesize participant retention data and its reporting in studies of outcomes after hospital discharge for survivors of ARF.
QUICK LOOK
Current knowledge
With more patients surviving acute respiratory failure, there is a growing number of follow-up studies focused on patient outcomes after hospital discharge. There are little data on participant retention and associated reporting in longitudinal studies of acute respiratory failure survivors.
What this paper contributes to our knowledge
Via a systematic review and meta-analysis of the acute respiratory failure literature, pooled participant retention rates at 3, 6, 12, and 24 month follow-up were > 80%; however, across all studies, participant retention rates were highly variable. Moreover, for a substantial proportion of studies, participant retention rates could not be calculated from the reported data. Also, there was variation in the nature of participant retention data reported. Standardizing collection and reporting of participant retention in longitudinal studies would help advance this field of research.
Methods
The publications included in this analysis were obtained from a previously completed comprehensive scoping review of outcomes measurement other than mortality after hospital discharge in ICU survivors.8 This scoping review searched 5 databases (PubMed, EMBASE, PsycINFO, CINAHL, and the Cochrane Controlled Trials Registry from 1970 to 2013, without language restrictions. The scoping review8 identified a total of 425 eligible papers that served as the population of studies to evaluate for eligibility for inclusion in this analysis.
From the 425 papers included in the scoping review,8 2 independent trained researchers (VR, ZW) screened for studies that specifically evaluated subjects with ARF. The researchers were not blinded to author or journal details. Studies were excluded if: (1) non-ARF subjects were included in study (ie, outside the focus of this report); (2) the ARF population was exclusively focused on neuromuscular disease or chronic pulmonary disease such as COPD (ie, a specific population that may not be generalizable to all ARF patients); or (3) there was only a single follow-up time point with consent occurring at the same time (ie, no prospective follow-up performed to evaluate participant retention).
Data abstraction was performed independently, in duplicate, by pairs of trained researchers (AA, VR, RN, ZW). Data abstractors were not blinded to author or journal details. Conflicts between reviewers were resolved, by consensus, in consultation with a senior researcher (DMN or VDD). Authors were contacted for additional data when necessary. For studies that had > 1 paper reporting on the same time point, we used the paper with the higher participant count (eg, preliminary analyses vs completed study). The following data were collected: participant retention rates at each follow-up time point; utilization of participant flow chart; modes of data collection (eg, in-person, telephone, mail); reporting of mortality during follow-up; blinding of assessors (if interventional study); accounting for loss to follow-up in sample size or power calculation; study exclusion criteria related to barriers to follow-up (eg, homelessness); and a description of participant-retention strategies. We also identified studies that focused solely on subjects with ARDS.
Risk of bias for randomized controlled trials (RCTs) was assessed using the Cochrane Risk of Bias methodology.13 For observational studies, the Newcastle-Ottawa Scale was used.14 We adapted the Newcastle-Ottawa Scale to omit the following criteria that were not applicable to this systematic review, given its focus on participant retention rates rather than a specific clinical outcome: demonstration that the outcome was not present at enrollment; assessment of the outcome; follow-up that was long enough for the outcome to occur.
Statistical Analysis
The pooled average participant retention rate was calculated as part of this meta-analysis. Follow-up time points among eligible studies were 0.5, 2, 3, 6, 12, 24, 36, 48, and 60 months. We did not include the 0.5-month (2-week) time point. However, the one 2-month study was pooled with the 3-month time point. Follow-up time points > 24 months could not be pooled due to only having one study at each time point. For studies with age reported only as median (interquartile range), the mean (SD) were estimated using established methods.15 Treatment groups or participant subgroups separately reported within a study for the same time point were tested for a statistically significant difference using the Fisher's exact test and combined when the test resulted in a nonsignificant difference (P < .05). For studies where retention rates were 100%, the Haldane-Anscombe correction was used to calculate the variance of the observed retention rate and presentation of confidence intervals.16,17
Two different approaches were used to calculate retention rates. The primary approach calculated the retention rate as the number of participants who had a study assessment (numerator) divided by the total number of participants who were eligible for follow-up at that same time point (denominator), excluding participants who died by that time point. The alternative approach excluded from the denominator participants who died and those who permanently discontinued participation in the study or were withdrawn from the study. Retention rates were not calculated if all requisite data were not reported or if mortality was combined with lost-to-follow-up data. The results using the primary approach are reported in this manuscript; the results using the alternative approach are available online (see the supplementary materials at http://www.rcjournal.com).
A linear random effects regression model for the log odds of the retention rate (logit transformation) was used to estimate the population average log odds of the retention rate for each follow-up. The model included a random intercept for each study, a set of indicators for follow-up times, and the observed variance within the study follow-up time was used as the study-specific known residual variance. The estimated average retention rate was computed by applying the inverse-logit transformation. To determine if the average retention rate varied as a function of participant (average age and percent male) or study publication characteristics, each characteristic was separately added to the model described above as a fixed effect.
Statistical heterogeneity among included studies was evaluated using the I2 statistic (with > 50% deemed to be substantial heterogeneity).18 The I2 statistic was calculated for each follow-up when there were > 2 studies reporting data.19 SAS 9.4 (SAS Institute, Cary, North Carolina) was used for all analyses. We followed the PRISMA checklist in reporting this manuscript.20 The protocol was registered with PROSPERO (CRD42018087835).
Results
The literature search completed for the scoping review found 20,189 citations, with 15,464 unique titles and abstracts reviewed, of which 1,207 full-text articles were reviewed, yielding 425 eligible papers that reported on post-hospital functional outcomes in critical illness survivors.8 From these 425 papers, a total of 39 publications reporting on 21 unique studies (Fig. 1) met the eligibility criteria for our systematic review focused on ARF survivors; of these 21 unique studies, 16 (76%) focused exclusively on ARDS survivors. Among these 21 studies, 6 (29%) were RCTs21–32 and 15 (71%) were cohort studies (Table 1). There were a total of 4,342 ARF subjects in the included studies (the 16 ARDS studies reported on 2,271 subjects with ARDS). Thirteen (62%) of the 21 studies were conducted in United States (Table 1). The most frequent time points for follow-up was 6 months, evaluated in 17 (81%) studies, and 12 months, evaluated in 13 (62%) studies; the earliest and latest time points were 2 weeks (1 study) and 60 months (1 study).
Flow chart. ARF = acute respiratory failure.
Study Cohort Characteristics
Risk of Bias Assessment
Among the 6 RCTs, randomization and allocation concealment was rated as a low risk of bias in all the studies. Blinding of outcome assessments, addressing incomplete outcome data, and selective reporting were adequate in 5 of the 6 RCTs. In observational cohort studies, 13 of 15 studies (87%) had low risk for representativeness of the exposed cohort, whereas 11 (73%) had adequate follow-up (see the supplementary materials at http://www.rcjournal.com).
Retention-Related Reporting
Eleven (52%) of 21 studies reported exclusion criteria related to inability to follow participants after hospital discharge, and 9 of 21 (43%) reported reasons for lost to follow-up at each time point (Table 2). Ten of 21 (48%) studies included flow charts that explicitly reported retention rates for each time point. Strategies to maximize participant retention were described in 13 of the 21 (62%) studies. Only 8 of 21 (38%) studies reported both retention strategies and retention rates,3,22,23,33–45 with 6 of these 8 studies reporting retention rates > 80% at all follow-up time points. Of 8 studies reporting participant retention rates without reporting on cohort-retention strategies, only 4 (50%) had retention rates > 80%. Three of the 21 (14%) studies reported a sample size or power calculation, of which only 1 specified a primary follow-up time point. Only 1 (5%) study reported consideration of loss to follow-up in calculating sample size or statistical power (Table 2).
Participant Retention-Related Data in Longitudinal Studies of Acute Respiratory Failure Survivors
Participant Retention
Participant retention rates could not be calculated for 5 of the 21 (24%) studies, representing 12 of the 47 (26%) time points. In 3 of the remaining 16 eligible studies, retention rates could not be calculated for 4 of the 35 (11%) time points. Among the 6 RCTs, we were able to calculate participant retention for 5 (83%) studies. Based on the available data from 31 time points in 16 studies, participant retention ranged from 32% to 100%, with a median (interquartile range) of 90% (74–95%). Participant retention was > 89% across available time points in 4 of the 5 RCTs reporting data.
Pooled Results
The pooled participant retention rate for each time point was 85% (95% CI 57–96%, I2 44%) at 3 months, 89% (95% CI 72–96%, I2 95%) at 6 months, 82% (95% CI 61–93%, I2 97%) at 12 months, and 88% (95% CI 44–99%, I2 75%) at 24 months (Fig. 2). As part of post hoc analyses to explore potential contributors to high heterogeneity, we pooled retention rates by following subgroups: ARDS studies, studies that conducted only in-person visits, and studies that used different modes of visits instead of or in addition to in-person visits. The I2 statistic did not qualitatively improve in these subgroups.
Pooled average retention rates in acute respiratory failure survivor follow-up studies. Retention rates were calculated as the number of participants assessed at each follow-up time point divided by the number presumed alive at that time point (this included participants who withdrew just prior to the time point). Points denote pooled average retention rates, and bars represent 95% CI.
There was no difference in retention rates across follow-up time points when comparing early (3 months) versus each later time point: 6 months (P = .61), 12 months (P = .68), and 24 months (P = .72). No qualitative differences in the studies could explain the substantial statistical heterogeneity observed at the 6-, 12-, and 24-month follow-up time points. The proportion of male participants in each study was associated with a decreased retention rate; the odds ratio for a 1% increase in male participants was 0.92 (95% CI 0.87–0.97, P = .006). Participant age and publication year were not significantly associated with retention rates. These results do not qualitatively differ when using the alternative definition for calculating retention rates, as described above. The pooled average retention rates using the second definition is in the supplementary materials (see the supplementary materials at http://www.rcjournal.com).
Discussion
In this systematic review and meta-analysis evaluating participant retention in 21 studies of ARF survivors' outcomes after hospital discharge, participant retention rates varied widely. Although the median retention rate (90%) was high, participant retention could not be calculated for 24% of studies. There was wide variability in reporting of important issues related to participant retention, such as reasons for loss to follow-up and use of strategies to maximize participant retention.
For studies in which retention rates could be calculated, pooled retention rates were high at the 3-, 6-, 12-, and 24-month time points (82–89%), but there was high variability among individual studies, with a range of 32–100% across all studies and follow-up time points. Researchers and stakeholders must be aware of potential limitations in the validity of studies with low participant retention. Participant age, publication year, and follow-up time point were not associated with retention rates. However, a greater proportion of males in studies was associated with reduced retention, which is noteworthy for evaluating any sex-specific findings from studies.
Although most studies reported some form of “lost to follow-up” data, the reporting was inconsistent and the type of data reported varied widely. For example, we were unable to calculate retention rates in 24% of studies because they either reported incomplete or no lost to follow-up data, or they combined lost to follow-up rates with mortality. Additionally, there was incomplete reporting of sample size calculations and other methodological issues relating to participant retention (eg, retention strategies utilized, participant exclusions).
Reporting guidelines do exist for RCTs and cohort studies. The CONSORT checklist for RCTs was first published in 1996,46 updated in 2001,47 and most recently revised in 2010.48 Since inception, the CONSORT checklist recommended utilization of a participant flow chart, with it being “strongly recommended” as of the 2001 update. The CONSORT checklist did not explicitly include reporting rates and reasons for participant loss and post-randomization exclusion until 2010, although the example flow chart in all 3 CONSORT versions mention “lost to follow-up.” For cohort studies, the STROBE checklist was published in 2007 and recommends reporting reasons for nonparticipation at each time point and consideration of a flow chart.49 In this systematic review, the 5 studies for which we could not calculate retention rates were all published prior to 2007 (one RCT published in 2006, 4 cohort studies published in 1994, 2000/2004, 2001, and 2006). Further improvement is needed to standardize reporting participant retention data and related issues, along with greater adherence to existing guidelines. A consensus process, including relevant stakeholders, may be useful for standardizing collection and reporting of data elements related to participant retention for all studies. Thereafter, journals may require more complete reporting of participant retention according to such consensus recommendations.
To assist with improving participant retention, there is a growing body of relevant publications and resources. A recent update to an earlier systematic review10,11 demonstrated a large increase in the number of publications related to participant-retention strategies. A total of 618 participant-retention strategies across 12 different themes, compiled from the most recent systematic review,11 are freely available as an online searchable database (https://www.improvelto.com/sysrevstrategies, Accessed February 19, 2020) to assist with understanding best practices in the field. Moreover, as part of a NIH-funded national research infrastructure project (R24HL111895), additional practical cohort-retention tools are freely available (https://www.improvelto.com/cohort-retention-tools, Accessed February 19, 2020) with a goal of providing additional research resources to assist investigators in this area.
Strengths and Limitations
To our knowledge, no prior systematic review has reported on cohort-retention rates and related methodology in ARF survivorship studies. Despite this strength, there are also potential limitations of this analysis. In an effort to reduce heterogeneity between studies and to optimize feasibility of this synthesis, we exclusively focused on studies of ARF survivors. However, this specific focus may result in limitations to precision and generalizability of these results. Despite this restriction to ARF studies, there remained substantial heterogeneity in pooled participant-retention rates beyond the 3-month follow-up time point, thus caution is advised when interpreting these results. Furthermore, studies published after 2013 were not included in this analysis because 2013 was the end date for the scoping review upon which this analysis was based. However, in comparing retention results over time (from 1970 to 2013), there was no temporal trend in retention, which may attenuate this potential limitation.
Conclusions
In this systematic review and meta-analysis evaluating 21 studies of ARF survivors at 3-, 6-, 12-, and 24-month follow-up time points, pooled participant retention rates were ≥ 82%. However, retention rates were highly variable across individual studies (range: 32–100%) and could not be calculated for nearly a quarter of the studies, with substantial differences in reporting of methodological issues related to participant retention. In addition to greater adherence to existing reporting standards for RCTs and cohort studies, additional reporting recommendations related to participant retention may be beneficial. Moreover, use of existing resources and best practices for optimizing participant retention data may benefit some acute respiratory survivorship studies.
Footnotes
- Correspondence: Victor D Dinglas MPH, Pulmonary and Critical Care Medicine, Johns Hopkins University, 1830 E Monument St, 5th Floor, Baltimore, MD 21205. E-mail: victor.dinglas{at}jhmi.edu
This project was supported through a grant from the National Heart, Lung and Blood Institute (R24HL111895). The authors have disclosed no conflicts of interest.
Supplementary material related to this paper is available at http://www.rcjournal.com.
- Copyright © 2020 by Daedalus Enterprises
References
