Review ArticleReviews

Methodological Reporting of Randomized Clinical Trials in Respiratory Research in 2010

Yi Lu, Qiuju Yao, Jie Gu and Ce Shen
Respiratory Care September 2013, 58 (9) 1546-1551; DOI: https://doi.org/10.4187/respcare.01877

Abstract

BACKGROUND: Although randomized controlled trials (RCTs) are considered the highest level of evidence, they are also subject to bias, due to a lack of adequately reported randomization, and therefore the reporting should be as explicit as possible for readers to determine the significance of the contents. We evaluated the methodological quality of RCTs in respiratory research in high ranking clinical journals, published in 2010.

METHODS: We assessed the methodological quality, including generation of the allocation sequence, allocation concealment, double-blinding, sample-size calculation, intention-to-treat analysis, flow diagrams, number of medical centers involved, diseases, funding sources, types of interventions, trial registration, number of times the papers have been cited, journal impact factor, journal type, and journal endorsement of the CONSORT (Consolidated Standards of Reporting Trials) rules, in RCTs published in 12 top ranking clinical respiratory journals and 5 top ranking general medical journals.

RESULTS: We included 176 trials, of which 93 (53%) reported adequate generation of the allocation sequence, 66 (38%) reported adequate allocation concealment, 79 (45%) were double-blind, 123 (70%) reported adequate sample-size calculation, 88 (50%) reported intention-to-treat analysis, and 122 (69%) included a flow diagram. Multivariate logistic regression analysis revealed that journal impact factor ≥ 5 was the only variable that significantly influenced adequate allocation sequence generation. Trial registration and journal impact factor ≥ 5 significantly influenced adequate allocation concealment. Medical interventions, trial registration, and journal endorsement of the CONSORT statement influenced adequate double-blinding. Publication in one of the general medical journal influenced adequate sample-size calculation.

CONCLUSIONS: The methodological quality of RCTs in respiratory research needs improvement. Stricter enforcement of the CONSORT statement should enhance the quality of RCTs.

Introduction

In recent years the concept of evidence-based medicine has been widely adopted because it claims to provide a method for determining the safety and efficacy of medical therapies and public health interventions more generally. The practice of evidence-based medicine means integrating individual clinical expertise with the best currently available scientific evidence from systematic research.1 Ideally, randomized controlled trials (RCTs) are the highest level and best evidence to assess the value of a particular therapy. However, RCTs are not always unbiased; it has been demonstrated that inadequacy of certain important methodological items, such as allocation concealment, are associated with an exaggeration of the estimated treatment effect.2 Such bias can lead to serious consequences for patient care and decision making. Furthermore, for clinicians it is often difficult to appraise the quality of RCTs and to judge whether the reliability is strong enough to support the effectiveness of therapies. This is the reason why quality assessment of RCTs is important. To maintain a high standard and the quality of RCTs, the Consolidated Standards of Reporting Trials (CONSORT) statement was published in 1996 and updated in 2001 and 2010; it was subsequently adopted by many high ranking journals.3,4

Methodological assessments of RCTs with the CONSORT statement have been performed in various specialties, and the results show that the CONSORT statement has improved the quality of RCT reporting overall.5–7 In the field of respiratory medicine only one study has evaluated RCT quality. Bausch et al assessed the quality of 344 RCTs on COPD, published between 1957 and 2006. They found appropriate generation of randomization in 27.0% of the RCTs, appropriate concealment of random allocation in 11.6%, and intention-to-treat analysis in 21.8%. The quality of COPD trials improved after the CONSORT statement.8 There have been no similar reports on other respiratory research. Therefore, we systematically evaluated respiratory RCTs reported in 17 high ranking respiratory and general medical journals in 2010, and we identified factors that influence methodological quality.

Methods

Selection of Journals and Randomized Controlled Trials

We manually reviewed all articles on clinical respiratory research, published in 2010, in 12 top ranking clinical respiratory journals (American Journal of Respiratory and Critical Care Medicine, Thorax, Chest, The European Respiratory Journal, Journal of Thoracic Oncology, Lung Cancer, Respiratory Research, Critical Care Medicine, Intensive Care Medicine, Respiratory Medicine, Respiratory Care, Lung) and 5 top ranking general medical journals (The New England Journal of Medicine, Lancet, JAMA, Annals of Internal Medicine, and British Medical Journal). We reviewed 372 issues of the 17 journals. We excluded the supplementary issues. Each issue was assessed by 2 of us independently, and all eligible RCTs were identified for evaluation. We excluded trials published in non-respiratory research, trials published as brief reports or correspondence, trials dealing with subgroups of subjects from RCTs or secondary analysis of previously published RCTs, trials including animals or normal subjects, and trials without outcomes of randomized subjects.

Assessment of Randomized Controlled Trials

Methodological quality was critically evaluated by 2 of us independently, according to the following elements described in the Cochrane Handbook for Systematic Reviews of Interventions:9

  • Sequence generation: classified as adequate (eg, computer generated random numbers) or unclear (not described)

  • Allocation concealment: classified as adequate (eg, central independent unit, sealed envelopes), unclear (not described), or inadequate (eg, open table of random numbers)

  • Double-blinding status: classified as adequate (eg, participants, caregivers, outcome assessors, and analysts unaware of treatment allocation by identical placebo), unclear (not described), inadequate (eg, tablets vs injections), or not double-blinded

  • Sample-size calculation: classified as clearly reported or not clearly reported (did not specify the power and sample size)

Trials conducted at more than one medical center or clinic were considered multi-center research trials. A result was considered positive when one intervention was significantly better than the other, and negative when there was no significant difference. A trial was considered industry funded if any portion was funded by a company, and was considered publicly funded if supported by a government or nonprofit source. Trials that acknowledged the use of industry-provided active drugs, controls, or placebos were considered non-industry-funded. The region/country of the trial was where the corresponding author was from. If a trial registration number was found in the publication, the trial was considered registered. The number of times the report has been cited was extracted from the ISI web site (http://thomsonreuters.com/web-of-knowledge).

According to the CONSORT Web site, 10 journals (American Journal of Respiratory and Critical Care Medicine, Thorax, Chest, Respiratory Research, Intensive Care Medicine, The New England Journal of Medicine, Lancet, JAMA, Annals of Internal Medicine, and British Medical Journal) had endorsed the CONSORT rules before 2011, whereas 7 journals (Critical Care Medicine, Respiratory Medicine, Respiratory Care, Lung, The European Respiratory Journal, Journal of Thoracic Oncology, and Lung Cancer) had not.

The inter-observer agreement of the 2 authors was calculated as a kappa value. Disagreements were resolved with the corresponding author (Dr Shen).

Statistical Analysis

Descriptive statistics were employed to summarize the selected findings. The number and proportion of articles that reported each of the methodological factors were calculated, and the differences were assessed by a chi-square test. The factors that influenced methodological quality were determined by multivariate logistic regression analysis. A stepwise forward method was used, and the entry and removal values were 0.05 and 0.10, respectively. The factors included in the multivariate logistic regression analysis were number of medical centers involved, sample size, type of funding source, type of intervention, registration status, number of times the paper has been cited, journal impact factor, journal type, and journal CONSORT endorsement status. All statistical analyses were performed with statistics software (SPSS 16.0, SPSS, Chicago, Illinois), and a P value < .05 was considered statistically significant.

Results

Characteristics of the Randomized Controlled Trials

We retrieved 3,190 articles published in the 17 journals in 2010 (Figure). Among these 3,190 studies, 640 were classified as publications based on RCTs, and 464 studies were subsequently excluded from analysis. The reasons for exclusion were: not a respiratory trial (420), subgroup or cohort study nested within an RCT (27), secondary or pooled analysis (10), and included animals or normal subjects (7).

Figure.
Figure.

Flow chart of trial selection.

Table 1 summarizes the 176 included trials. Generally, there were more trials with positive results (66%) than with negative results (34%). There were 83 (47%) trials from Europe, 54 (31%) from the United States and Canada (North America), and 19 (11%) from Asia. There was considerable variation in the sample size (range 10 to 8,424 subjects, 25th percentile 41 subjects, 50th percentile 118 subjects, 75th percentile 289 subjects). The mean sample size was 473.10,11

View this table:
Table 1.

Principal Characteristics of the Included Trials

The included trials were on 14 kinds of disease: COPD (61), lung cancer (31), asthma (31), obstructive sleep apnea (10), respiratory-tract infection (13), pulmonary fibrosis (6), acute lung injury or ARDS (7), pulmonary tuberculosis (3), respiratory failure (5), cystic pulmonary fibrosis (3), pulmonary thromboembolism (2), chronic cough (2), bronchopulmonary dysplasia (1), and emphysema (1).

Assessment of Key Methodological Items Reported

The kappa values for inter-observer agreement were 0.79 for generation of allocation sequence, 0.77 for allocation concealment, 0.85 for double-blinding, and 0.86 for sample-size calculation, which indicate good agreement.

Generation of the allocation sequence was adequate in 93 (53%) of the 176 trials, and unclear in the other 83 trials. The allocation concealment was adequate in 66 (38%) trials, and unclear or inadequate in 110 trials. Adequate double-blinding was reported in 79 (45%) trials, and was unclear or not double-blinded in 97 trials. Of the trials without adequate double-blinding, 39% (38/97) dealt with medication. Of the 103 medical trials, 63% (65/103) were double-blinded. Adequate sample-size calculation and intention-to-treat analysis were reported in 123 (70%) trials and 88 (50%) trials, respectively. Eligibility criteria were reported adequately in 176 (100%) trials. Flow diagrams were provided in 69% (122/176) of these trials.

In COPD research, adequate allocation sequence generation was reported in 48% (29/61) of the trials, adequate allocation concealment in 36% (22/61), adequate double-blinding in 56% (34/61), adequate sample-size calculation in 66% (40/61), intention-to-treat analysis in 39% (24/61), and flow diagram in 77% (47/61).

Univariate analysis revealed that registered trials had better quality in adequate allocation sequence generation (P = .048), adequate allocation concealment (P = .001), and adequate double-blinding (P = .001) than non-registered trials. Trials cited ≥ 10 times in the last 2 years had better quality in adequate allocation concealment (P = .01) and adequate sample-size calculation (P = .03) than those cited < 10 times. Trials published in journals with impact factor ≥ 5 had better quality in adequate allocation sequence generation (P = .008) and adequate allocation concealment (P = .001) than trials published in journals with impact factor < 5. The trials in the general medical journals had better quality in adequate allocation sequence generation (P = .03), adequate allocation concealment (P = .007), and adequate sample-size calculation (P = .02) than the trials in the respiratory journals. The trials performed in multiple centers (P = .01), supported by industry (P = .01), and with medical interventions (P = .001) had better quality in adequate double-blinding (Table 2).

View this table:
Table 2.

Methodological Quality of Randomized Controlled Trials in Respiratory Research

Multivariate logistic regression analysis revealed that impact factor ≥ 5 (P = .009, odds ratio 2.26) was the one factor that influenced adequate allocation sequence generation. Registration (P = .006, odds ratio 2.91) and impact factor ≥ 5 (P = .001, odds ratio 3.7) were the factors that influenced adequate allocation concealment. Medical intervention (P = .001, odds ratio 2.52), registration (P = .02, odds ratio 2.57), and journal endorsement of the CONSORT rules (P = .003, odds ratio 3.73) were the factors that influenced adequate double-blinding. Publication in one of the general medical journals (P = .042, odds ratio 8.34) was the one factor that influenced adequate sample-size calculation.

Discussion

Many key RCT methods were not used or not reported. Adequate allocation concealment, adequate double-blinding, and intention-to-treat analysis were reported in only 50% of the trials. These findings suggest that there is much room for improvement in respiratory RCTs. However, improvement was found in the COPD RCTs, in which there was adequate allocation sequence generation, adequate allocation concealment, and intention-to-treat analysis in 27.0%, 11.6%, and 21.8%, respectively.8 Registered trials, trials cited ≥ 10 times, trials in journals with impact factor ≥ 5, and trials in the general medical journals were more likely to be of adequate methodological quality.

Important aspects of RCT methodology, such as randomization methods, allocation concealment, blinding, and sample-size calculation, are commonly inadequate in RCTs.12 That was the reason we conducted the analysis: to find the related facts on these methodological items. We decided to include trials published in leading journals in clinical respiratory research or general medical research, and the methodological reporting of respiratory RCTs had not been systematically studied. The reason why we decided to study the trials published in 2010 was that they could reflect the current quality status of respiratory RCTs.

A well designed and conducted RCT provides the highest level of evidence for clinicians to practice evidence-based medicine, as it gives investigators the most awesome factors that could confound the causal relationship between the intervention and health outcomes. However, like any other study design, an RCT is subject to bias due to lack of adequate randomization. Flaws in the randomization methods are associated with 30% overestimated intervention benefits, and trials without double-blinding may exaggerate the benefits by 14%.13–15 Sample-size calculation is also an important methodological item, as underpowered studies are prone to type II errors, which render the results inconclusive, and trials with excessively large sample sizes can be a waste of resources.16 In addition, the flow diagram is also important in demonstrating the completeness of recruitment, follow-up, and outcome assessment.17 The CONSORT statement was developed and revised to improve RCTs and guarantee their quality.3 To date, the CONSORT statement has been endorsed by over 50% of the core medical journals.4

The present study found a significant association between RCT quality and a high impact factor, as did Bausch et al.8 Kuroki et al found that the proportion of high-quality RCTs in high-impact-factor journals was nearly 3 times that in journals with moderate impact factor.18 In the present study we found that the proportion of high-quality RCTs in the high-impact-factor journals was nearly 1.5 times that in the journals with impact factor < 5. There are several potential explanations. First, the statistical reporting requirements in the high-impact-factor journals may be better listed and described in the instructions for authors. Second, the editing criteria of the editors, reviewers, and statistical consultants may be stricter.18 The trials in the general medical journals, which have higher impact factors, had better methods than those in the respiratory journals. The overall trial registration rate was 64%, and trial registration affected the adequacy of allocation, allocation concealment, and double-blinding.

The strengths of the present study are that the included trials covered a broad range of respiratory research in 12 top ranking respiratory journals and 5 top ranking general medical journals, and the included trials were published after the CONSORT statement. In addition, 2 reviewers independently assessed the methods.

There are some limitations to the present study. One was that we assessed RCTs published only during 2010. Although we thought the included RCTs should reflect the current quality of respiratory research, and there was little reason to suspect that the methodological quality would be different in 2010 than in other years, our study design disallowed assessment of changes or trends in methodological quality.

Another limitation was that we assessed only 12 respiratory journals and 5 general medical journals, and selected those journals based on their impact factors, so we did not cover the other respiratory research literature, the quality of which might be different from the journals we studied. Most important, certain therapies cannot be blinded, yet studying them according to the methodological reporting is not invalid, and the methodological quality assessment is also affected by the quality of the reporting. A well designed and conducted trial may appear to have substantial bias if it is inadequately reported, whereas a trial with severe methodological limitations may be considered excellent if the limitations are not reported.19 The difference between “not conducted” and “not reported” should be noted by readers.

Conclusions

The methods and methods reporting of RCTs in respiratory research were not adequate in some trials, and there is considerable room for improvement. RCTs in higher-impact-factor journals, and registered trials were more likely to have appropriate and/or adequately reported randomization. To make RCTs a reliable source of evidence for the evaluation of intervention efficacy, enforcement of the CONSORT statement by editors and reviewers is necessary.

Footnotes

  • Correspondence: Ce Shen MD PhD, Department of Respiratory Medicine, Shanghai Sixth People's Hospital, Shanghai Jiaotong University, 600 Yi Shan Road, Shanghai, 200233, China. E-mail: shence6{at}126.com.

  • The authors have disclosed no conflicts of interest.

References

  1. 1.↵
    1. Sackett DL,
    2. Rosenberg WM,
    3. Gray JA,
    4. Haynes RB,
    5. Richardson WS
    . Evidence based medicine: what it is and what it isn't. BMJ 1996;312(7023):71-72.
    OpenUrlFREE Full Text
  2. 2.↵
    1. Jüni P,
    2. Altman DG,
    3. Egger M
    . Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001;323(7303):42-46.
    OpenUrlFREE Full Text
  3. 3.↵
    1. Moher D,
    2. Schulz KF,
    3. Altman DG
    . The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357(9263):1191-1194.
    OpenUrlCrossRefPubMed
  4. 4.↵
  5. 5.↵
    1. Bai Y,
    2. Gao J,
    3. Zou DW,
    4. Li ZS
    . Methodological reporting of randomized clinical trials in major gastroenterology and hepatology journals in 2006. Hepatology 2009;49(6):2108-2112.
    OpenUrlPubMed
  6. 6.
    1. Lai TY,
    2. Wong VW,
    3. Lam RF,
    4. Cheng AC,
    5. Lam DS,
    6. Leung GM
    . Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol 2007;14(6):390-398.
    OpenUrlCrossRefPubMed
  7. 7.↵
    1. Danilla S,
    2. Wasiak J,
    3. Searle S,
    4. Arriagada C,
    5. Pedreros C,
    6. Cleland H,
    7. Spinks A
    . Methodological quality of randomised controlled trials in burns care. A systematic review. Burns 2009;35(7):956-961.
    OpenUrlCrossRefPubMed
  8. 8.↵
    1. Bausch B,
    2. Spaar A,
    3. Kleijnen J,
    4. Puhan MA
    . Quality of randomised trials in COPD. Eur Respir J 2009;34(5):1060-1065.
    OpenUrlAbstract/FREE Full Text
  9. 9.↵
    1. Higgins JPT,
    2. Green S
    , editors. Cochrane handbook for systematic reviews of interventions, version 5.1.0. Updated May 2011. The Cochrane Collaboration. http://handbook.cochrane.org. Accessed June 24, 2013.
  10. 10.↵
    1. Yeh GY,
    2. Roberts DH,
    3. Wayne PM,
    4. Davis RB,
    5. Quilty MT,
    6. Phillips RS
    . Tai chi exercise for patients with chronic obstructive pulmonary disease: a pilot study. Respir Care 2010;55(11):1475-1482.
    OpenUrlAbstract/FREE Full Text
  11. 11.↵
    1. Aubier M,
    2. Buhl R,
    3. Ekström T,
    4. Ostinelli J,
    5. van Schayck CP,
    6. Selroos O,
    7. Haughney J
    . Comparison of two twice-daily doses of budesonide/formoterol maintenance and reliever therapy. Eur Respir J 2010;36(3):524-530.
    OpenUrlAbstract/FREE Full Text
  12. 12.↵
    1. Hewitt C,
    2. Hahn S,
    3. Torgerson DJ,
    4. Watson J,
    5. Bland JM
    . Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ 2005;330(7499):1057-1058.
    OpenUrlFREE Full Text
  13. 13.↵
    1. Schulz KF,
    2. Chalmers I,
    3. Hayes RJ,
    4. Altman DG
    . Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-412.
    OpenUrlCrossRefPubMed
  14. 14.
    1. Moher D,
    2. Pham B,
    3. Jones A,
    4. Cook DJ,
    5. Jadad AR,
    6. Moher M,
    7. Tugwell P,
    8. Klassen TP
    . Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352(9128):609-613.
    OpenUrlCrossRefPubMed
  15. 15.↵
    1. Kjaergard LL,
    2. Villumsen J,
    3. Gluud C
    . Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135(11):982-989.
    OpenUrlCrossRefPubMed
  16. 16.↵
    1. Freiman JA,
    2. Chalmers TC,
    3. Smith H Jr.,
    4. Kuebler RR
    . The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 “negative” trials. N Engl J Med 1978;299(13):690-694.
    OpenUrlCrossRefPubMed
  17. 17.↵
    1. Egger M,
    2. Jüni P,
    3. Bartlett C
    ; CONSORT Group (Consolidated Standards of Reporting of Trials). Value of flow diagrams in reports of randomized controlled trials. JAMA 2001;285(15):1996-1999.
    OpenUrlCrossRefPubMed
  18. 18.↵
    1. Kuroki LM,
    2. Allsworth JE,
    3. Peipert JF
    . Methodology and analytic techniques used in clinical research: associations with journal impact factor. Obstet Gynecol 2009;114(4):877-884.
    OpenUrlPubMed
  19. 19.↵
    1. Colditz GA,
    2. Miller JN,
    3. Mosteller F
    . How study design affects outcomes in comparisons of therapy. I: medical. Stat Med 1989;8(4):441-454.
    OpenUrlPubMed
Back to top

In this issue

 

Print
Download PDF
Article Alerts
Email Article
Citation Tools

Jump to section

Related Articles

Cited By...

Keywords