Abstract
BACKGROUND: Although randomized controlled trials (RCTs) are considered the highest level of evidence, they are also subject to bias, due to a lack of adequately reported randomization, and therefore the reporting should be as explicit as possible for readers to determine the significance of the contents. We evaluated the methodological quality of RCTs in respiratory research in high ranking clinical journals, published in 2010.
METHODS: We assessed the methodological quality, including generation of the allocation sequence, allocation concealment, double-blinding, sample-size calculation, intention-to-treat analysis, flow diagrams, number of medical centers involved, diseases, funding sources, types of interventions, trial registration, number of times the papers have been cited, journal impact factor, journal type, and journal endorsement of the CONSORT (Consolidated Standards of Reporting Trials) rules, in RCTs published in 12 top ranking clinical respiratory journals and 5 top ranking general medical journals.
RESULTS: We included 176 trials, of which 93 (53%) reported adequate generation of the allocation sequence, 66 (38%) reported adequate allocation concealment, 79 (45%) were double-blind, 123 (70%) reported adequate sample-size calculation, 88 (50%) reported intention-to-treat analysis, and 122 (69%) included a flow diagram. Multivariate logistic regression analysis revealed that journal impact factor ≥ 5 was the only variable that significantly influenced adequate allocation sequence generation. Trial registration and journal impact factor ≥ 5 significantly influenced adequate allocation concealment. Medical interventions, trial registration, and journal endorsement of the CONSORT statement influenced adequate double-blinding. Publication in one of the general medical journal influenced adequate sample-size calculation.
CONCLUSIONS: The methodological quality of RCTs in respiratory research needs improvement. Stricter enforcement of the CONSORT statement should enhance the quality of RCTs.
- randomized controlled trial
- RCT
- methodological reporting
- CONSORT
- quality
- respiratory
- impact factor Background
Introduction
In recent years the concept of evidence-based medicine has been widely adopted because it claims to provide a method for determining the safety and efficacy of medical therapies and public health interventions more generally. The practice of evidence-based medicine means integrating individual clinical expertise with the best currently available scientific evidence from systematic research.1 Ideally, randomized controlled trials (RCTs) are the highest level and best evidence to assess the value of a particular therapy. However, RCTs are not always unbiased; it has been demonstrated that inadequacy of certain important methodological items, such as allocation concealment, are associated with an exaggeration of the estimated treatment effect.2 Such bias can lead to serious consequences for patient care and decision making. Furthermore, for clinicians it is often difficult to appraise the quality of RCTs and to judge whether the reliability is strong enough to support the effectiveness of therapies. This is the reason why quality assessment of RCTs is important. To maintain a high standard and the quality of RCTs, the Consolidated Standards of Reporting Trials (CONSORT) statement was published in 1996 and updated in 2001 and 2010; it was subsequently adopted by many high ranking journals.3,4
Methodological assessments of RCTs with the CONSORT statement have been performed in various specialties, and the results show that the CONSORT statement has improved the quality of RCT reporting overall.5–7 In the field of respiratory medicine only one study has evaluated RCT quality. Bausch et al assessed the quality of 344 RCTs on COPD, published between 1957 and 2006. They found appropriate generation of randomization in 27.0% of the RCTs, appropriate concealment of random allocation in 11.6%, and intention-to-treat analysis in 21.8%. The quality of COPD trials improved after the CONSORT statement.8 There have been no similar reports on other respiratory research. Therefore, we systematically evaluated respiratory RCTs reported in 17 high ranking respiratory and general medical journals in 2010, and we identified factors that influence methodological quality.
Methods
Selection of Journals and Randomized Controlled Trials
We manually reviewed all articles on clinical respiratory research, published in 2010, in 12 top ranking clinical respiratory journals (American Journal of Respiratory and Critical Care Medicine, Thorax, Chest, The European Respiratory Journal, Journal of Thoracic Oncology, Lung Cancer, Respiratory Research, Critical Care Medicine, Intensive Care Medicine, Respiratory Medicine, Respiratory Care, Lung) and 5 top ranking general medical journals (The New England Journal of Medicine, Lancet, JAMA, Annals of Internal Medicine, and British Medical Journal). We reviewed 372 issues of the 17 journals. We excluded the supplementary issues. Each issue was assessed by 2 of us independently, and all eligible RCTs were identified for evaluation. We excluded trials published in non-respiratory research, trials published as brief reports or correspondence, trials dealing with subgroups of subjects from RCTs or secondary analysis of previously published RCTs, trials including animals or normal subjects, and trials without outcomes of randomized subjects.
Assessment of Randomized Controlled Trials
Methodological quality was critically evaluated by 2 of us independently, according to the following elements described in the Cochrane Handbook for Systematic Reviews of Interventions:9
Sequence generation: classified as adequate (eg, computer generated random numbers) or unclear (not described)
Allocation concealment: classified as adequate (eg, central independent unit, sealed envelopes), unclear (not described), or inadequate (eg, open table of random numbers)
Double-blinding status: classified as adequate (eg, participants, caregivers, outcome assessors, and analysts unaware of treatment allocation by identical placebo), unclear (not described), inadequate (eg, tablets vs injections), or not double-blinded
Sample-size calculation: classified as clearly reported or not clearly reported (did not specify the power and sample size)
Trials conducted at more than one medical center or clinic were considered multi-center research trials. A result was considered positive when one intervention was significantly better than the other, and negative when there was no significant difference. A trial was considered industry funded if any portion was funded by a company, and was considered publicly funded if supported by a government or nonprofit source. Trials that acknowledged the use of industry-provided active drugs, controls, or placebos were considered non-industry-funded. The region/country of the trial was where the corresponding author was from. If a trial registration number was found in the publication, the trial was considered registered. The number of times the report has been cited was extracted from the ISI web site (http://thomsonreuters.com/web-of-knowledge).
According to the CONSORT Web site, 10 journals (American Journal of Respiratory and Critical Care Medicine, Thorax, Chest, Respiratory Research, Intensive Care Medicine, The New England Journal of Medicine, Lancet, JAMA, Annals of Internal Medicine, and British Medical Journal) had endorsed the CONSORT rules before 2011, whereas 7 journals (Critical Care Medicine, Respiratory Medicine, Respiratory Care, Lung, The European Respiratory Journal, Journal of Thoracic Oncology, and Lung Cancer) had not.
The inter-observer agreement of the 2 authors was calculated as a kappa value. Disagreements were resolved with the corresponding author (Dr Shen).
Statistical Analysis
Descriptive statistics were employed to summarize the selected findings. The number and proportion of articles that reported each of the methodological factors were calculated, and the differences were assessed by a chi-square test. The factors that influenced methodological quality were determined by multivariate logistic regression analysis. A stepwise forward method was used, and the entry and removal values were 0.05 and 0.10, respectively. The factors included in the multivariate logistic regression analysis were number of medical centers involved, sample size, type of funding source, type of intervention, registration status, number of times the paper has been cited, journal impact factor, journal type, and journal CONSORT endorsement status. All statistical analyses were performed with statistics software (SPSS 16.0, SPSS, Chicago, Illinois), and a P value < .05 was considered statistically significant.
Results
Characteristics of the Randomized Controlled Trials
We retrieved 3,190 articles published in the 17 journals in 2010 (Figure). Among these 3,190 studies, 640 were classified as publications based on RCTs, and 464 studies were subsequently excluded from analysis. The reasons for exclusion were: not a respiratory trial (420), subgroup or cohort study nested within an RCT (27), secondary or pooled analysis (10), and included animals or normal subjects (7).
Flow chart of trial selection.
Table 1 summarizes the 176 included trials. Generally, there were more trials with positive results (66%) than with negative results (34%). There were 83 (47%) trials from Europe, 54 (31%) from the United States and Canada (North America), and 19 (11%) from Asia. There was considerable variation in the sample size (range 10 to 8,424 subjects, 25th percentile 41 subjects, 50th percentile 118 subjects, 75th percentile 289 subjects). The mean sample size was 473.10,11
Principal Characteristics of the Included Trials
The included trials were on 14 kinds of disease: COPD (61), lung cancer (31), asthma (31), obstructive sleep apnea (10), respiratory-tract infection (13), pulmonary fibrosis (6), acute lung injury or ARDS (7), pulmonary tuberculosis (3), respiratory failure (5), cystic pulmonary fibrosis (3), pulmonary thromboembolism (2), chronic cough (2), bronchopulmonary dysplasia (1), and emphysema (1).
Assessment of Key Methodological Items Reported
The kappa values for inter-observer agreement were 0.79 for generation of allocation sequence, 0.77 for allocation concealment, 0.85 for double-blinding, and 0.86 for sample-size calculation, which indicate good agreement.
Generation of the allocation sequence was adequate in 93 (53%) of the 176 trials, and unclear in the other 83 trials. The allocation concealment was adequate in 66 (38%) trials, and unclear or inadequate in 110 trials. Adequate double-blinding was reported in 79 (45%) trials, and was unclear or not double-blinded in 97 trials. Of the trials without adequate double-blinding, 39% (38/97) dealt with medication. Of the 103 medical trials, 63% (65/103) were double-blinded. Adequate sample-size calculation and intention-to-treat analysis were reported in 123 (70%) trials and 88 (50%) trials, respectively. Eligibility criteria were reported adequately in 176 (100%) trials. Flow diagrams were provided in 69% (122/176) of these trials.
In COPD research, adequate allocation sequence generation was reported in 48% (29/61) of the trials, adequate allocation concealment in 36% (22/61), adequate double-blinding in 56% (34/61), adequate sample-size calculation in 66% (40/61), intention-to-treat analysis in 39% (24/61), and flow diagram in 77% (47/61).
Univariate analysis revealed that registered trials had better quality in adequate allocation sequence generation (P = .048), adequate allocation concealment (P = .001), and adequate double-blinding (P = .001) than non-registered trials. Trials cited ≥ 10 times in the last 2 years had better quality in adequate allocation concealment (P = .01) and adequate sample-size calculation (P = .03) than those cited < 10 times. Trials published in journals with impact factor ≥ 5 had better quality in adequate allocation sequence generation (P = .008) and adequate allocation concealment (P = .001) than trials published in journals with impact factor < 5. The trials in the general medical journals had better quality in adequate allocation sequence generation (P = .03), adequate allocation concealment (P = .007), and adequate sample-size calculation (P = .02) than the trials in the respiratory journals. The trials performed in multiple centers (P = .01), supported by industry (P = .01), and with medical interventions (P = .001) had better quality in adequate double-blinding (Table 2).
Methodological Quality of Randomized Controlled Trials in Respiratory Research
Multivariate logistic regression analysis revealed that impact factor ≥ 5 (P = .009, odds ratio 2.26) was the one factor that influenced adequate allocation sequence generation. Registration (P = .006, odds ratio 2.91) and impact factor ≥ 5 (P = .001, odds ratio 3.7) were the factors that influenced adequate allocation concealment. Medical intervention (P = .001, odds ratio 2.52), registration (P = .02, odds ratio 2.57), and journal endorsement of the CONSORT rules (P = .003, odds ratio 3.73) were the factors that influenced adequate double-blinding. Publication in one of the general medical journals (P = .042, odds ratio 8.34) was the one factor that influenced adequate sample-size calculation.
Discussion
Many key RCT methods were not used or not reported. Adequate allocation concealment, adequate double-blinding, and intention-to-treat analysis were reported in only 50% of the trials. These findings suggest that there is much room for improvement in respiratory RCTs. However, improvement was found in the COPD RCTs, in which there was adequate allocation sequence generation, adequate allocation concealment, and intention-to-treat analysis in 27.0%, 11.6%, and 21.8%, respectively.8 Registered trials, trials cited ≥ 10 times, trials in journals with impact factor ≥ 5, and trials in the general medical journals were more likely to be of adequate methodological quality.
Important aspects of RCT methodology, such as randomization methods, allocation concealment, blinding, and sample-size calculation, are commonly inadequate in RCTs.12 That was the reason we conducted the analysis: to find the related facts on these methodological items. We decided to include trials published in leading journals in clinical respiratory research or general medical research, and the methodological reporting of respiratory RCTs had not been systematically studied. The reason why we decided to study the trials published in 2010 was that they could reflect the current quality status of respiratory RCTs.
A well designed and conducted RCT provides the highest level of evidence for clinicians to practice evidence-based medicine, as it gives investigators the most awesome factors that could confound the causal relationship between the intervention and health outcomes. However, like any other study design, an RCT is subject to bias due to lack of adequate randomization. Flaws in the randomization methods are associated with 30% overestimated intervention benefits, and trials without double-blinding may exaggerate the benefits by 14%.13–15 Sample-size calculation is also an important methodological item, as underpowered studies are prone to type II errors, which render the results inconclusive, and trials with excessively large sample sizes can be a waste of resources.16 In addition, the flow diagram is also important in demonstrating the completeness of recruitment, follow-up, and outcome assessment.17 The CONSORT statement was developed and revised to improve RCTs and guarantee their quality.3 To date, the CONSORT statement has been endorsed by over 50% of the core medical journals.4
The present study found a significant association between RCT quality and a high impact factor, as did Bausch et al.8 Kuroki et al found that the proportion of high-quality RCTs in high-impact-factor journals was nearly 3 times that in journals with moderate impact factor.18 In the present study we found that the proportion of high-quality RCTs in the high-impact-factor journals was nearly 1.5 times that in the journals with impact factor < 5. There are several potential explanations. First, the statistical reporting requirements in the high-impact-factor journals may be better listed and described in the instructions for authors. Second, the editing criteria of the editors, reviewers, and statistical consultants may be stricter.18 The trials in the general medical journals, which have higher impact factors, had better methods than those in the respiratory journals. The overall trial registration rate was 64%, and trial registration affected the adequacy of allocation, allocation concealment, and double-blinding.
The strengths of the present study are that the included trials covered a broad range of respiratory research in 12 top ranking respiratory journals and 5 top ranking general medical journals, and the included trials were published after the CONSORT statement. In addition, 2 reviewers independently assessed the methods.
There are some limitations to the present study. One was that we assessed RCTs published only during 2010. Although we thought the included RCTs should reflect the current quality of respiratory research, and there was little reason to suspect that the methodological quality would be different in 2010 than in other years, our study design disallowed assessment of changes or trends in methodological quality.
Another limitation was that we assessed only 12 respiratory journals and 5 general medical journals, and selected those journals based on their impact factors, so we did not cover the other respiratory research literature, the quality of which might be different from the journals we studied. Most important, certain therapies cannot be blinded, yet studying them according to the methodological reporting is not invalid, and the methodological quality assessment is also affected by the quality of the reporting. A well designed and conducted trial may appear to have substantial bias if it is inadequately reported, whereas a trial with severe methodological limitations may be considered excellent if the limitations are not reported.19 The difference between “not conducted” and “not reported” should be noted by readers.
Conclusions
The methods and methods reporting of RCTs in respiratory research were not adequate in some trials, and there is considerable room for improvement. RCTs in higher-impact-factor journals, and registered trials were more likely to have appropriate and/or adequately reported randomization. To make RCTs a reliable source of evidence for the evaluation of intervention efficacy, enforcement of the CONSORT statement by editors and reviewers is necessary.
Footnotes
- Correspondence: Ce Shen MD PhD, Department of Respiratory Medicine, Shanghai Sixth People's Hospital, Shanghai Jiaotong University, 600 Yi Shan Road, Shanghai, 200233, China. E-mail: shence6{at}126.com.
The authors have disclosed no conflicts of interest.
- Copyright © 2013 by Daedalus Enterprises