Abstract
BACKGROUND: While the number of reports of randomized controlled trials in physical therapy has increased substantially in the last decades, the quality and reporting of randomized trials have never been systematically investigated in the subdiscipline of cardiothoracic physical therapy. The primary aim was to determine the methodological quality and completeness of reporting of cardiothoracic physical therapy trials. Secondary aims were to investigate the range of clinical conditions investigated in these trials and the degree of association between trial characteristics and quality.
METHODS: All reports of randomized trials indexed on the Physiotherapy Evidence Database (PEDro) and coded as being relevant to cardiothoracic physical therapy were surveyed. PEDro scale individual items and total score were downloaded, and some characteristics included in the Consolidated Standards of Reporting Trials (CONSORT) statement were extracted for each trial report.
RESULTS: The mean ± SD total PEDro score for the 2,970 included reports of cardiothoracic trials was 4.7 ± 1.4, with 27% being of moderate to high quality. The clinical conditions studied included chronic lung diseases (32% of the trials), cardiac diseases (20%), cardiovascular surgical conditions (5%), sleep disorders (5%), peripheral vascular disease (4%), acute lung disease (4%), critical illness (3%), and other surgical conditions (3%). The multivariate linear regression analysis revealed that endorsement of the CONSORT statement by the publishing journal, time since publication, evidence of trial registration, sources of funding, description of the sample size calculation, and identification of the primary outcome(s) had associations with the total PEDro score.
CONCLUSIONS: There is great potential to improve the quality of the conduct and reporting of trials evaluating the effects of cardiothoracic physical therapy.
Introduction
While the number of reports of randomized controlled trials in physical therapy has increased substantially in the last decades, there has been very slow improvement in the methodological quality and statistical reporting of these trials over this period.1 The quality and reporting of trials are moderate, on average, ranging from poor to excellent.2 This wide variability in trial quality poses a barrier to physical therapists hoping to interpret and use high-quality research to guide the teaching and practice of physical therapy.
Two resources that can be used to quantify the conduct and reporting of physical therapy trials are the Physiotherapy Evidence Database (PEDro) scale3 and the Consolidated Standards of Reporting Trials (CONSORT) statement.4 The PEDro resource (http://www.pedro.org.au) was launched in 1999 to facilitate and optimize the search for high-quality research in physical therapy. All trial reports indexed in PEDro are rated for methodological quality and statistical reporting, using the 11-item PEDro scale.3 These 11 items are: eligibility criteria and source of subjects; random allocation; concealed allocation; baseline comparability; blinding of subjects, therapists, and assessors; adequate follow-up; intention-to-treat analysis; between-group statistical comparisons; and reporting of point measures and measures of variability. The last 10 items are used to calculate the total PEDro score, which is determined as the sum of items met. (Item 1 is not included in the total PEDro score because it is related to generalizability rather than methodological quality and statistical reporting.) The PEDro scale has satisfactory measurement properties (ie, reproducibility and validity).5–7
The CONSORT statement is a set of recommendations developed to improve the reporting of trials. The CONSORT statement has 25 items related to internal and external validity.4 Some of the CONSORT items related to transparent reporting are not included in the PEDro scale (eg, sample size, statement of primary and secondary outcomes, trial registration). Therefore, the PEDro scale and CONSORT statement could be used together to describe the quality and reporting of physical therapy trials.
Methodological quality and statistical reporting have been investigated in 2 physical therapy subdisciplines: neurology8 and sports.9 Based on 238 trial reports, the median total PEDro score for neurological physical therapy trials was 5/10.8 Sports physical therapy trials scored an average of 4/10 points on the PEDro scale.9 These results demonstrate that, although the number of neurology8 and sports9 trials is high, there is still room for improvement in the methodological quality and statistical reporting of trials in these areas. The methodological quality and statistical reporting of trials in cardiothoracic physical therapy do not appear to have been systematically investigated.
A number of trial characteristics appear to be associated with higher quality of trial reports in medicine and in physical therapy in general, including larger sample size,10 having industry funding,11 prospective registration of the trial,12 being published in English,13 and being published more recently.1,13 A survey of the trials in cardiothoracic physical therapy could be used to describe the overall quality of trials as well as to determine whether some trial characteristics are associated with higher quality.
The primary aim of this study was to describe the methodological quality and statistical reporting of trials in cardiothoracic physical therapy. Our secondary aims were to analyze the range of clinical conditions and interventions investigated in this research and to identify any characteristics of the trial reports that are associated with methodological quality and with completeness of reporting.
QUICK LOOK
Current knowledge
Randomized controlled trials in physical therapy have increased in number, but their methodological quality and statistical reporting have not improved substantially. Poor trial quality poses a barrier to caregivers hoping to use high-quality research to guide the teaching and practice of physical therapy.
What this paper contributes to our knowledge
The proportion of clinical trials in cardiothoracic physical therapy that met the standards for methodological quality and statistical reporting was low. Lack of randomization, reporting of sample-size determination, and participant flow diagrams are common shortcomings.
Methods
Selection of Trial Reports
We used the PEDro database to identify trials in cardiothoracic physical therapy, because PEDro is one of the most comprehensive databases for indexing of reports of trials of physical therapy interventions.14,15 All trial reports indexed on PEDro are coded for the subdiscipline of physical therapy and assessed for methodological quality and statistical reporting using the PEDro scale. Each trial report is evaluated by 2 independent raters and, in the case of disagreement between the raters for any item, arbitration is provided by a third rater. On March 7, 2012, all trial reports that fulfilled the following criteria were downloaded from PEDro:
Coded as being relevant to cardiothoracic physical therapy
Complete PEDro scale scoring (ie, no in-process records)
No language restrictions were applied.
Data Extraction
The data downloaded from the PEDro database were: PEDro scale (11 individual items plus the total PEDro score), the language of publication, the intervention investigated, and the citation details (including the year of publication and the journal name). The year of publication was subtracted from 2011 to calculate the number of years since publication. A full-text copy of each included trial report was obtained to identify the disease or health condition being treated, and the presence of 9 items from the CONSORT statement.
To determine whether the journals in which the trials were published endorsed the recommendations of the CONSORT statement, we accessed each journal's Web site and examined the instructions to authors section. The trial reports published in journals that followed the CONSORT recommendations were classified as “yes” (coded as 1), and those that did not were classified as “no” (coded as 0). Trial reports published before the creation of the CONSORT statement in August 1996 were classified as “not applicable” (also coded as 0).
While the CONSORT statement4 consists of 25 items, only 9 items were extracted in our survey. These 9 items were features not included in the PEDro scale. These items are detailed in the following list. We did not include CONSORT statement items related to the introduction (1 item) and discussion (3 items) sections. Also we did not extract data from items that are not related to methodological quality, such as the description of the interventions, objectives and hypothesis of the trial, implementation, and adverse events.
Identification as a randomized controlled trial in the title. This item was satisfied if the authors used the word “randomized” in the title. This feature helps to ensure that the trial report is appropriately indexed and easily identified by the readers. This item was classified as “yes” or “no.”
Number of randomized subjects. This refers to the number of subjects initially allocated to groups. A larger sample size indicates that the estimate of the treatment effect is likely to be more accurate and precise.
Description of sample size calculation. Authors should indicate how the intended sample size for the trial was determined. This item was satisfied if the author simply reported that the sample size calculation was performed prospectively (coded as 1 for “yes”). Retrospective calculations were not considered to fulfill this criterion. The number of subjects indicated by the sample calculation was also extracted.
Description of the country or countries in which the trial was carried out and whether it was a uni-center or multicenter trial. This information assists with determining the applicability and generalizability of the trial results. The country was extracted from the methods section of the report, and in the absence of this information it was assumed that the trial was carried out in country of the first author's address. Each trial was classified according to the world region where the research took place (ie, Europe, North America, Asia, Oceania, South America, Africa); if the trial took place in multiple regions, it was classified as “international.” In the case of multicenter trials, the number of centers was recorded. In the absence of information about the number of centers, we coded the trial as uni-center.
Primary outcomes. This refers to the number of pre-specified key outcomes (outcomes considered at the beginning of the trial to be of greatest importance to the relevant stakeholders). Trials may have more than one primary outcome. Other outcomes of interest are secondary outcomes. Outcomes explicitly identified as “primary,” “main,” “major,” or “key” were considered as primary outcomes. When primary outcome(s) were nominated, this item was classified as “yes” (coded as 1). The number of primary outcomes was also recorded.
Statistical adjustment for multiple primary outcomes. This adjustment is necessary to avoid a false positive result (type 1 error). These data were extracted from the statistical analysis section, and the following key words were used: “adjustment for primary outcomes,” “Bonferroni,” “Tukey,” or “Duncan.” This item was classified as “yes” (for trials that explicitly stated any type of statistical adjustment) or “no” (for trials that did not perform statistical adjustment or if the information was unclear).
Subject flow diagram. This illustrates, for each group, the number of subjects who were randomly assigned, who were excluded, and were analyzed for the primary outcome. We did not consider flow diagrams of the trial design alone to fulfill this criterion. This item was classified as “yes” (coded as 1) or “no” (coded as 0).
Trial registration. The trial needs to be registered in a public register in order to demonstrate that it has avoided selective outcome reporting bias. This item was classified as “yes” if the trial was registered (coded as 1) or “no” if there was no explicit evidence of registration in the published report (coded as 0).
Sources of funding. The trial can receive various types of funding. Trials that receive funding from scientific agencies are more likely to have better quality, as they are peer-reviewed in advance. In contrast, trials funded by private companies can incur a conflict of interest and thus may contain bias.16,17 This item was classified as “yes” if the trial received funding (coded as 1) or “no” if the trial did not receive funding or the funding status was unclear (coded as 0).
Because the PEDro scale data were extracted directly from the PEDro Web site, these data were generated by 2 trained independent raters with a final consensus provided by a third trained rater when necessary. A single investigator extracted the remaining data. Ten trial reports were used for training the data extractors, and data from these trials were included in the final sample of the present study.
Statistical Analysis
For the primary data analysis we used descriptive statistics to summarize the number of trial reports classified under the subdiscipline of cardiothoracic physical therapy, the language and year of publication, the total PEDro score, and the proportion of trials investigating particular interventions and conditions. We calculated the number of trials published in each decade and the average total PEDro score for each decade to determine whether the increase in the number of trials has been accompanied by improvement in quality. We also calculated the proportion of trials that met the criteria for each individual item in the PEDro scale and that adhered to each of the 9 CONSORT items.
To determine which trial characteristics predict better methodological quality and statistical reporting (ie, the secondary objective), we calculated a multivariate linear regression model. This type of analysis allows the control of other factors that might have been associated with the quality of the trial report (ie, potential confounding variables). Multivariate linear regression analysis was used to predict the total PEDro score (dependent variable), including the following terms in the regression equation (independent variables): endorsement of the CONSORT statement by the publishing journal (coded as 1) versus non-endorsement or publication before creation of the CONSORT statement (coded as 0); number of years since publication; registration (coded as 1) versus non-registration (coded as 0); funding (coded as 1) versus no funding or no information about funding (coded as 0); presence of a sample size calculation (coded as 1) versus no calculation (coded as 0); nomination of a primary outcome(s) (coded as 1) versus no primary outcome (coded as 0); number of subjects randomized; and uni-site (coded as 0) versus multi-site (coded as 1) trials. For the construction of the multivariate regression model we used univariate linear regression analyses between the dependent variable and each of the independent variables, and all variables with a P value ≤ .20 were selected for building the final multivariate model. This model was considered complete when all variables reached a P value < .05. All linearity and collinearity prerequisites were prospectively tested.
Results
On March 7, 2011, the PEDro database contained 2,970 reports of trials coded as being relevant to cardiothoracic physical therapy and with complete ratings using the PEDro scale. The trial reports were published in 598 different journals, in 9 different languages (predominantly English), and the total PEDro score ranged from a mean ± SD of 4.0 ± 1.6 out of 10 for reports written in Italian, to 5.3 ± 1.7 for reports written in French (Table 1). Overall, the mean total PEDro score was 4.7 ± 1.4, and ranged from 0 to 9 points. 27% of the trial reports were of moderate to high quality (ie, scoring 6 or more on the PEDro scale).
Trial Reports and Physiotherapy Evidence Database Scores
The first trial related to cardiothoracic physical therapy was published in 1952.18 Since then the number of reports has increased rapidly and exponentially. In the 1950s there were only 7 trials published. The number of trials rose to 15 in the 1960s, 97 in the 1970s, 366 in the 1980s, 849 in the 1990s, and 1,636 in 2000 to 2011. Over time, there was an increase in the total PEDro score, from a mean of 2.9 ± 1.7 out of 10 in the 1950s, to 5.0 ± 1.4 in 2000 to 2011 (Fig. 1).
Mean Physiotherapy Evidence Database scale scores by decade.
The proportion of trial reports in cardiothoracic physical therapy that met each item of the PEDro scale is illustrated in Figure 2. The most commonly satisfied items were: random allocation (96% of the trials), between-group comparisons (94%), and point estimates and measures of variability (89%). In contrast, only 19% of the trials used concealed allocation, 16% used intention-to-treat analysis, and the proportion of blinding in the trials was low.
Proportions of individual Physiotherapy Evidence Database scale items satisfied by 2,970 cardiothoracic trials.
A total of 953 trial reports (33%) were published in journals that endorsed the CONSORT statement, while 966 trials (34%) were published in journals that did not endorse these recommendations. The 1,051 remaining trials (33%) were published prior to the creation of the CONSORT statement.
On average, the cardiothoracic trials randomized 132 subjects (with a sample size projection of 185). Most were conducted in a single treatment center in Europe or North America. About half the trials received external funding. Only one fifth of trial reports used the term “randomized” in the title, described the sample size calculation, specified primary outcomes, or included a subject flow diagram. Very few trials adjusted their analyses for multiple primary outcomes or were registered. These data are listed in Table 2.
Characteristics of Trials That Adhered to 9 CONSORT Statement Items
Most of the interventions investigated were related to respiratory therapy (which included interventions such as manual techniques to promote airway clearance, breathing techniques, and the application of positive pressure to the airways), fitness training, and education (Table 3). The main diseases or conditions investigated in the 2,970 included cardiothoracic physical therapy trials were chronic pulmonary disease (32%), particularly COPD (n = 503, 17%) (Table 4). Heart disease was the second most prevalent condition investigated (20%). Subjects in intensive care or using invasive mechanical ventilation (classified as critical illness) and postoperative conditions were less frequent.
Main Interventions Investigated in the Trials
Diseases and Conditions Investigated in Cardiothoracic Physical Therapy Trials
The multivariate linear regression analysis revealed that endorsement of the CONSORT statement by the publishing journal, time since publication (number of years before 2011), evidence of trial registration, sources of funding, description of the sample size calculation, and identification of the primary outcome(s) had associations with the total PEDro score (Table 5).
Multivariate Model
A trial report published in a journal that endorsed the CONSORT statement was associated with a 0.26 point higher total PEDro score, compared to a trial that did not follow CONSORT recommendations. For every year in time since publication, the trials had a 0.03 point lower total PEDro score on average, compared to trials published in 2011. Registration, funding, description of the sample size calculation, and reporting of the primary outcomes were associated with a higher total PEDro score, by 0.47, 0.14, 0.67, and 0.44 points, respectively, compared to trials that did not have these characteristics.
Discussion
The primary objective of this study was to describe the methodological quality and completeness of statistical reporting of trials indexed in the PEDro database and classified in the subdiscipline of cardiothoracic physical therapy, using the items of the PEDro scale and some of the items of the CONSORT statement recommendations. The trial reports had a mean total PEDro score of 4.7 ± 1.4, indicating that, on average, trial reports were of low quality. Less than one quarter of trials used concealed allocation, blinding, and intention-to-treat analysis; used the term “randomized” in the title; included a sample size calculation; specified the primary outcome(s); and included a subject flow diagram. Very few adjusted their statistics for multiple primary outcomes or were registered. These results suggest that there is still great potential to improve the quality of the conduct and reporting of trials evaluating the effects of cardiothoracic physical therapy. Chronic pulmonary diseases, particularly COPD, were the conditions most commonly investigated in the cardiothoracic physical therapy trials we evaluated. Respiratory techniques were the most commonly investigated class of intervention in these trials.
The mean total PEDro score of 4.7 for the cardiothoracic physical therapy trials was slightly higher than the average of 4 for the sports physical therapy trials,9 and slightly lower than the median of 5 for the neurological physical therapy trials.8 Regarding the individual items on the PEDro scale, we found that the items concealed allocation, blinding, and intention-to-treat analysis were the least commonly satisfied, which is the same as in neurological8 and sports physical therapy trials.9 Although blinding in physical therapy trials is often difficult, blinding of subjects was achieved in over 200 of the reports of cardiothoracic trials. This suggests that it may be possible to blind for some interventions, so researchers should consider how blinding of subjects might be achieved, rather than assuming it is impossible to achieve. Where it is not possible, a convincing sham might be used to avoid some of the bias associated with lack of blinding. Blinding of therapists was also achieved in a small number of trials. Blinding of assessors was achieved in about 25% of trial reports. Blinding of assessors is usually technically easy, but it may be limited by lack of funds or resources. Concealed allocation and intention-to-treat analysis are possible in all trials.
We also found low adherence to several items recommended by the CONSORT statement, including trial registration in a public registry, and statistical adjustment for multiple primary outcomes. A recent study19 evaluating the quality of reporting of PubMed-indexed trial reports published in 2000 and 2006 found very similar results. While the quality of reporting had improved over time, it was still below an acceptable level in 2006 (eg, only 45% of trials included a sample size calculation). This suggests that, despite the promotion of the CONSORT statement over the last decade, a large proportion of authors, journal reviewers, and journal editors have still not implemented these recommendations.
Multivariate linear regression analysis showed that endorsement of the CONSORT statement by the journal publishing the trial report, time since publication, evidence of registration, evidence of funding, sample size calculation, and identification of the primary outcome(s) were associated with the total PEDro score. This suggests that recent, well reported and funded trials are more likely to have better quality, compared to trials without these characteristics. Our study confirms the data from a recent published Cochrane systematic review,20 which concluded that the quality of reporting of trials is highest in journals that endorse the CONSORT statement.
A strength of this study was the number of trial reports included in the analysis (n = 2,970). These trial reports were published in 9 languages, and were all trials coded as being relevant to cardiothoracic physical therapy indexed on the PEDro database, which is considered the most comprehensive for physical therapy trials.14,15 A limitation of this study is that the assessment of trial quality was based on information in the published report of each trial, rather than on the trial itself. Some authors may have used but not reported methods that satisfied a criterion on the PEDro scale or an item in the CONSORT statement. Therefore, our study may have underestimated the quality of some trials.
The results of this study suggest that if all journals that publish cardiothoracic physical therapy trials endorsed the CONSORT statement, the quality of reporting would improve substantially. Adherence to the CONSORT recommendations would benefit authors, reviewers, and readers of physical therapy journals. The authors would find it easier to write their trial reports because, by following the recommendations, they can be certain that all relevant aspects of the research design are included. Reviewers would be able to verify if all the information needed in the submitted trial reports has been included and judge the trial's methods more comprehensively. This information would guide decisions and suggestions made by the reviewers to improve the trial reports. From the reader's perspective, the interpretation and assessment of the results would be simplified and would facilitate decision-making in the practice and teaching of cardiothoracic physical therapy. Finally, health insurers and government agencies may have a greater interest in investing in treatments with a greater likelihood of success, given that these would be supported by trials with high methodological quality and transparent reporting.
The present study did not include systematic reviews or clinical practice guidelines in cardiothoracic physical therapy. Given the volume of scientific papers published each year, many physical therapists rely on systematic reviews and guidelines to keep abreast of high-quality research. Therefore, we suggest further studies to verify the methodological quality and reporting of these other publication types. We also suggest further investigation to verify the influence of the CONSORT statement on the improvement of the methodological quality and statistical reporting of studies in other areas of physical therapy, such as musculoskeletal or geriatrics.
Conclusions
There is great potential to improve the quality of the conduct and reporting of trials evaluating the effects of cardiothoracic physical therapy.
Footnotes
- Correspondence: Leonardo OP Costa PT PhD, Master's and Doctoral Programs in Physical Therapy, Universidade Cidade de São Paulo, Rua Cesário Galeno 448, São Paulo, 03071–000 Brazil. E-mail: lcos3060{at}gmail.com.
Drs Moseley, Elkins, and Costa have disclosed a relationship with the Physiotherapy Evidence Database. The other authors have disclosed no conflicts of interest.
- Copyright © 2013 by Daedalus Enterprises
