Original articles
Health Services Utilization Reporting in Respiratory Patients

https://doi.org/10.1016/S0895-4356(98)00117-6Get rights and content

Abstract

Health services utilization information is important for outcomes research. This study assessed the reliability of self-reports of health services utilization in respiratory patients. Patients reported health services use and other information during three telephone interviews over 6 months. Reports of visits to general practitioners (GPs), specialists, emergency room (ER)/clinics, and hospital admissions were compared with corresponding fee service claims in the Ontario Health Insurance Plan administrative database in 83 subjects. Agreement between the two sources was calculated using observed agreement and estimated kappa. Substantial agreement was found for hospital admissions and visits to respiratory specialists. Agreement was moderate for GP visits and slight for ER/clinic visits. Patient self-report of ER use appeared unreliable and may be related to imprecise questionnaire wording and inadequately defined fee service codes. The findings emphasize the importance of the methods used to assess the reliability of patient self-reports used in outcomes research.

Introduction

Outcomes research and health economic evaluations require detailed measurement of health service use. Whereas data on the use of some health services may be retrieved from administrative databases such as provincial claims files [1], these data may be incomplete or the level of accuracy may not be known 2, 3. To judge the appropriateness of the use of health care interventions and services, information on sociodemographics, health status, and behavioral and lifestyle information may be necessary. These and other explanatory variables, such as disease severity, may also be needed for comparative economic evaluations and technology assessment 4, 5. Researchers frequently rely on information obtained directly from health care consumers to ensure that the data are comprehensive. Before the results of evaluations relying on patient self-reports are used for clinical decision making, policy formulation, and resource allocation, the quality of the data collection methods must be scrutinized. This includes assessments of the validity and reliability of patient self-reports.

Asthma is a disease that has attracted growing public concern owing to evidence of increasing prevalence 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. The changing epidemiology of respiratory disease and its consequences for public health policy have created a demand for studies that monitor health outcomes and economic end points. A large prospective study that evaluated the economic burden of asthma was conducted using information from self-reports of respiratory patients [16]. As part of the project, a study was conducted to measure the level of agreement for several categories of health service utilization (physician visits, emergency room [ER]/clinic visits, and hospitalizations) between patient self-reports and the provincial health service administrative database. This report describes the results of the agreement study and focuses on issues pertaining to the measurement of agreement that have widespread relevance for outcomes research and economic evaluations across diverse patient populations.

Section snippets

Patient Sample

Subjects included in the study were drawn from participants in the Pharmacy Medication Monitoring Program (PMMP) Bronchial Inhalers pilot project. The PMMP is a pharmacoepidemiologic surveillance project established in 1992 by McMaster University. The primary purpose of the PMMP is to collect longitudinal data on health outcomes, quality of life, and patterns of use of various categories of prescription medications in a diverse outpatient population. This program is characterized by the use of

Demographics

Complete baseline information was available for all 83 consenting subjects. Seventy-eight subjects completed the 3-month interview and 76 completed the 6-month interview, resulting in a drop-out rate of 8% over the 6-month follow-up period. The baseline demographics of patients consenting to participate were compared with those declining, to detect the presence of selection bias. The results are displayed in Table 1.

Owing to the small group sizes, there was insufficient power to assess the

Discussion

The proportions of observed agreement (Po) were strong for all variables except respiratory-related GP visits. Observed agreement was slightly better for on-study visits compared to prestudy visits. Five of seven health services utilization variables displayed estimated values of kappa that were moderate or better.

Somewhat surprising was the finding of slight agreement between the self-report and OHIP data for the occurrence of on-study ER/clinic visits (estimated kappa = 0.04). Given the

Conclusion

Hospital admissions, one of the most costly components of respiratory care [48], was the most reliably reported. Further work is required to improve the interview questionnaire and the specification of GP and ER/clinic visits so that the patient self-reports of these health services may be considered reliable data sources for epidemiological and health economic evaluations.

The assessment of self-reports for measuring the use of health services is an important component of efforts to improve the

Acknowledgements

We thank Dr. Linda MacKeigan and Dr. Ken Chapman for their insightful contributions throughout this project. The assistance of Ms. Terry Stevens with the OHIP database is gratefully acknowledged. Dr. Ungar was supported by a Medical Research Council of Canada/Pharmaceutical Manufacturers Association of Canada studentship cosponsored by Ciba Canada Inc., an Ontario Respiratory Care Society Fellowship, and a grant-in-aid from Glaxo-Wellcome Canada Inc. Dr. Coyte is supported by grants from the

References (48)

  • M.T. McKenna et al.

    Agreement between patient self-reports and medical records for Pap smear histories

    Am J Prev Med

    (1992)
  • R.I. Horwitz

    Comparison of epidemiologic data from multiple sources

    J Chron Dis

    (1986)
  • S. Green et al.

    The collection of service utilisation dataa research note on validity

    Soc Sci Med

    (1979)
  • L.L. Roos et al.

    Registries and administrative dataorganization and accuracy

    Med Care

    (1993)
  • E.S. Fisher et al.

    The accuracy of Medicare’s hospital claims dataProgress has been made, but problems remain

    Am J Public Health

    (1992)
  • L.L. Roos et al.

    Using administrative data for longitudinal researchComparisons with primary data collection

    J Chron Dis

    (1987)
  • J.M. Eisenberg et al.

    Pharmacoeconomicseconomic evaluation of pharmaceuticals

  • E.J. Power et al.

    Technology assessment and public health

    Ann Rev Public Health

    (1994)
  • H.R. Anderson

    Epidemiology of asthma

    Br J Hosp Med

    (1992)
  • P. Burney

    Epidemiology of asthma

    Allergy

    (1993)
  • M.L. Burr

    Epidemiology of asthma

    Monogr Allergy

    (1993)
  • P.J. Gergen et al.

    The increasing problem of asthma in the United States

    Am Rev Respir Dis

    (1992)
  • R. Jackson et al.

    International trends in asthma mortality1970–1985

    Chest

    (1988)
  • E.R. McFadden et al.

    Medical progressAsthma

    N Engl J Med

    (1992)

    • Cited by (0)

    †

    E. Keith Borden, Kathryn A. Gaebel, Charles H. Goldsmith, Xiaohan Hu, Mitchell A. H. Levine, Jeffrey W. Poston, and Donald J. Willison.

    View full text
    Copyright © 1998 Elsevier Science Inc. All rights reserved.