ELECTRONIC ARTICLE |

* MRC Health Services Research Collaboration, Department of Social Medicine
Department of Social Medicine, University of Bristol, Bristol, United Kingdom
| ABSTRACT |
|---|
|
|
|---|
Methods. A comprehensive search using computerized databases (including Medline, Embase, Econlit, and databases specific to economic evaluation), Web searches, and citation tracking was undertaken to identify cost-utility studies of interventions that were aimed at those who were younger than 16 years and published before April 2004. The methods of individual studies were compared with the recognized published guidelines of the US Panel on Cost-Effectiveness in Health and Medicine and the National Institute for Clinical Excellence in England and Wales, which recommend the use of a generic health state classification system (eg, Health Utility Index, EuroQol-5D), a choice-based valuation method (eg, standard gamble or time trade-off) and preferences of the general public in estimating QALYs. Studies therefore were categorized and evaluated according to the methods used to describe the health state, the valuation technique, and source of preferences.
Results. Fifty-four studies were reviewed, 34 (63%) of which were published in the past 5 years. A generic health status classification instrument was used in 22 (35%) cases; the remainder developed study-specific health state descriptions or elicited preferences directly from patients or proxies. In 3 (5%) cases, sources were unclear. Preference weights were elicited using choice-based techniques in 28 (42%) cases, either as tariffs for health status classification instruments (17 cases) or by directly valuing health state descriptions or patient health (11 cases). Preferences of the general public were used in only 23 (37%) cases. Four studies aggregated QALYs for mother/child or parents/child pairs without giving any theoretical justification. Although there was an increasing tendency for studies to use generic health status classification instruments, choice-based methods, and preferences of the general public, the majority of studies still did not adhere to these standard recommendations even in the period between January 2000 and March 2004. Despite increasing standardization in the methods advocated for economic evaluation over the past 10 years, there remains extensive variation in the actual methods used by researchers to calculate QALYs for children and adolescents. It is unclear whether these results suggest poor practice or a set of positive (or reactive) choices made by analysts in a methodologically uncertain area in which specific guidance is lacking regarding how to address the complexities of pediatric outcomes within the QALY framework. Many aspects of QALY measurement in children are not yet fully developed. In particular, there is (1) a lack of appropriate health state classification instruments that take account of the dynamics of child development, (2) a lack of health state classification instruments for use in children and infants who are younger than 5 years, and (3) the need to understand fully the role of proxies for measuring and valuing child health. Additional research efforts are also required to develop methods that account for the health benefits of parents or caregivers of the child and to consider the implications of combining different forms of utility measurement in childhood and adulthood.
Conclusions. Although variations from standard recommendations may be attributable to poor practice among researchers who are either unaware of these recommendations or choose not to follow them, they could equally be the result of attempts to make research more rigorous and more defensible than it might be if the standard recommendations were followed. There are 4 potential approaches to conducting cost-utility analysis in pediatric populations: (1) the explicit development of a generic instrument designed to be applicable across both child and adult populations (likely to be difficult in practice), (2) insistence on use of a generic instrument developed for adults, (3) the use of generic instruments specifically developed for children without being concerned about comparability with interventions aimed at adults, and (4) abandoning attempts to use single outcome measures that combine mortality with quality weights. In the absence of a clear way forward, it is suggested that an expert panel be convened to debate and further consider these potential solutions and recommendations for best practice and future research. In the interim, comparisons of the relative cost-effectiveness reported as cost per QALY gained across interventions for different diseases and populations should be treated with extreme caution.
Key Words: health policy health services research health status measurement
Abbreviations: QALY, quality-adjusted life-year EQ-5D, EuroQol-5D NICE, National Institute for Clinical Excellence
Over the last 2 decades, economic evaluation has increasingly been recognized as an important tool to assist decision makers in resource-constrained health care systems in deciding which interventions and programs represent good value for money and whether to fund or reimburse particular interventions or diagnostic tests. This has resulted in an increasing number of published studies that include economic appraisals of interventions and services that are aimed at children and adolescents.1 These studies are potentially influential in determining the extent of funding for particular pediatric interventions, and so their methodologic quality is extremely important.
Economic evaluation aims to compare interventions in terms of their costs and benefits. The main analytic techniques are cost-benefit analysis and cost-effectiveness analysis, with cost-utility analysis being one type of cost-effectiveness analysis (Table 1). 2 Whereas the measurement and valuation of costs is principally the same in all types of economic evaluations, the measurement and valuation of health benefits differs. Among all study types, cost-utility analysis, whereby health benefits are quantified in terms of quality-adjusted life-years (QALYs), has become the standard type and is now recommended in the great majority of health economics guidelines as the method of choice.3,4
|
To facilitate comparability between studies that are intended to inform resource allocation, the US Panel on Cost-Effectiveness in Health and Medicine defined a reference case as a standard set of methods considered to be most appropriate for cost-effectiveness analysis.4 With regard to the estimation of QALYs, the panel recommended the use of a generic health status classification system (eg, Health Utility Index, EuroQol-5D [EQ-5D]), a choice-based valuation technique (eg, standard gamble or time trade-off method), and preferences from a representative and fully informed sample of the general population. Recently, the National Institute for Clinical Excellence (NICE) in England and Wales also adopted these specific recommendations in its reference case.5
However, a number of reviews have highlighted extensive variability in the methods used in economic evaluations in general6 and cost-utility analyses in particular.7,8 Of particular concern here, the methods used for estimating QALYs have been shown to deviate from these recommendations with little improvement over time.9
None of these reviews, however, has explicitly appraised cost-utility studies of child health interventions and programs within their own context. The examination of the validity of these studies is an important research issue because of the added complexities in pediatric outcomes research10,11 currently not addressed in standard guidelines for economic evaluation3 and the recommendations of the US Panel on Cost-Effectiveness in Health and Medicine4 and NICE,5 although there is growing recognition that the differences between children and adults have methodologic implications for the design and use of these studies.11 One of the main obstacles to simply adapting adult measures for use in children stems from rapid developmental changes that take place in childhood and adolescence.10,1214 Outcome measures that not only are sensitive to changes in development and changes in health but also make allowances for different cognitive abilities of children at various ages with regard to reporting and valuing health status are required.10 Furthermore, children's dependence on parents and family members and the resulting interdependence of quality of life between them may suggest that the impact of child health interventions on parental utility should be incorporated into economic evaluations that are conducted from a societal perspective.11
The study reported here examines the methods used to calculate QALYs in cost-utility studies of pediatric health interventions published before April 2004 and compares them with the recommendations of the US Panel on Cost-Effectiveness in Health and Medicine and NICE. Additionally, it discusses some of the methodologic issues raised by these studies.
| METHODS |
|---|
|
|
|---|
|
Data Extraction and Criteria
Data were extracted by I.G. using a specifically designed data extraction form. The reliability of data extraction was monitored. Eleven (20%) studies were selected randomly and assessed independently by 2 authors (I.G. and J.C.). Agreement on each domain was compared qualitatively to identify inconsistencies and to ensure that forms were used in a standardized way.
Background information for each study was collected, including journal name and year of publication, country of study, description of intervention under investigation, condition (according to the International Classification of Diseases, 10th Revision classification), and prevention stage (primary, secondary, or tertiary). Furthermore, type of study, study perspective, the age of the population under investigation, the time horizon of the analysis, and the discount rate used to adjust for future health benefits were recorded for each study. To assess the methods used to calculate QALYs, we recorded the health status classification instrument used, the measurement technique used for valuing health states, the group of individuals whose quality of life was assessed (children, parents, both, or adults generally), and the group of individuals from whom valuations were obtained. When the study was based on a primary data collection (eg, randomized clinical trial), we examined which individual completed the instrument or valued the health state. Studies were grouped according to the health status classification system as in a previous review.9 Four groups were defined: generic (eg, instruments such as the EQ-5D, Health Utility Index, or Quality of Well-Being scale designed to be applicable across different conditions and interventions), study or disease specific (health states describing disease-specific outcomes, eg, diabetes), general disability (studies that described the health state in terms of general disability associated with the disease, eg, neurologic disability), and direct elicitation (study respondents valuing either their own health or hypothetical health states). When >1 method to derive preference weights was used or >1 group of individuals was considered, eg, in studies that modeled health benefits over a lifetime, papers were categorized in all relevant groups. Finally, we investigated whether studies performed sensitivity analyses to assess the uncertainty in the estimation of preference weights or resulting estimates of QALYs.
| RESULTS |
|---|
|
|
|---|
|
|
|
|
Four studies derived incremental cost-effectiveness ratios by summing up QALYs gained and costs incurred for child/mother or child/parent pairs. These studies evaluated interventions related to child births,23,24 antenatal care,25 and antenatal screening.26
Adherence to Standards Over Time
Although there is a tendency for studies increasingly to use generic health status classification instruments, choice-based methods, and preferences of the general public, the majority of studies still did not adhere to these standard recommendations even in the period 2000 to March 2004 (Table 6).
|
| DISCUSSION |
|---|
|
|
|---|
8 years. Because this is not the case, it is important to examine the reasons. Does this review suggest poor practice, with analysts ignoring sound and expert advice from internationally respected bodies? Or does it suggest a set of positive (or reactive) choices by analysts in a methodologically uncertain area in which the general advice for estimating QALYs seems, at best, limited and, at worst, impossible to apply in any rigorous manner? The remainder of the discussion focuses on this issue, considering the nature of QALY measurement and valuation for children and the issue of combining QALY measures. First, however, some limitations of the review are noted.
Limitations
This review selected only papers that were published in peer-reviewed journals, which may have led to a smaller overall number of retrieved studies than might have been the case if the gray literature had also been searched. A more comprehensive search, however, would have been unlikely to alter the findings reported here; indeed, if anything, it could be expected that incorporating research that is available only through nonpublished sources would have reduced further the reported quality of studies, given that such research has not been subject to the peer-review process.
The review was also limited in scope, focusing as it does only on the methods used for estimating QALYs. Undoubtedly, there are other important aspects of economic evaluation in the pediatric population that also merit review,11 but the focus here was limited deliberately so as to enable detailed examination of this particular topic.
An additional limitation concerns the assessment of reliability. Within the resources available, it was possible for only a small proportion of studies to be assessed by 2 reviewers. There thus was believed to be little value in providing a quantitative assessment of reliability; rather the process of 2 people's assessing each study was used to ensure a more reliable process throughout, through clarification of issues of contention. Nevertheless, a quantitative assessment of reliability would have been helpful in assessing the extent to which there may have been bias in the assessment of the studies by the main reviewer.
QALY Measurement and Valuation for Children
QALY measurement and valuation for children is intrinsically more difficult than that for adults for a number of reasons. First, because children undergo dramatic changes in growth and function (eg, mobility, self-care) at different rates, difficulties may arise to attribute improvements to health care interventions rather than to normal development. There is no methodologic guidance about how this confounding should or even might be dealt with. One solution that researchers in the studies reviewed here may have chosen to use is the development of their own health status scenarios that allow for changes from the age-related norm as a means of adjusting for this problem rather relying on instruments that have been developed for adults.27 This goes against the standard recommendations but may be perceived by researchers as a methodologic improvement.
Second, all current generic measures are derived from adult populations with the exception of the Health Utility Index Mark 2,28 and additional attributes that are particularly relevant to child health, including, for example, autonomy, body image, cognitive skills, and family relationships,29 may not be captured by these measures. Furthermore, no generic instrument for children and infants younger than 5 years is available. Although attempts have been made to establish feasibility and validity of the EQ-5D30 and the Quality of Well-Being Scale31 and there have been attempts to adapt the EQ-5D linguistically to produce a child-friendly version,32 the applicability of these instruments to pediatric populations is questionable. The result may be an implicit reluctance on the part of researchers to use these measures (note that only one third of studies here chose to use generic measures), instead choosing other routes toward utility measurement, such as the use of study of disease-specific states or general estimates of disability.
Third, children, particularly young children (note that >70% of the studies reported here were conducted for children who initially were aged 5 years and under) do not have the cognitive ability to comprehend and complete valuation or even measurement tasks. The implication is that, for very young children, some form of proxy inevitably will be used for measurement tasks, whether this be the clinician or the parent. Although parents may be perceived by economists as the more appropriate source of measurement and/or valuation,33 the potential for interaction between the utility function of the parent and the proxy (their child) for whom he or she is making the measurement/valuation may lead researchers to choose to use clinician judgment to avoid this problem.
Combining QALY Measures
The estimation of pediatric QALYs inevitably leads to questions about how to combine utility values both across various periods of a lifetime and between various individuals. The first problem may arise if it is acknowledged that different measures or sources of values are required for childhood than in adulthood. If different measures are used to determine quality of life weights in childhood and in adulthood as in some of the studies reviewed here, then can the utility values that are applied to different years be combined to produce 1 overall QALY? If not, then must the same measure be used across both periods of life, despite the inadequacy of such an approach?
The second problem may arise when QALYs are developed for both mother (and/or father) and child. In studies that evaluate interventions related to antenatal screening and child birth, for which there are impacts on both mother and child and both are, essentially, the patient, there is an inevitable question about how to combine the 2 sets of utility values. Furthermore, these questions are not restricted to this period; for health interventions that are provided to older children, the outcomes of treatment may clearly affect parents' health (eg, in terms of anxiety, ability to carry out usual activities). Simple aggregation of QALYs is unlikely to be a valid solution because of interdependence between the utility function of the child and the parent,34 but it is difficult to see an alternative within the cost-effectiveness framework with its desire for a single outcome measure (unless, of course, 1 source of utility gain is ignored).
| CONCLUSIONS |
|---|
|
|
|---|
There seem to be a number of potential approaches to solving these problems. The choice of approach, however, is likely to be affected by beliefs about the best way forward in health care decision making.
The first set of solutions assumes that the use of comparable generic instruments across all research studies is an absolute requirement. Within this requirement, there are 2 broad approaches. The first is to develop measures that are applicable across both child and adult populations; this is likely to be difficult, particularly when taking into account children under 5 years. The second is to use current measures developed for adult populations with an (implicit or explicit) acceptance that their use could seriously under- or overestimate the utility associated with particular options and thus result in incorrect decisions. This option would effectively maintain the current status quo, in which generic adult measures are recommended but, because of problems with their use, analysts may choose "alternative" forms of QALY measurement.
The second set of solutions relaxes the absolute requirement for comparability across all research studies. One approach here would be to develop generic measures for use specifically in pediatric populations, without requiring comparability with adult populations. This would allow comparison across childhood interventions but if the benefits of an intervention were expected to last beyond the age of 16 years would result in the problem of how to combine with adult measures. An alternative solution would be to use cost-consequences analysis34 rather than aiming to obtain a single outcome variable. This makes comparison across different studies more difficult but has the benefit of allowing information about outcomes to different people to be explicitly included in the decision-making process.
It has to be accepted than none of these approaches, apart perhaps from the first, is ideal. The problem with the first approach is that it could prove difficult, time consuming, and ultimately unfeasible. Although the number of published pediatric cost-utility studies continues to grow exponentially1 (and can be expected to do so into the future as new, high-cost diagnostic tests and medical procedures emerge), it is vital to improve on the current situation in which comparability is lost and what replaces it is ideal from no one's point of view.
As shown in this article, the estimation of QALYs in pediatric studies should not yet be regarded as standardized. Comparisons of the relative cost-effectiveness reported as cost per QALY gained across interventions for different diseases and populations should be treated with extreme caution. There is a pressing need for additional methodologic research to resolve the issues identified in this article before the QALY framework can be applied confidently in this population. In the absence of a clear way forward, it is suggested here that an expert panel, along the lines of the US Panel on Cost-Effectiveness in Health and Medicine, should be convened to debate and consider further the potential solutions outlined above and to make clear recommendations for best practice and future research. In the absence of such clear recommendations, future research using outcomes in the form of QALYs (or, indeed, choosing not to use the QALY as an outcome measure) should clearly justify their choice of methods for measurement and valuation.
|
| ACKNOWLEDGMENTS |
|---|
| FOOTNOTES |
|---|
Reprint requests to (I.G.) MRC Health Services Research Collaboration, University of Bristol, Department of Social Medicine, Whiteladies Road, Bristol BS8 2PR, United Kingdom. E-mail: Ingolf.Griebsch{at}bristol.ac.uk
No conflict of interest declared.
| REFERENCES |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
H.-I. Huppertz, J. Forster, U. Heininger, R. Roos, H.-U. Neumann, and T. Hammerschmidt The Parental Appraisal of the Morbidity of Diarrhea in Infants and Toddlers (PAMODI) Survey Clinical Pediatrics, May 1, 2008; 47(4): 363 - 371. [Abstract] [PDF] |
||||
![]() |
G. de Lissovoy, L. S. Matza, H. Green, M. Werner, and T. Edgar Cost-effectiveness of Intrathecal Baclofen Therapy for the Treatment of Severe Spasticity Associated With Cerebral Palsy J Child Neurol, January 1, 2007; 22(1): 49 - 59. [Abstract] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||