The checklist of questions shown in Table I informs those who are preparing submissions of the information that reviewers will be seeking. The list does not prescribe the methods by which those questions should be answered. Currently there is no single best methodological strategy for handling these questions in all circumstances. For some kinds of products, one methodology may be more appropriate than another. In other circumstances there is no consensus about a single best strategy or methodology. As such, this document is not intended to provide specific guidance on methodological issues. Manufacturers can receive further guidance on methodological issues by referring to the "guidelines document" arising from the Canadian Collaborative Workshop on Pharmacoeconomics held in June 1993 (still under preparation in June 1994). However, certain issues are worthy of comment here.

A full economic analysis can be divided into three components: establishing effectiveness, estimating quality of life or utility, and measuring cost (Laupacis et al. 1992).

The highest quality evidence for effectiveness of pharmaceutical products is provided by well designed randomized controlled trials with low false positive and false negative errors (Sackett 1989), or by overviews (meta-analyses) in which only randomized controlled trials are included. A comprehensive search method is used to locate all relevant studies, the variations in findings of the studies are analyzed, and the results of primary studies are combined in an appropriate manner. The work of Sacks et al. (1987) and L'Abbe et al. (1987) can be used to guide the preparation of submissions that use overview methodology to estimate effectiveness.

Because all pharmaceutical products considered for listing in the Ontario Drug Benefit Formulary will have been reviewed by the Health Protection Branch, it is expected that efficacy, and perhaps effectiveness, will have already been demonstrated by randomized clinical trials with low risks of false positive and false negative errors. However, there is an important difference between the efficacy/ effectiveness demonstrated in randomized trials and effectiveness after the product is used in routine clinical circumstances. This difference is due to a variety of reasons including the restrictive inclusion/exclusion criteria for clinical trials, and compliance exhibited by both clinicians and patients in randomized trials.

Therefore, cost-effectiveness analyses must consider the difference between efficacy (evidence that demonstrates the value of a product used in the optimal circumstances of most randomized trials) and effectiveness (the benefit derived from using the product in usual circumstances). judgement may be required to extrapolate the results of efficacy trials by incorporating other studies in more generalizable samples to economic analyses. In particular, the estimated effectiveness must be seen to apply to recipients of the Ontario Drug Benefit Program. This does not necessarily mean that the trials have to be performed in Canadians who are older than 65. However, it must be demonstrated that there is reason to believe that the subjects in the studies that form the basis of the evidence used in the analysis are similar to those covered by the Ontario Drug Benefit Program.

If the analysis uses utility as a measure of outcome, then it should be estimated in a sample of subjects, with the disorder of interest, using measures that have previously demonstrated validity, reliability and responsiveness to change (Kirshner & Guyatt 1985). In some cases, proxies for patients are appropriate, such as members of the general public who are informed about the clinical course through descriptive scenarios. In order to allow comparison with cost-utility ratios for other interventions, it is preferable to use a utility measure that incorporates general health status rather than disease- specific measures of quality of life.

It should be noted that there is considerable variation in the experience of clinical investigators in measuring utilities and quality of life across different types of interventions and diseases. For some diseases such as cardiovascular disease, respiratory disease, and endstage renal disease, there is considerable experience in applying utility measurements to give analysts reasonable assurance that the instruments are valid and reliable. For other types of diseases such as mental health, the measurement of utility and quality of life has occurred much less frequently. As a result, there will be different expectations about the incorporation of quality of life and utility issues for products that affect different types of diseases, depending on experience of other investigators prior to performing this particular economic analysis.

When estimates of utility were not performed, submissions should indicate whether this was because there was no known valid and reliable instrument. Those who prepare such submissions, however, may wish to develop instruments for measuring utilities in order to pursue present and future economic analyses over time. Such work may be considered as developmental but will clearly be important for further justification of expenditures on new products. (See section 3.4 for more discussion on measures of outcomes).

The highest quality estimates of costs include direct measurement of resources used by the competing therapeutic strategies, i.e., a measurement derived from sampling the population that uses those resources. There are two components of costs: the volume of services that are utilized (e.g., hospital days, laboratory tests, physician visits); and unit prices for each of those services. A high quality study measures both the volume of services and also uses clearly identifiable unit prices that apply to the service (e.g., health ministry rates for physician services) and direct measurement of costs borne by institutions, including an appropriate allocation of overhead.

Some investigators will have to use a sampling technique to estimate the quantity of individual services delivered under competing strategies. Others can use published information from other clinical trials.

Similar to the issue of using randomized trials to extrapolate effectiveness, analyses must consider the differences in the cost of actually using products in clinical practice and those resulting from using those products in trials. Protocol driven costs required by randomized trials designed to measure efficacy and safety (e.g., extensive blood tests) will not be relevant to cost-effectiveness analyses. As a result of this potential problem, it is sometimes preferable to perform economic analyses separately from studies designed to demonstrate efficacy or effectiveness. At other times it will be possible to use the economic data from clinical trials by isolating protocol driven costs from the measurement of relevant costs. Once again the costs must be relevant to Canada. This will usually mean that the unit prices for the resources need to be estimated in Canada but may allow for the use of quantities of individual services to be estimated from non-Canadian studies.

Because all cost-effectiveness analyses are comparative, the costs and clinical outcomes of the pharmaceutical product must be compared with those of an alternative product or strategy for treating the same patient. In some cases the choice is clear-cut [e.g., streptokinase versus alteplase (tissue plasminogen activator) as the thrombolytic agent in acute myocardial infarction]. In other cases there are several possible choices for the alternative therapy (e.g., new cephalosporin for the treatment of bronchitis or otitis media). For breakthrough products, the appropriate comparison may be nonpharmaceutical interventions, such as surgery or psychotherapy.

This issue is no different than the choice of the appropriate agent for clinical trial comparison and requires the same kinds of judgements. However, there are many cases where clinical trials compare the new product with a placebo to establish efficacy, whereas decisions about Formulary listing require comparison with other known effective alternative strategies. Comparisons should be made with the least expensive currently available strategy (e.g., generic price for alternative products used to treat the clinical problem), and may also be made with the most commonly used alternative product, if it differs from the least expensive product. In individual cases, other comparisons may also be relevant and the manufacturers may include them as well.

It is clear from this section, that the new product may be compared to more than one alternative in the report. Of course, in each case the analysis must be incremental and some products may prove to be dominated by others (i.e., result in higher costs and worse outcomes). For the rest of this report we use the term "comparison" for each incremental ratio; some reports will contain several comparisons.

The time horizon chosen for an economic evaluation is important and can dramatically affect the size of the incremental cost-effectiveness ratio. However, the data on which efficacy is based usually is derived from randomized trials or non-experimental studies that follow patients for a relatively short period of time. Modelling techniques must be used to project lifetime costs and effects if such a time frame is appropriate. Unfortunately, however, the data on which to project lifetime costs and clinical effects must almost certainly be much more speculative than those with a short time frame. Submissions should clearly state the time horizon chosen. The analysis should delineate the time horizon on which estimates can be based from currently available high quality empirical data (e.g., randomized trials that follow patients for months to a few years) or from modelled data based on extrapolations. The assumptions underlying these models should be clearly and transparently presented in the reports.

This issue points out the fact that economic analyses consider the structure of all relevant outcomes resulting from the use of one pharmaceutical product as opposed to another product or strategy. That is, events such as treatment failures necessitating hospitalization will result in hospital costs that may differ with the two alternative therapies being compared. An economic analysis considers not only the difference in the price of the alternatives themselves, but also all downstream events that result in differences in clinical outcomes and costs.

In many cases the randomized trials on which demonstration of efficacy is based will use outcomes that are not the final health objective of the product (e.g., measure change in blood pressure or cholesterol rather than cardiovascular morbidity and mortality). In order to perform an economic analysis, these intermediate health outcomes must then be used to project changes in final health outcomes and ultimately QALYs via simulation or other appropriate models. The structure of outcomes in these models will determine the incremental cost-utility ratios of the new products. The models must be explicitly presented to determine the validity of the estimated ratios. In particular, if analyses use intermediate health variables to project final health outcomes, the validity of the relationship between the two must be established, e.g., lower cholesterol and reduced deaths from coronary heart disease.

There are three possible types of measures of outcomes :   non-monetary outcome measures such as QALYs or utilities; monetary outcome measures such as willingness to pay; and discrete clinical outcome measures ("natural units").

In recent years a wide variety of approaches to the measurement of quality of life have emerged. Such diversity is both inevitable and appropriate because quality of life is a broad, multidimensional concept. Different measures may tap different domains of quality of life and investigators may have a variety of objectives that require different combinations of instruments. For the purpose of performing an economic evaluation of different health interventions, a single global score is essential to allow for comparisons across different interventions.

A commonly used non-monetary measure of outcome for economic evaluation is the quality adjusted life years (QALYs) gained. The advantage of QALYs gained lies in its provision of a common unit of measurement that allows comparisons across interventions. Furthermore this unit of outcome enables us to compare interventions that affect both quality and length of life since it combines both of these outcomes. It also enables us to compare different effects on different dimensions of quality of life.

The QALYs measure falls under the rubric of a "health status index". It is a weighting scheme where each definable health status is assigned a weight from zero (death) to one (full health), and then the number of years spent at a given health status is multiplied by the corresponding weight to yield a number that might be thought of as an equivalent number of years with full health. It is important to mention that the source of these weights is ultimately subjective and that many methods exist to generate the weights (Mehrez & Gafni 199 1). A method that is highly advocated is to calculate the QALYs as a utility-weighted index (Torrance & Feeny 1989). The appeal of such a method is in relating the QALYs index to utility theory which provides a solid theoretical foundation of decision making under uncertainty.

The QALYs measure however, suffers from some limitations. The first one is the fact that a utility weighted index is not a utility function (Torrance & Feeny 1989). Only under very restrictive conditions that are unlikely to fully exist, would QALYs be a utility function (Torrance & Feeny 1989). In addition, some researchers have shown that there is a theoretical possibility that QALYs may not truly represent people's preferences, and may, in some cases, misrepresent them (Mehrez & Gafni 1991). At the present time there is not enough empirical evidence to show whether or not these theoretical problems are empirically important. More research is needed to answer this question and it is expected that some of the evidence will be derived from studies used to support submissions for Formulary listing.

Another option is to use an alternative measure of outcome, the healthy years equivalents (HYE) (Mehrez & Gafni 1991, 1992). This measure is rooted in the theoretical foundations of utility theory. HYE stems directly from the individual's utility function and thus fully (rather than partly) reflects her or his preferences. It combines outcomes of both morbidity and mortality and thus can serve as a common unit of measure for all programs, allowing comparisons across programs. It also preserves the intuitive appealing meaning of QALYs. In theory it is possible that the HYE method avoids the problems of QALYs discussed above since it is said to fully represent individuals' preferences.

Since both methods of non-monetary outcome measures (QALYs or HYE) involve subjective measurement with methods left to the discretion of the researchers, it is important to state that different measurement methods may well result in different estimates of clinical outcomes. Incremental cost-utility ratios involve the difference in clinical outcomes between the new and old strategies, which are then divided into the difference in costs, thus different methods may result in large variations in the incremental cost-utility ratios. For this reason, reports should use extensive sensitivity analyses to demonstrate that the conclusions of the analyses are not sensitive to the type of measurement chosen. Reports that use several measurement techniques for converting the clinical effects into values are more likely to provide empirical demonstration of the insensitivity of the conclusion to the methods chosen. Alternatively, modelling techniques can use very wide confidence limits on these utilities in order to assure the reviewers that the results are not sensitive to the measurement method.

It should be noted that disease-specific utility scales are unacceptable for calculating QALYs because they do not permit comparison across programs (e.g., health care interventions). Only instruments measuring the utility for general health status are acceptable if the ratio is to be compared with those interventions associated with other health/disease states. Disease-specific quality of life or utility scales, if used, will only allow comparison of the economic consequences of different interventions within that disease category. Of course, manufacturers are allowed to use and present disease-specific scales. However, if they are not accompanied by general health measures, the DQTC will have to apply its own judgement about the economic attractiveness of these products compared to other products used in other diseases.

It is also important to note that the use of QALYs or HYE implies a particular decision about equity, namely that each individual's QALYs or HYE is exactly equal to another individual's QALYs. At the present time this seems to be the standard equity decision implied by most economic analyses simply because most of them use QALYs as the outcome. There are, however, other possible equity perspectives such as a premature death averted, no matter what the subsequent life expectancy, or years of life extended at a particular age (i.e., some might value years of life extended for young people as being different from years of life extended for older people). Some investigators have pointed out the importance of the choice of outcome measure in terms of its impact on equity considerations and reports should also be sensitive to this issue (Gafni & Birch 1991).

In the last two decades conversion of clinical effects into monetary outcome measures has fallen out of favour. However there has been a renewed interest in using this method, known as costbenefit analysis, by using willingness to pay (WTP) as an approach in placing a value on health outcomes (Gaffii 1991). Once again there is some controversy about the best method of eliciting a measurement of willingness to pay (whether it should be in a risk-free or riskincorporating framework, relation to income, etc.,). Those preparing submissions should be encouraged to try this technique, even though it will require flexible interpretation of the results.

Because the utility, QALYs, and monetary approaches combine several clinical effectiveness issues into a single measure, some reviewers of economic studies feel that it is difficult to interpret these values in terms that are easily understood by clinicians and policy makers. We will use the term "natural units" to refer to discrete clinical endpoints such as myocardial infarctions, minor side effects, strokes, premature deaths from a disease, etc. While there is no standard set of the natural units for each disease category, the Australian government is beginning a process of consensus meetings to derive such units for different diseases.

The disadvantage of using these units alone is that it does not permit comparison of incremental cost-effectiveness ratios across interventions. However, they are more easily interpreted and may allow readers to compare these values against their own willingness to pay (e.g., if the cost per minor side effect prevented, such as mild transient pain, is $3000 it may be too high, but $.50 may be perfectly acceptable).

In summary, a variety of approaches for measuring and valuing clinical outcomes has emerged. The present standard method of QALYs is by no means the only appropriate technique and in some circumstances other methods may well be preferred. In order to permit reviewers the best conditions for interpreting the studies, clarity, and transparency of methods and assumptions are crucial. Clinical outcomes should be presented in a variety of ways including natural units, such as the number of myocardial infarctions or premature deaths avoided. As the DQTC gains more experience in reviewing economic submissions, and as the science of pharmacoeconomics advances, we expect these suggestions and guidelines to change. At the present time, flexible interpretation of studies will be the rule.

There are some circumstances where a measurement of quality of life will not be necessary. If an analysis shows that the incremental cost per QALYs gained is entirely insensitive to alterations in quality of life (usually because the main effect of the product is to result in prolongation of life) then the analysis need not directly measure those utilities on subjects. As well, if the main economic impact of a new product is to alter the total cost of treating patients with that disease because the product offers some other advantage that is reflected in resource utilization while providing equivalent clinical outcomes, then one can simply do a cost comparison without measuring utilities for the clinical outcomes. An example of this situation is the comparison between two pharmaceutical products that result in equal clinical outcomes for patients with a particular disease.

Also, as mentioned above, there will be some pharmaceutical products for which QALYs are extremely difficult to measure at the present time. These include calculating cost-utility ratios for interventions that reduce very short term disabilities, e.g., nausea, vomiting, or minor GI distress associated with a medication. Some investigators are using willingness to pay in order to measure short term disabilities such as these. In other areas, such as pharmaceutical products aimed at mental health disorders, the measurement of utility is currently difficult because there has not been extensive experience and there is insufficient evidence of validity and reliability. In some of these circumstances the economic analysis will consist of a comparison of costs and a comparison of outcomes expressed as discrete clinical events such as episodes of illness avoided, side-effects avoided or disease-specific quality of life scales. In these cases, the DQTC will have to apply its own judgement about the economic attractiveness of these products relative to others.

The following is a suggested format and order for the presentation of differences in costs and outcomes associated with alternate measurement strategies discussed above that will permit the reviewer to transparently understand the steps in the analysis.

1.   Cost-comparison. As a first step, the analysis should present the cost differences between the two comparative strategies. It would be helpful to divide the cost differences into two components: the difference in unit prices for the pharmaceutical product compared to the alternate strategy (usually another pharmaceutical product) over the course of the illness; and the "cost offsets" that result from issues such as prevention of adverse events caused by the disease under treatment and/or difference in costs resulting from different side-effect profiles. For example, if drug A costs $200 more than drug B when one considers the pharmaceutical costs alone, that would be the first component. However, if drug A resulted in fewer events such as myocardial infarction or stroke, as well as a lower side-effect profile such as reduced risk of GI bleeds, then the cost offsets might be $50, resulting in an incremental cost per patient of $150.

2.   Cost-consequence analysis. The multiple outcomes that are clinically important can be expressed as a vector. The differences between alternative therapeutic strategies can thus be expressed as a vector of differences in probabilities of each of these clinical events. For example, the vector of outcomes for antiplatelet agents might include deaths, myocardial infarctions, strokes, and gastrointestinal bleeding. The differences in the probabilities of each of these events, expressed as a list, would constitute a vector of consequences. Together with the cost information, this is an example of a cost- consequence analysis.

3.   Cost-effectiveness analysis. One can then take the most important clinical outcome, e.g., premature death avoided from disease, and calculate a cost-effectiveness ratio by putting that difference in the denominator and the difference in cost in the numerator. Usually the outcome chosen is one of the outcomes from the cost-consequence analysis.

4.   Cost-utility analysis. Outcomes can be translated into monetary utilities by converting the vector of outcomes into utilities and the analysis can then be expressed as a cost-utility ratio.

5.   Cost-benefit analysis. Outcomes can also be translated into benefits and expressed as a cost-benefit difference or ratio.

Analyses need not present all five steps. However, at a minimum, they should express cost-comparison and cost-consequence analysis and probably cost-effectiveness analysis as well. If the study has gone on to translate outcomes into utilities or benefits, it would allow DQTC to put these ratios in the context of other health care interventions. In order to permit reviewers to fully understand the analysis, a transparent display of the results in the order listed here is preferable. If the analysis has not gone beyond a cost-consequence analysis, the reasons should be stated, e.g., no appropriate utility measure available, outcomes are equivalent, etc.

Reviewers will have to determine whether the patient group on which the economic analysis is based represents those patients who will utilize the product if it is listed on the Formulary. The drug may be shown to be economically attractive for a specific clinical scenario that represents a small proportion of those who may use it if it is fully listed on the Formulary; for other patients, the use may be economically unattractive. Manufacturers who are applying for full listing should address this issue in calculating the incremental cost-effectiveness ratios and aggregate costs.

© Queen's Printer for Ontario, 2002  |  Privacy Policy  |  Disclaimers  |   Last Modified :