Costs and cost determination. Decision Modeling Techniques. Markov Modeling in Decision Analysis

Costs and cost determination. Decision Modeling Techniques. Markov Modeling in Decision Analysis

 

1.1 Basic concepts

1. 2 Cost classification

2.1 Decision Modeling Techniques

2.2 Decision Modeling Paradigm

2.2.1 Types of Decision Modeling Techniques

2.2.2 Decision Trees

2.2.2.1 Steps in Conducting a Decision Analysis

2.2.2.2 Step 1: FRAME the Question

2.2.2.3 Step 2: STRUCTURE the Clinical Problem

2.2.2.4 Step 3: Estimate the PROBABILITIES

2.2.2.5 Step 4: Estimate the VALUES of the Outcomes

2.2.2.6 Step 5: ANALYZE the Tree (Average Out/Fold Back)

2.2.2.7 Step 6: TEST ASSUMPTIONS (Sensitivity Analysis)

2.2.2.8 Step 7: INTERPRET the Results

2.2.3 Markov Models

2.2.4 Simulation Models

2.2.4.1 Microsimulation

2.2.4.2 Discrete Event Simulation

2.2.4.3 Agent-Based Simulation

2.2.5 Deterministic (Mechanistic) Models

2.2.6 Summary of Modeling Types

2.3 Example

2.3.1 Step 1: Framing the Question

2.3.2 Step 2: Structuring the Clinical Problem

2.3.3 Step 3: Estimate the Probabilities

2.3.4 Step 4: Estimate the Values of the Outcomes

2.3.5 Step 5: Analyze the Tree

2.3.6 Step 6: Test Assumptions (Sensitivity Analysis)

2.3.7 Step 7: Interpret the Results

4.1 Markov Modeling in Decision Analysis

4.2 The Markov Process and Transition Probabilities

4.2.1 Stochastic Processes

4.2.2 Markov Processes

4.2.2.1 Transition Probabilities

4.2.2.2 Working with a Transition Probability Matrix

4.2.3 Absorbing Markov Models

4.2.3.1 Behavior of the Absorbing Model

4.2.3.2 Use of Absorbing Markov Models in Clinical Decision Analysis

4.3 Markov Model Example: Cervical Cancer

 

1.1 Basic concepts

 

Health care funders (governments, social security funds, insurance companies) are struggling to meet their rising costs. They make many efforts to contain drug costs, by price negotiation, patient co-payments or dedicated drug budgets. Expenditure on drug therapy is a particular target for their attention for several reasons: the size of the drug bill (10-15% of most national health care budgets, and usually the second largest item after salaries); the ease of measurement of pharmaceutical costs in isolation, in contrast to most other health care costs; evidence of wasteful prescribing; and a perception that many drugs are overpriced and that the profits of the pharmaceutical industry are excessive.

But this focuses on drug costs in isolation, when what should be of greater concern to decision makers, health care professionals and the public is the value of drug therapy, a function of its benefits as well as its costs. Payers acknowledge that spending on drugs, which may for instance reduce the need for hospitalisation or produce greater health gain for the same resources than other medical interventions, may be a very efficient use of scarce resources. They therefore respond by demanding evidence of value for money from drug therapy. Drug therapy is open to this simply because there are high quality trials to support most new drugs’ licensing applications, in contrast to the poor evidence around most other health care interventions.

Health economics is the science of assessing cost and benefits, not to make decisions about resource use, but to inform those decisions. The aim is to identify what is most efficient, so that the greatest amount of benefit can be bought for a given amount of money or resources. But we must remember that in health care, efficiency may not be the most important objective -we might for instance prioritise caring for dying patients or treating patients with serious disease who have relatively little hope of surviving. Pharmacoeconomics is a branch of health economics that particularly considers drug therapy. It is of particular interest to pharmaceutical companies who in developing a new drug and after the traditional hurdles of efficacy, safety and tolerability must now jump over a fourth hurdle of cost effectiveness. It should also interest clinical pharmacologists, either in their roles assessing new drugs or in the conduct of clinical trials that now often include an economic component. In some areas, economic studies have become an accepted part of evaluations for reimbursement.

This chapter aims to explain the basic concepts and language of pharmacoeconomics, and of economic evaluation, and to introduce what for many clinical pharmacologists is a new area. Health economics is about making choices between options, when there is scarcity of resources. It is fundamentally comparative, weighing the costs and benefits of option 1 with those of option 2 (for instance, a new drug and the previous best therapy – traditional medical evaluation focused only on the benefits), to determine which is the most efficient way to use our limited resources. Efficiency is a key concept in economics, i.e. how to buy the greatest amount of benefit for a given resource use.

Another key concept is opportunity cost: this is defined as the “benefit foregone when selecting one therapy alternative over the next best alternative”. When we have limited money and we spend it on one health care intervention, we cannot spend the same money on something else. So we should be less concerned with how much a health care intervention costs, but rather with what other benefits we are giving up by using the money in that way. We need to be sure that spending money on the new therapy will buy more benefit than spending that money in some other part of the health care system.

The comparative nature of health economics means that we are interested in an Incremental analysis of costs and benefits. There is usually a current treatment for most conditions, with associated costs and benefits. We would not advocate stopping all existing treatment for the condition, so the question is not what are the costs and benefits of the new treatment, but what are its added costs and benefits, over and above those of the existing treatment.

A related concept is marginal costs. For instance, if a new treatment enables patients to be discharged from hospital a day earlier than an older treatment, it might be tempting to count the average cost of a hospital bed day as a saving of resources. But all the fixed capital charges for a hospital bed, which go into the average cost, e.g. costs of laboratories, kitchens, and building maintenance, will be largely unchanged. The only costs which change may be those of having a patient physically occupy the bed – the costs of the patient’s meals, treatment and perhaps nursing time. These are the marginal costs, where the resource use actually changes substantially. Incremental analysis is concerned with the marginal and not the average costs. Marginal costs are often very difficult to measure, and there is a temptation to use average costs instead. This may be justified if for instance, enough bed days are saved by the widespread adoption of a new treatment to actually reduce bed numbers and to close wards.

 

1.2 Cost classification

 

What is a cost? According to the American Heritage Dictionary, a cost is 1. An amount paid or required in payment for a purchase; a price; or 2. The expenditure of something, such as time or labor, necessary for the attainment of a goal.

Costs can be subdivided into three components: Direct costs, Indirect costs, and Intangible costs.  Direct costs are expenditures on tangible items which contribute to the gross domestic product.  In health care we further divide the direct costs into direct medical costs and direct non medical costs.

Costs therefore can be classified as:

A. Direct – i.e. costs from the perspective of the healthcare funder: including staff costs, capital costs, drug acquisition costs. These should (in theory) be relatively easy to measure.

Direct Medical Costs-

1)                    The amount of money spent on medical services as a direct result of an illness; or 2)  The amount of resources consumed directly to produce a certain outcome such as health.

Examples of money spent as a result of an illness include this includes hospital care, pharmaceutical products, physician care, nursing services, etc.

Examples of resources consumed to produce health include a physician’s , nurse’s, or pharmacist’s time, equipment, routine supplies, etc. 

Examples of money spent as a result of an illness include this includes hospital care, pharmaceutical products, physician care, nursing services, etc.

Examples of resources consumed to produce health include a physician’s , nurse’s, or pharmacist’s time, equipment, routine supplies, etc. 

Direct Non-medical Costs-

1)  Expenditures outside the medical market such as costs borne by patients seeking care.

This includes costs such as transportation, child care, and lodging, etc.

B. Indirect – i.e. costs from the perspective of society as a whole: for example, these might include loss of earnings, loss of productivity, loss of leisure time, due to the illness, and cost of travel to hospital etc. This would include not just the patient themselves but also their family and society as a whole. Many of these are difficult to measure, and there is some controversy over how to value these. (The UK National Institute for Clinical Excellence, NICE, adopts a limited societal perspective in its evaluations and considers the direct costs falling on the UK National Health Services, and those indirect costs funded by the state such as unemployment and sickness benefits).

Indirect Costs-

1)  Costs resulting from a patient being unable to perform normal activities due to illness and therefore borne by either the patient, the patient’s family, or an employer.

2)  Expenditures or losses as an indirect consequence of illness or consumption of medical care.

3)                      Economic value of changes in health status.  measured by lost wages, willingness to pay, or human capital theory.

Examples include lost earnings, decreased productivity. 

C. Intangible – i.e. the pain, worry or other distress which a patient or their family might suffer. These may be impossible to measure in monetary terms, but are sometimes captured in measures of quality of life.

Intangible Costs-

1)                    Include humanistic measures of changes in health status such as quality of life and satisfaction.

This is the cost of pain and suffering.  It is very difficult to measure and is ofteot included in analyses.

Direct Medical Costs-

1)  The amount of money spent on medical services directly due to an illness.  this includes hospital care, pharmaceutical products, physician care, nursing services, etc.

2)  The amount of resources consumed directly to produce a certain outcome such as personnel time, equipment, supplies, etc.

Direct Non-medical Costs-

1)  Expenditures outside the medical market such as costs borne by patients seeking care.  This includes costs such as transportation, child care, and lodging, etc.)

Indirect Costs-

1)  Costs resulting from a patient being unable to perform normal activities due to illness and therefore borne by either the patient, the patient’s family, or an employer.

2)  Expenditures or losses as an indirect consequence of illness or consumption of medical care.  Examples include lost earnings, decreased productivity. 

3)  Economic value of changes in health status.  measured by lost wages, willingness to pay, or human capital theory.

Intangible Costs-

1) Include humanistic measures of changes in health status such as quality of life and satisfaction.

Let us review the cost terminology which can become confusing at times.

Variable costs are those costs which are related to production.  That is costs that increase as units of output increase or decrease as units of output decrease.  A pharmacy example of a variable cost would be drug product costs.  The more prescriptions you dispense, the more money you will have to spend to purchase drug products.  Variable costs can often be allocated directly to a program.

Fixed costs are costs that do not change relative to production.  These are costs that remain constant no matter how the output changes.  A pharmacy example of a fixed cost is the mortgage on the pharmacy building itself.  The mortgage will remain the same if zero prescriptions are dispensed or if 1,000 are dispensed. It may or may not be  possible to allocate fixed costs directly to a program. Fixed costs that cannot be allocated to a specific program are referred to as overhead costs.  These costs are share among programs.

Total costs are the sum of all costs.  Total direct health care costs would include the sum of all medical, laboratory, hospital, and pharmacy costs.

Marginal cost is the cost to produce one more unit of output.  For example, the cost to dispense one more prescription or the cost to treat one more patient.

Average cost is simply the total costs divided by the the unit of analysis.  If we wanted to calculate the average health care cost per member of an insured population, we would divide the total health care costs for that population by the total number of people in that population. 

In this worksheet you can see the mathematical relationship between marginal, average, and total costs.  The marginal cost is = to the (total cost₂ – total cost₁).  Total cost is the sum of all costs, and average cost is the total cost divided by the number of participants.  Notice the relationship between marginal and average costs in the graph.

It is highlighted for you on this slide.  The average cost always follows the marginal costs.  If the marginal cost is decreasing then the AC is decreasing.  When the MC is increasing, then the AC increases.

Next we will go over the 5 steps involved in determining costs.  The first is Specifying the inputs; the second is counting the units; the third is assigning $ values; the fourth is adjusting for time preferences; and the Fifth is allowing for uncertainty.

Specifying the Inputs

Ø     Develop comprehensive list of ALL relevant inputs (i.e. resources consumed) to produce a given output or consequence

Ø     The list will be determined by the perspective of the analysis

Ø     Common to all forms of economic evaluation

Let’s discuss perspective for a moment.  This is a very important concept in pharmacoeconomics because it determines which costs and outcomes are included in an analysis.  The most common perspective is the payer perspective.  Imagine a pharmacists run asthma management service.  Can you think of direct and indirect costs that a third party payer might pay for related to a pharmacist run asthma clinic?

Payer Perspective

Ø     Direct

Ø Hospitalizations

Ø Laboratory costs

Ø Medications

Ø Medical devises

Ø Physician / pharmacist fees

Ø  Indirect (overhead / accounting costs)

Here are some examples that we determined a third party payer might cover.  Notice that these are almost always direct medical costs.  Care must be taken when allocating costs to focus on who is paying and what are they paying for?  It is tempting for people to include the cost for the third party to do business.  However, in this example we are only interested in the costs that a third party would pay for for a pharmacy asthma management service.

Another common perspective is the provider perspective.  Think again of a pharmacist run asthma management service in a community pharmacy.  Can you think of direct and indirect costs that the pharmacy would have to cover to provide this service?

Ø     Direct

v Equipment / leases

v Fees

v Education

v Marketing

v  Dedicated personnel

Ø     Indirect (overhead / accounting costs)

v Maintenance

v Utilities

v Mortgage / rent

v Business licenses

v Office supplies

v laundry, haz. waste disposal, etc

Here are some examples that we developed.  Notice that from the provider perspective, we are talking about the cost of doing business.  Some people will only include those costs which you can allocate directly to the service, while others will work to try to allocate overhead (indirect) costs as well.

Patient Perspective

Another perspective is that of the patient.  Imagine a pharmacists run asthma management service again.  Can you think of direct and indirect costs that a patient would pay for related to a asthma service?

Societal Perspective

The final perspective that we will discuss today is the societal or governmental perspective.  This is the broadest perspective, that is it encompasses the most costs.  Although this perspective is not common in the US, it is common in countries that have national health care systems because the government is the largest payer of health care benefits.  The government is also concerned with productivity and the gross domestic product.  Imagine again the pharmacists run asthma management service.  Can you think of direct and indirect costs that society or the government might incur related to a pharmacist run asthma service?

Ø     Direct

Ø healthcare costs,

Ø e.g. premiums,

Ø Medicare & Medicaid,

Ø Taxes,

Ø Medication R&D,

Ø Healthcare workforce,

Ø     Indirect

Ø lost productivity,

Ø lost wages

These or the costs that we wrote down.   Again in the US, the third party payer perspective is most common since third party insurance is how we pay for health care in the US.  Furthermore, most studies will include only direct medical costs.

The next step in determining costs is to count the units.  In order to do this you have to determine the unit of use or the unit of analysis, then you have to count the number of units that are consumed over time.

The next step is to assign dollar values to the costs.  This is relatively straight forward for consumable items that you have to purchase to do business.  For example, you are the owner of a pharmacy and you want to determine how much it costs to run your asthma management service you could determine how much it costs you to purchase the medications you use, the educational pamphlets that you distribute, and the peak flow meters that you distribute.  Most of these costs can be obtained from the pharmacy’s accounting system. 

However how do you cost the pharmacist’s time.  Do you use the market price (Salary divided by hours spent) of do you use opportunity costs (that is the amount of revenue would the pharmacist be generating if he or she were filling prescriptions rather than running the asthma service)?  Opportunity costs are economic costs where as market prices or market costs are accounting prices.

Here is an example of a spread sheet from a hospital examining the costs to treat patients with antibiotics.  The total cost is the sum of the fixed costs plus the variable costs plus the abx cost (a variable cost that was broken out for this project).  You can see how much they charged and how much payment they received.  You can also see the length of stay in the first column and the age of the patient in the second column.  In this example the allocation of fixed cost to each hospitalization was done by the accounting department.  Notice that the costs and charges are different.  This is important because people sometimes confuse these terms and use them incorrectly interchangeably.   A cost is how much it requires to run a business or provide a service.  A charge is the amount of money that the consumer must pay in order to receive that product or service.  A charge from a provider perspective may be a cost from a third party payers perspective.  That is why perspective is so important, so is the use of proper terminology. 

Where do you get the cost data?  You can get them from primary data collection sources.  That is, you decide to conduct a clinical or naturalistic trial to determine the quantity and value of the resources are consumed. 

You could also obtain cost data from secondary data sources.  These are data that were collected for some other purpose that you can use to answer your research question.  Examples include claims databases,  the literature, or expert opinion. 

If you do not have access to primary or secondary data, you can use a model to estimate the costs.  We will discuss this in a couple of lectures.

Do you know the difference between a clinical trial and a naturalistic trial?  A clinical trial has at least two groups (control and intervention group) and the participants are randomly selected or assigned to a specific group.  Then one factor or variable is changed (such as which drug the intervention group received) and all the other factors are held constant, then you compare the difference between the groups.  In a naturalistic study, you have two groups and people may or may not be randomly assigned to the groups.  The one factor is changed, but the outhers are not controlled because you want to see the effecti this change might have in the real world.  (A clinical trial demonstrates efficacy, while a naturalistic trial demonstrates effectiveness).

 

2.1 Decision Modeling Techniques

The fundamental purpose of a pharmacoeconomic model is to evaluate the expected costs and outcomes of a decision (or series of decisions) about the use of a pharmacotherapy compared with one or many alternatives. Decision modeling provides an excellent framework for developing estimates of these outcomes in a flexible analytic framework that allows the investigator to test many alternative assumptions and scenarios. In addition to providing an “answer” to a specific pharmacoeconomic decision, one of the major advantages of having a model of a particular decision is that the model can provide significant information regarding how the answer changes with different basic assumptions, or under different conditions. It is this ability to evaluate multiple “what if” scenarios that provides a substantial amount of the power of pharmacoeconomic modeling.

This chapter provides a brief introduction to the many methods of constructing decision models for the purpose of pharmacoeconomic analyses. After describing the basic methods of decision analysis, basic branch and node decision trees are described in the context of an actual pharmacoeconomic problem. Many of the techniques used to make these models more clinically detailed and realistic are detailed in other chapters in this book, and these chapters are referenced where appropriate.

 

2.2 Decision Modeling Paradigm

The most important aspect of the decision modeling process is that it must represent the choice that is being made. When constructing a model of a clinical or pharmacological decision, a series of characteristics of the actual problem must be represented in the model structure and method. First, the model should represent the set of reasonable choices between which the decision maker must choose. Leaving out reasonable potential or common strategies subjects the model to criticisms of bias and selecting comparators that make the superiority of a particular strategy more likely. Even if “doing nothing” is not a viable clinical alternative, it is often useful to include such a strategy as a baseline check of the model’s ability to predict the outcomes of the natural history of untreated disease.

Once the strategies are outlined, the modeler must enumerate the possible outcomes implied by each strategy. These outcomes are not always symmetric; a surgical therapy may have an operative mortality whereas a medical therapy may not. However, all potential outcomes that can occur and are considered relevant to clinicians taking care of the problem should be included. Pharmacoeconomic models are characterized by their simultaneous assessment of the clinical and cost consequences of various strategies, so even clinically insignificant outcomes that incur significant costs may need to be modeled. To make an appropriate decision regarding what consequences and outcomes to include, the modeler must make decisions regarding four characteristics: the perspective of the analysis, the setting or context of the analysis, the appropriate level of detail or granularity, and the appropriate time horizon.

Perspective: The perspective of the analysis determines from whose point of view the decision is being made. Defining the perspective of the analysis is especially important in pharmacoeconomic analyses because the costs that are incurred depend heavily on the perspective. The most typical perspectives used in pharmacoeconomic analyses are that of the payer (insurance companies, HMOs, Medicare), in which only those costs incurred by the payer are included, a provider (hospital, health system, provider group) in which the costs and reimbursements for providing a particular service are included, and society, in which all costs and effects are included, irrespective of who has borne them (see Table 2.1).

 

 

A more detailed description of perspective is provided in standard texts. For example, an analysis conducted from the payer perspective on a particular treatment for a neurological condition might not take into account the differential effects of the various therapies being studied on the patients’ ability to return to work, as these are not costs or benefits that are borne by the payer. However, these costs and benefits should be included if the analysis is being conducted from the perspective of society.

Setting: The setting defines the characteristics for which a particular decision is being made. Just as any study desigeeds to define the population, the study will evaluate (by inclusion and exclusion criteria in randomized controlled trials or by case and control definition in many observational designs), a decision model must explicitly state the type of patient(s) to which the decision will be applied. For example, in developing a pharmacoeconomic model of the use of statins in hypercholesterolemia, the modeler must decide the distribution of age, gender, lipid levels, comorbid disease, and other variables that are important and need to be represented in the model. A model that demonstrated a particular result in one group of patients is not likely to have the same result in populations with different characteristics.

Granularity: The correct amount of detail to include in a model of a given clinical situation is one of the most difficult decisions a modeler must make in the development of a representation of a particular decision and its consequences. Albert Einstein once said: “Things should be made as simple as possible … but not simpler.” Although this concept is directly translatable to building decision models, it provides little actual guidance; the clinical and pharmacoeconomic characteristics of the problem dictate the level of detail required to represent the problem. For example, in many analyses of medications, the modeler must represent side effects of the medication. Should a model contain all of the individual potential side effects and their likelihoods of occurring, or can they be grouped into side effects of various severities such as mild (which might only be assumed to change the quality of life of the patient and perhaps decrease medication adherence) and major (which might be assumed to require some form of medical intervention)? One of the best methods to decide the appropriate level of detail is to engage in discussions and collaborations with clinicians who treat the particular condition in question such that the areas of importance to them can be sufficiently detailed. The model itself can sometimes be used to test whether more detail is necessary. Conducting sensitivity analysis on a particular aspect of the model can indicate whether more detail is required. If multiple sensitivity analyses on the parameters of a more aggregated or simplistic section of a model do not have a significant impact on the results, it is not likely that expanding the detail of that section of the model will provide new or important insights.

Time horizon: The time horizon indicates the period of time over which the specific strategies are chosen and the relevant outcomes occur. This time frame is generally determined by the biology of the particular problem. If an analysis is being done comparing different treatments for acute dysuria in young women, the time frame of the analysis may be as short as a week, as long-term sequelae are extremely uncommon in this condition. In contrast, in an analysis of the effects of various interventions to alter cardiovascular risk, the time frame might very likely be the entire lifetime of the patient. It is important to remember that the time frame does not include only those events directly related to the various strategies, but all of the future events implied by choosing each strategy. If a particular intervention increases the risk of a life-changing complication (stroke, heart attack, pulmonary embolism), the long-term effects of the complications need to be taken into account as well.

 

2.2.1 Types of Decision Modeling Techniques

Many methodologies and modeling types can be used to create and evaluate decision models, and the modeler should use the method most appropriate to the particular problem being addressed. The choice is dependent upon the complexity of the problem, the need to model outcomes over extended periods of time, and whether resource constraints and interactions of various elements in the model are required. We will describe in detail the development of simple branch and node decision trees, which set the context for many of the other techniques. A brief review of several methodologies is then provided; more detailed descriptions of many of these techniques can be found in other chapters in this book.

 

2.2.2 Decision Trees

The classic decision analysis structure is the branch and node decision tree, which is illustrated in Figure 2.1.

 

Figure 2.1 Basic structure of a branch and node decision tree, illustrating two choices in a particular clinical situation. After each choice is made, outcomes occur with specific probabilities, these outcomes are associated with values, which may be measured in clinical or cost metrics.

 

The decision tree has several components that are always present and need to be carefully developed. A decision model comprises the modeling structure itself (the decision tree), which represents the decision that is being made and the outcomes that can occur as the result of each decision, the probabilities that the various outcomes will occur, and the values of the outcomes if they do occur. Similar to any other research problem, the decision tree should start with a specific problem formulation, which in the figure is a choice between therapy A and therapy B in a particular condition. In pharmacoeconomic models, these should represent the actual choice being made, and should include the necessary descriptors of the population in which the decision is being made to allow the reader to understand the context of the choice. The context is followed by a decisioode (represented in the figure as a square), and should include as comparators the relevant, real choices the decision maker has at his or her disposal. In the figure, this particular decision has only two choices represented by the branches off the decisioode labeled Choose Therapy A and Choose Therapy B. Each choice is followed by a series of chance nodes (represented in the figure by circles), which describe the possible outcomes that are implied by making each of the respective choices. Each outcome occurs with a specific probability (p1 through p4 in the figure). Each outcome is also associated with one or more values (represented in the figure by the rectangles), which describe the clinical effects and costs of arriving at that particular outcome. We will use this figure in the following description of the basic steps that should be conducted each time a decision analysis or pharmacoeconomic model is developed.

 

2.2.2.1 Steps in Conducting a Decision Analysis

In the following sections, we describe the basic steps through which the modeler should proceed in the construction of a model of a pharmacoeconomic decision. The basic question should be framed and the perspective chosen, the structure of the problem should be developed, the probabilities and values for the outcomes should be estimated, the tree should be analyzed to obtain the expected value of the outcomes, and sensitivity analysis should be conducted to evaluate the effect of assumptions on the results. These are not necessarily linear; often evaluation of the tree or sensitivity analysis will indicate that a particular part of the structure of the model needs either more or less detail. Often, several of these steps are cycled through many times during the development of a model. We illustrate a specific example of these steps for the development of a published pharmacoeconomic model of the use of low molecular weight heparin as prophylaxis for thromboembolism in patients with cancer on.

 

2.2.2.2 Step 1: FRAME the Question

As in any study design, the modeler must decide several basic details regarding for whom and from whose perspective the decision is being made. Deciding for whom the decision is being made is similar to the development of inclusion and exclusion criteria for a typical randomized controlled trial; the decision problem must specify exactly who would be affected by the decision. The description should be as detailed as necessary to describe the problem at hand, and should specify, if important, the age and gender of the population being studied, the specific disease and comorbid conditions that the patients may have, and the specific treatments or strategies that are being evaluated. Choosing the perspective of the decision maker is also very important, as it determines the appropriate metric in which to measure the outcomes and costs of the analysis. Typical perspectives from which to conduct analysis are society, the payer, or the patient.

 

2.2.2.3 Step 2: STRUCTURE the Clinical Problem

The structuring of the problem entails diagramming the branches and nodes that represent the particular problem being modeled. Several aspects of the process are important to remember. The first is that the choices one makes from the decisioode must be mutually exclusive; one and only one of the choices (branches of the decisioode) can be made. If there are several aspects to the choice, then these aspects should be described as a series of mutually exclusive options, rather than described as sequential or embedded decisions. This is illustrated in Figure 2.2, which describes a decision to treat a particular cancer with surgery, medical therapy, or both, and also investigates the order in which the two therapies are applied. The structure on the top of the panel describes all of the possibilities, but at a decisioode, all of the decisions should be listed as branches of the initial decisioode itself, as in the bottom panel of the figure. This allows for a comparison between all of the specific choices individually, and allows for direct comparisons across each of the choices. However, the appropriate construction for chance nodes is different.

 

Figure 2.2 Embedded decisions. It is very difficult to analyze trees with embedded or sequential decisions, as drawn in the top panel. Each strategy should be its own choice, as shown in the lower panel.

 

For example, Figure 2.3 describes a portion of a model of a surgical therapy that has several possible outcomes; for example, the patient may die or have a major surgical complication, a minor surgical complication, or no surgical complication. In the top panel of Figure 2.3 all possible outcomes are drawn as branches of the root node. As shown, the probabilities of each complication are indicated separately and the probabilities of all four branches must sum to one. If this structure is used it becomes somewhat complicated to conduct sensitivity analysis on the probability of surgical death or major or minor surgical complications. However, if this same tree is drawn as a series of binary chance nodes, as shown in the lower panel of Figure 2.3, sensitivity analysis and the ability to vary prospective probabilities becomes easier. The first chance node indicates whether the patient dies or survives. If the patient survives, whether he or she has a complication or not. If the patient has a complication, it is either a major or minor complication. In this setting, it is much easier to directly model the relationships between complication rates, survival rates, and normal outcomes.

It is important to remember that the structure drawn into a decision tree represents the disease process, treatments, and outcomes that the modeler has decided are important in this particular representation of the disease. Any particular model represents a specific version of the reality that the modeler is trying to represent. The art of modeling is the ability to have the model, as created in software, depict the version of reality that the modeler is hoping to represent.

 

Figure 2.3 Superiority of binary chance nodes. It is generally preferable to make complex chance nodes a sequence of individual binary nodes (bottom panel) rather than a complex multi-branch node (top panel).

 

2.2.2.4 Step 3: Estimate the PROBABILITIES

Once the structure of the decision tree has been developed, the probabilities must be estimated for the various chance nodes in the tree. Modelers can use several sources to find and estimate probabilities for various parameters in a decision model. It is important to understand that the typical hierarchy of evidence-based grading does not necessarily apply to all of the various parameters that are necessary to calibrate a decision analysis or a pharmacoeconomic model. For example, the typical hierarchy for evidence-based medicine ranks randomized controlled trials as the best type of evidence for efficacy. However, Retrospective Database Analysis, randomized controlled trials are very poor at estimating many other types of the parameters that are important in a decision model. For example, the incidence of a particular disease cannot be estimated by a randomized controlled trial, nor can the complication rate of a particular therapy when it is applied in general practice. Therefore, the quality of the evidence that a modeler uses to calibrate a decision model is entirely dependent upon the type of data necessary for a particular parameter in the model. Indeed, parameters on effectiveness of therapy may well be best derived from the reports of randomized controlled trials or meta-analyses of randomized controlled trials, whereas incidence and prevalence data may best come from observational studies and large cohort or administrative database analyses, and medication use data may best come from claims databases maintained by large health insurance plans. The important concept is that a model requires the best unbiased estimates of the specific parameters in the model; these parameters do not need to come from the same source nor do they all need to be of the same type of study or accuracy of data. These sorts of differences can be investigated in sensitivity analysis.

 

2.2.2.5 Step 4: Estimate the VALUES of the Outcomes

Similar to estimating the probabilities of various events, the modeler needs to assess the values for the outcomes that occur as a consequence of each one of the choices. The appropriate outcome measure will have previously been determined in the framing of the question when the perspective of the analysis is decided. This will direct the modeler to choose the appropriate outcome measure for the analysis. For example, in an analysis conducted from a societal point of view, the appropriate outcome measure is usually QALYs. The choice of outcome is also determined by the particular disease the treatment is designed to ameliorate. For example, in a pharmacoeconomic model of a treatment for depression, it may be that the appropriate outcome measure is depression-free days or a similar disease-related outcome metric. In a model of a particular intervention for oral hygiene, the appropriate outcome might simply be the number of cavities avoided. The outcomes used must be those that are clinically relevant to the particular decision makers involved in the decision. One of the advantages of developing a model of a pharmacoeconomic problem is that clinical and cost outcomes may be evaluated and modeled simultaneously. Therefore, in most economic models, the model will simultaneously account for the clinical and cost consequences of each potential decision.

 

2.2.2.6 Step 5: ANALYZE the Tree (Average Out/Fold Back)

The evaluation of the decision tree is conceptually quite simple. The overall goal is to calculate the expected value of the outcomes implied by choosing each branch of the root decisioode. For example, in Figure 2.1 there are two choices: Therapy A and Therapy B. If therapy A is chosen a portion of the population (indicated by p1) will experience Outcome 1, which has a utility U1 and another portion of the population (indicated by p2) will experience Outcome 2, which has a utility U2. Assume the utilities represent life expectancies, then the expected value of choosing Therapy A represents the life expectancy of a cohort of people who would be given that therapy, p1 of them living U1 years, p2 of them living U2 years. Mathematically, the expected value of choosing Therapy A is:

E(Therapy A) = (p1*U1) + (p2*U2)

Similarly, the expected value of choosing Therapy B is:

E(Therapy B) = (p3*U3) + (p4*U4)

The choice that has the highest expected value is then chosen as superior.

Essentially, no matter how complicated the tree becomes, the process of finding the expected value is the same. Starting with the terminal nodes, each chance node is replaced by the expectation of that chance node (the expected value of the outcome at that chance node), and that process is continued until one is left with the expected value of each branch of the initial decisioode. Pragmatically, a modeler is never required to do this calculation by hand; there are several decision analysis software packages that do the analysis and calculations automatically.

 

2.2.2.7 Step 6: TEST ASSUM PTIONS (Sensitivity Analysis)

After the model has been developed, calibrated, and the initial analyses completed, one of the most useful steps in modeling is conducting sensitivity analyses. In its simplest form, the definition of sensitivity analysis is the evaluation of the outcomes of the model for various different levels of one or more input variables. Sensitivity analyses have several purposes. They can be used to “debug” a model to make sure that the model behaves as it is designed to behave. It is often the case that the modeler and the content experts with whom the modeler has developed a model will be able to predict the optimal choice under certain specified conditions. By using basic theoretical principles or knowledge of the given disease process the modeler may be able to make predictions about the direction the value of a particular strategy should move under different assumptions. For example, in a decision between surgical and medical therapy, it seems obvious that the relative value of the medical therapy choice should increase compared with the surgical therapy choice as the mortality from surgery increases. If a sensitivity analysis on surgical mortality is conducted and the expected finding does not occur, this may indicate programming or structural errors in the development of the model.

Another important use of sensitivity analysis is in the determination of which variables in the model have the most impact on the outcomes. This is the traditional use of sensitivity analysis and is the basis for many initial valuations of the stability of a particular decision modeling result over a wider range of underlying assumptions and probabilities. There are many types of sensitivity analyses, the simplest of which is a one-way sensitivity analysis in which the changes in the outcomes are evaluated as the value of a single variable is changed. Slightly more complicated is a two-way sensitivity analysis, which plots the optimal choice implied with various combinations of two different input variables, and a multiway sensitivity analysis is conducted by changing and evaluating the results across many input variables simultaneously. Finally, probabilistic sensitivity analyses are used to test the stability of the results over ranges of variability in the input parameters. We describe a simple sensitivity analysis from published work on.

 

2.2.2.8 Step 7: INTERPRET the Results

Once the analysis has been completed, the stability of the model has been tested with sensitivity analysis, and a modeler is convinced that the model represents the clinical and pharmacoeconomic characteristics of the problem adequately, the results must be interpreted and summarized. It is often the case that a specific answer that the model gives under one particular set of conditions is not the most important attribute of the model itself. Oftentimes, it is the manner in which the answer varies with changes in underlying parameter estimates and underlying probabilities and values for outcomes that are the most interesting aspect of the interpretation of an analysis.

However, most pharmacoeconomic analyses will result in an estimate of a costeffectiveness ratio or similar metric of each choice as its major finding.

 

2.2.3 Markov Models

In a traditional branch and node decision tree, as illustrated in Figure 2.1, the terminal nodes are all single outcomes. For example, the value of the outcome might be measured as a life expectancy and quality-adjusted life expectancy or a cost.

 

However, for any model, the outcomes that are expected to occur after each choice are actually quite complex combinations of events that happen in the lives of the people proceeding down that path. Many times, the intervention being modeled at a decisioode affects the risks of future events, such as heart attacks and strokes in the case of cholesterol-modifying therapy, or might affect the rate of recurrence of a particular event, such as asthma episodes in an analysis of the use of corticosteroids in patients with reactive airway disease. When a model must consider events that occur over time or events that may recur in time, the traditional branch and node structure is an inefficient method for representing these events. Standard decision analytic methods typically use a Markov process to represent events that occur over time. As illustrated in Figure 2.4, a simple decision tree would terminate in single values such as a life expectancy shown in the upper panel of Figure 2.4. However, that life expectancy is actually determined by the average life histories of many people who would proceed down that choice. This can be represented as seen in the lower half of Figure 2.4 by replacing the single life expectancy value with a Markov process that represents the events the modeler wants to detect that occur after the decision is made and certain outcomes occur. A Markov process is simply a mathematical representation of the health states in which a patient might find him- or herself and the likelihood of transitioning between those states. The Markov process itself, when it is evaluated, calculates the average life expectancy of a cohort proceeding through the Markov process.

 

Figure 2.4 Use of Markov processes. The terminal node of a standard decision tree typically represents life expectancy, which is a complex summary of many possible paths and events. These can often be represented by a Markov process, in which the actual events that occur over time are specifically modeled. The dots in the lower model represent the same health states and transitions as outcome 1 for outcomes 2 and 3.

 

2.2.4 Simulation Models

Over the past 10 to 15 years, the decision analytic and pharmacoeconomic investigators have started to rely more on simulation methodologies to create progressively more complicated and clinically realistic models of disease processes and treatments. Although a detailed exposition of these methods is beyond the scope of this chapter, we will briefly describe the three most common simulation methodologies used in current pharmacoeconomic analysis. They differ by their ability to model progressively more complicated clinical situations as well as interactions between individual patients in the model.

 

2.2.4.1 Microsimulation

The term microsimulation has come to represent those models in which individual patients are modeled, one at a time, as they proceed through the model. The advantage of microsimulation is that it eliminates a problem with standard Markov process models in that it releases the assumption of path-independent transition probabilities. Although this is discussed in more detail in Chapter 4, the basic problem is that in standard Markov decision models, transition probabilities are dependent only upon the state the patient is in; information regarding where the patient was in the prior time period is lost. Because only one patient is in the model at any given time in a microsimulation, the patient’s specific history can be recorded and transition probabilities can be made to depend on those variables, allowing for remarkable clinical complexity in the development of a model. There are several examples of the use of microsimulation in the current literature: Freedberg has used microsimulation to evaluate the cost-effectiveness of various treatment and prevention strategies in HIV disease.

 

2.2.4.2 Discrete Event Simulation

One of the problems with many of the modeling systems previously discussed is that they cannot easily model the competition for resources. Therefore, although a decision analysis or a cost-effectiveness analysis might be able to determine that a particular diagnostic or therapeutic strategy should be adopted, these analyses cannot tell whether the resources, delivery systems, geographic constraints, or other problems allow for the optimal strategy to actually be implemented. Discrete event simulation, which was originally developed over 50 years ago by industrial engineering to model production processes in factories, provides the modeler with a set of tools that can represent queues, resource limitations, geographic distribution, and many other physical structures or limitations that constrain the implementation of a particular strategy or therapy.

In health care, discrete event simulation has been used for many years to allow for understanding flows and bottlenecks in operating room scheduling, emergency vehicle distribution and response time, throughput in emergency rooms, and many other resource constraint problems. More recently, as the ability to blend highly detailed clinical data with discrete event simulation models has improved, discrete event simulation has been used to address and evaluate more clinically interesting problems. For example, we have used discrete event simulation to model the U.S. organ allocation process and evaluate the effects of various organ allocation policy changes prior to their implementation.1,6 The advantage of discrete event simulation, in this case, is that it has specific structures to allow for the formation of queues, waiting lists, and arrival of both patients and donated organs.

 

2.2.4.3 Agent-Based Simulation

One of the purposes of making models more complex is to represent more realistic physiological or biological systems. Many components of biological systems act entirely independently and simply respond to their environments based on internal sets of processes that govern their behavior. Cells respond to cytokines, hormones, and other biological signals; organs (the pancreas) respond to levels of hormones (insulin) and a myriad of other factors and signals. Agent-based models, in which each “agent” or component of the model independently contains all of the information it needs to interact with and respond to the actions of the other agents in the model, have been increasingly used to understand and model complex biological systems, from individual cells and organs to populations. One fundamental concept of agent-based models is that the aggregated behavior of multiple individual autonomous agents can replicate and predict very complex social and group behaviors. In the realm of medicine and public health, agent-based models have been used recently in the modeling of epidemics and population reactions to epidemics.

 

2.2.5 Deterministic (Mechanistic) Models

Deterministic models seek to capture and characterize specific biological relationships and causes and effects directly through a series of equations. Some of the first medical problems to be evaluated using deterministic models were what are termed “compartment models” that represented the spread of infectious diseases in a community. Also called “susceptible, infected, recovered” (SIR) models, they have been widely used over the past 50 years to model the effects of interventions, such as quarantines and vaccines, on epidemic and pandemic infections. Basically, the relevant population is divided into compartments, and the flows among those compartments are represented as series of differential equations that are related to both the level and rates of flow of each of the compartments.

More recently, these sorts of models have been used to model physiological processes. At their highest level of abstraction, these models represent physiology and disease as one might see in a physiology textbook, with diagrams that indicate how one hormone or cytokine, or level of some electrolyte or other substance, affects the production and level of another. These typically form feedback loops; examples might be that thyroid stimulating hormone (TSH) is produced in response to low thyroid hormone levels and TSH acts on the thyroid to produce more thyroid hormone. Recent examples of the application of deterministic modeling to health care have been the development of complex systems models of sepsis and injury. More physiologically complex, and more directly applicable to problems in pharmacoeconomics, the Archimedes model of disease uses a very complex system of mathematical and differential equations in the concept of an agent-based model to represent multiple metabolic processes and diseases that include diabetes, heart disease, and some cancers. It has been recently used to compare and evaluate the cost-effectiveness of different strategies for the prevention of diabetes.

 

2.2.6 Summary of Modeling Types

A wide variety of mathematical modeling types are available to the modeler to represent disease, treatments, and costs. There is a tradeoff between complexity of the process being modeled and the type of model that should be used to represent the problem. In general, the simplest modeling technique that accurately represents the components of the problem according to a clinical expert is sufficient. It is our experience that most problems can be addressed with either simple branch or node decision trees or standard Markov process-based state transition models. In the next section, we will illustrate the development and analysis of a simple branch and node decision tree model to evaluate a clinical treatment problem.

 

2.3 Example

To illustrate the seven steps used to conduct a decision analysis, we will use an analysis performed by Aujesky et al. examining the use of low molecular weight heparin as secondary prophylaxis for venous thromboembolism in patients with cancer.

 

2.3.1 Step 1: Framing the Question

Venous thromboembolism frequently occurs in patients with cancer and carries a poor prognosis. In addition, cancer patients who have had an episode of venous thromboembolism are prone to recurrent episodes. Because of this recurrence risk, prolonged use of anticoagulants as secondary prophylaxis has been advocated, typically for 6 months or longer. Recent data suggest that low molecular weight heparin (LMWH) is more effective than warfarin for this patient group, leading to recommendations for LMWH as first line therapy in this clinical scenario. However, the costs of LMWH and the potential need for home nursing to administer daily subcutaneous injections raises questions about whether effectiveness gained through LMWH use is worth its significantly increased cost.

Thus, the question this analysis seeks to answer is: what are the costs and benefits of using LMWH compared with warfarin for secondary prophylaxis of venous thromboembolic disease in cancer patients. In the base case analysis, patient cohorts were 65 years old, based on the mean patient age in studies of cancer-related venous thromboembolism. Because venous thromboembolism can recur throughout the remaining life span of cancer patients, a lifetime time horizon was chosen for the analysis. However, the life expectancy of cancer patients with venous thromboembolism averages only 1–2 years, due to venous thromboembolism itself, the high prevalence of advanced cancer in patients with thromboembolism, and the age of the patient group.

This analysis sought to inform physicians and policy makers about the incremental value, defined broadly, of LMWH use compared with warfarin use. For decisions framed in this fashion, cost and effectiveness metrics should be as comprehensive and generalizable as possible. With this in mind, the analysis took the societal perspective, where costs include both direct medical costs and the costs of seeking and receiving care, and used life expectancy and quality-adjusted life expectancy for the effectiveness measures.

 

Figure 2.5 Basic decision tree for low molecular weight heparin as secondary prevention for cancer-induced thromboembolism. Reproduced with permission.

 

2.3.2 Step 2: Structuring the Clinical Problem

A decision tree model was chosen to depict this problem, based on the relatively short time horizon of the model and the concentration on outcomes related to venous thromboembolism and its treatment. If a longer time horizon or more outcomes had been required to adequately model the problem, another model structure, such as a Markov process, could have been used. The decision tree model is shown in Figure 2.5. This model assumes that all events that are not related to venous thromboembolism or its treatment are unaffected by the choice between LMWH and warfarin.

In the decision tree, the square node on the left depicts the decision to use either LMWH or warfarin. Circular nodes depict chance nodes, where events occur based on their probabilities. All patients are at risk for early complications, whose probabilities differ based on treatment choice. Patients who survive the first 6 months after a venous thromboembolism episode are at risk for later complications. The triangular nodes on the right represent the cost and effectiveness values associated with that particular path through the model. In addition, the model assumes that patients suffering a hemorrhagic stroke had anticoagulation permanently discontinued, with only transient interruption of anticoagulation with noncerebral bleeding, and that a second venous thromboembolic episode resulted in permanent inferior vena cava filter placement.

 

2.3.3 Step 3: Estimate the Probabilities

Probabilities for the model were obtained from a variety of sources. A large clinical trial of cancer patients with venous thromboembolism provided data on mortality, recurrent thromboembolism, and major bleeding associated with LMWH or warfarin use. Anticoagulation-related intracranial bleeding rates, which could not be reliably estimated from single trials, were obtained from a meta-analysis of venous thromboembolism therapy in a wide variety of patient groups; its base case value (9%) was varied over a broad range (5–30%) in sensitivity analyses to account for the possibility of greater risk in cancer patients. Intracranial bleeding risk was assumed to be the same with either anticoagulation regimen. In the model, an estimated 20% of patients receiving LMWH required daily home nursing, and 50% of patients with deep venous thrombosis received outpatient treatment.

 

2.3.4 Step 4: Estimate the Values of the Outcomes

Model outcomes were cost and effectiveness. U.S. Medicare reimbursement data were used to estimate costs for hospitalization, emergency department, physician and home nursing visits, laboratory tests, and medical procedures. Anticoagulant drug costs were 2002 average wholesale prices; base case daily pharmacy costs for LMWH and warfarin averaged $48 and $1, respectively. Costs related to intracranial bleeding and late complications were obtained from medical literature sources. Because the analysis took the societal perspective, patient costs for seeking and receiving care were incorporated into the analysis, including patient transportation expenses for care visits and anticoagulation monitoring and patient time costs related to continuing care needs.

Effectiveness was measured as life expectancy and quality-adjusted life expectancy. Life expectancy was estimated using 6- and 12-month mortality data from randomized trials of secondary venous thromboembolism prophylaxis in cancer patients and longer-term survival data from a cohort study of cancer patients with venous thromboembolism. Quality-adjusted life expectancy was calculated by multiplying quality of life utility values (Patient-Reported Outcomes) for chronic health states by the length of time spent in those states. These utilities were obtained from the medical literature. In addition, decreases in utility from acute complications were accounted for by subtracting days of illness, based on U.S. average hospital length of stay data, from quality-adjusted life expectancy totals.

 

2.3.5 Step 5: Analyze the Tree

Averaging out and folding back the tree results in Table 2.2, the LMWH strategy was more effective than warfarin, whether in terms of life expectancy or quality adjusted life expectancy, while also being nearly twice the cost of the warfarin strategy. Effectiveness differences between strategies translated to about 24 days in the unadjusted life expectancy analysis or about 19 quality-adjusted days in qualityadjusted life expectancy. Two incremental cost-effectiveness ratios resulted, because two effectiveness metrics were used, both of which were more than $100,000 per effectiveness unit gained.

 

2.3.6 Step 6: Test Assumptions (Sensitivity Analysis)

 

Figure 2.6 Tornado diagram of multiple one-way sensitivity analyses of the important variables in the low molecular weight heparin example. Reproduced with permission.

 

In a series of one-way sensitivity analyses, varying parameter values over clinically plausible ranges, individual variation of 11 parameters was found to change base case results by 10% or more. These parameters and the incremental cost-effectiveness ratios resulting from their variation are shown in Figure 2.6 as a tornado diagram, where the range of incremental cost-effectiveness results that occur with variation of that parameter are shown as horizontal bars arranged from the greatest range to the least. Results were most sensitive to variation of parameters at the top of the figure; low values for early mortality with warfarin or high values for early mortality with LMWH caused the LMWH strategy to be dominated, i.e., to cost more and be less effective than the warfarin strategy. Variation of an individual parameter did not cause cost per QALY gained for the LMWH strategy to fall below $50,000. However, when simultaneously varying both early mortality due to LMWH and to warfarin in a two-way sensitivity analysis, cost per QALY gained was < $50,000 if mortality differences between the two agents were > 8%. The LMWH strategy cost < $100,000/QALY gained if the utility for warfarin was <0.93, daily pharmacy cost for LMWH was < $41, or if the early mortality difference between agents was > 3%.

A probabilistic sensitivity analysis was also performed, where all sensitive parameters were varied simultaneously over distributions 1000 times. In this analysis, warfarin was favored in 97% of model iterations if the societal willingness-to-pay threshold was $50,000/QALY, or in 72% when the threshold was $100,000/QALY gained.

 

2.3.7 Step 7: Interpret the Results

The results of this analysis suggest that treatment with LMWH in cancer patients with a history of venous thromboembolism is relatively expensive when compared with warfarin therapy, with gains in effectiveness and decreased costs resulting from fewer early complications with LMWH offset by its much higher pharmacy costs. These results were relatively robust in sensitivity analyses when parameters were varied individually and collectively over clinically reasonable ranges. A key exception was when the cost of LMWH was varied; this agent became more economically reasonable when its daily cost was in the range of $40 or less. Interestingly, in many countries other than the United States, LMWH costs are well below this range ($10–13 per day in Europe and Canada).

Thus, we can conclude that LMWH for secondary prophylaxis of venous thromboembolism in U.S. cancer patients is expensive, calling into question whether the documented improvement in outcomes is worth the added cost. However, the added expense of the newer intervention is largely driven by the cost of the agent itself, making LMWH a much more economically reasonable strategy when (and where) it costs less.

 

4.1 Markov Modeling in Decision Analysis

A pharmacoeconomic problem is attacked using a formal process that begins with constructing a mathematical model. In this book a number of pharmacoeconomic constructs are presented, ranging from spreadsheets to sophisticated numerical approximations to continuous compartment models. For more than 40 years the decision tree has been the most common and simplest formalism, comprising choices, chances, and outcomes. The modeler crafts a tree that represents near-term events within a population or cohort as structure, and attempts to balance realism and attendant complexity with simplicity. In problems that lead to long-term differences in outcome, the decision model must have a definite time horizon, up to which the events are characterized explicitly. At the horizon, the future health of a cohort must be summed and averaged into “subsequent prognosis.” For problems involving quantity and quality of life, where the future natural history is well characterized, techniques such as the Declining Exponential Approximation of Life Expectancy or differential equations may be used to generate outcome measures. Life tables may be used directly, or the results from clinical trials may be adopted to generate relevant values. Costs in decision trees are generally aggregated, collapsing substantial intrinsic variation into single monetary estimates.

Most pharmacoeconomic problems are less amenable to these summarizing techniques. In particular, clinical scenarios that involve a risk that is ongoing over time, competing risks that occur at different rates, or costs that need to be assessed incrementally lead to either rapidly branching decision trees or unrealistic pruning of possible outcomes for the sake of simplicity. In these cases a more sophisticated mathematical model is employed to characterize the natural history of the problem and its treatment. Dasbach, Elbasha, and Insinga reviewed the types of models used in the human papillomavirus (HPV) vaccination problem, and identified cohort, population dynamic, and hybrid approaches. This chapter explores the pharmacoeconomic modeling of cohorts using a relatively simple probabilistic characterization of natural history that can substitute for the outcome node of a decision tree. Beck and Pauker introduced the Markov process as a solution for the natural history modeling problem in 1983, building on their and others’ work with stochastic models over the previous 6 years.4 During the ensuing 25 years, more than 1,000 articles have directly cited either this paper or a tutorial published a decade later, and more than 1,700 records in PubMed can be retrieved using (Markov decision model) OR (Markov cost-effectiveness) as a search criterion. This chapter will define the Markov process model by its properties and illustrate its use in pharmacoeconomics by exploring a simplified HPV vaccination example.

 

4.2 The Ma rkov Process and Transition Probabilities

4.2.1 Stochastic Processes

A Markov process is a special type of stochastic model. A stochastic process is a mathematical system that evolves over time with some element of uncertainty. This contrasts with a deterministic system, in which the model and its parameters specify the outcomes completely. The simplest example of a stochastic process is coin flipping. If a fair coin is flipped a number of times and a record of the result kept (H = “heads”; T = “tails”), a sequence such as HTHHTTTHTHHTHTHTHHTHTHTTTT might arise. At each flip (or trial), either T or H would result with equal probability of one half. Dice rolling is another example of this type of stochastic system, known as an independent trial experiment. Each flip or roll is independent of all that have come before, because dice and coins have no memory of prior results. Independent trials have been studied and described for nearly 3 centuries.

 

4.2.2 Markov Processes

The Markov process relaxes this assumption a bit. In a Markov model the probability of a trial outcome varies depending on the current result (generally known as a “state”). Andrei Andreevich Markov, a Russian mathematician, originally characterized such processes in the first decade of the 20th century. It is easy to see how this model works via a simple example. Consider a clerk who assigns case report forms to three reviewers: Larry, Maureen, and Nell. The clerk assigns charts to these readers using a peculiar method. If the last chart was given to Larry, the clerk assigns the current one to Larry, Maureen, or Nell with equal probability. Maureeever gets two charts in a row; after Maureen, the clerk assigns the next chart to Larry with probability one quarter and Nell three quarters. After Nell gets a chart, the next chart goes to Larry with probability one half, and Nell and Maureen one quarter. Thus, the last assignment (Larry, Maureen, or Nell) must be known to determine the probability of the current assignment.

 

4.2.2.1 Transition Probabilities

Table 4.1 shows this behavior as a matrix of transition probabilities. Each cell of Table 4.1 shows the probability of a chart’s being assigned to the reviewer named at the head of the column, if the last chart was assigned to the reviewer named at the head of the row. An nXn matrix is a probability matrix if each row element is nonnegative, and each row sums to 1. Because the row headings and column headings refer to states of the process, Table 4.1 is a special form of probability matrix—a transition probability matrix.

This stochastic model differs from independent trials because of the Markov Property: the distribution of the probability of future states of a stochastic process depends on the current state (and only on the current state, not the prior natural history). That is, one does not need to know what has happened with scheduling in the past, only who was most recently assigned a chart. For example, if Larry got the last review, the next one will be assigned to any of the three readers with equal probability.

 

4.2.2.2 Working with a Transition Probability Matrix

The Markov property leads to some interesting results. What is the likelihood that, if Maureen is assigned a patient, that Maureen will get the patient after next? This can be calculated as follows:

After Maureen, the probability of Larry is one quarter and Nell three quarters. After Larry the probability of Maureen is one third, and after Nell it is one quarter. So, the probability of Maureen–(anyone)–Maureen is one quarter × one third + three quarters × one quarter, or 0.271. A complete table of probabilities at two assignments after a known one is shown in Table 4.2. This table is obtained using matrix multiplication, treating Table 4.1 as a 3 × 3 matrix and multiplying it by itself (Matrix multiplication can be reviewed in any elementary textbook of probability or finite mathematics, or at http://en.wikipedia.org/wiki/Matrix_multiplication). Note that the probability of Maureen’s going to Maureen in two steps is found in the corresponding cell of Table 4.2.

This process can be continued, because Table 4.2 is also a probability matrix, in that the rows all sum to 1. In fact, after two more multiplications by Table 4.1, the matrix is represented by Table 4.3.

The probabilities in each row are converging, and by the seventh cycle, after a known assignment the probability matrix is shown in Table 4.4. This is also a probability matrix, with all of the rows identical, and it has a straightforward interpretation. Seven or more charts after a known assignment, the probability that the next chart review would go to Larry is 0.380, to Maureen 0.225 and to Nell 0.394. Or, if someone observes the clerk at any random time, the likelihood of the next chart’s going to Larry is 0.380, etc. This is the limiting Markov matrix, or the steady state of the process. This particular scheduler, despite the idiosyncratic behavior, gives a little less than 40% of the charts each to Larry and Nell over time, and assigns Maureen only 22.5%.

 

4.2.3 Absorbing Markov Models

The chart review example is known as a regular Markov chain. The transition probabilities are constant, and depend only on the state of the process. Any state can be reached from any other state, although not necessarily in one step (e.g., Maureen cannot be followed immediately by Maureen, but can in two or more cycles). Regular chains converge to a limiting set of probabilities. The other principal category of Markov models is absorbing. In these systems the process has a state that it is possible to enter, in a finite set of moves, from any other state, but from which no movement is possible. Once the process enters the absorbing state, it terminates (i.e., stays in that state forever). The analogy with clinical decision models is obvious; an absorbing Markov model has a state equivalent to death in the clinical problem.

 

4.2.3.1 Behavior of the Absorbing Model

This is shown in Figure 4.1, a simplified three-state absorbing clinical Markov model. In a clinical model the notion of time appears naturally. Assume that a clinical process is modeled where definitive disease progression is possible, and that death often ensues from progressive disease. At any given month the patient may be in a Well state, shown in the upper left of Figure 4.1, the Progressive state in the upper right, or Dead in the lower center. If in the Well state, the most likely result for the patient is that he or she would remain well for the ensuing month, and next be found still in the Well state. Alternatively, the patient could become sick and enter the Progressive state, or die and move to the Dead state. If in Progressive, the patient would most likely stay in that state, but could also die from the Progressive state, presumably at a higher probability than from the Well state. There is also a very small probability of returning to the Well state.

Figure 4.1 Simple three-state absorbing Markov model.

 

A possible transition probability matrix for this model is shown in Table 4.5. In the upper row a Well patient remains so with probability 80%, has a 15% chance of having progressive disease over one cycle, and a 5% chance of dying in the cycle. A sick patient with progressive disease is shown with a 2% chance of returning to the Well state, a 28% chance of dying in 1 month, and the remainder (70%) staying in the Progressive state. Of course, the Dead state is absorbing, reflected by a 100% chance of staying Dead.

Table 4.5 is a probability matrix, so it can be multiplied as in the prior example. After two cycles the matrix is shown in Table 4.6.

Thus, after two cycles of the Markov process, someone who started in the Well state has slightly less than a twothirds chance of staying well, and a 22.5% chance of having Progressive disease. By the 10th cycle, the top row of the transition matrix is:

So, someone starting well has a 75% chance of being dead within 10 cycles, and of the remaining 25%, roughly an even chance of being well or having Progressive disease. This matrix converges slowly because of the moderate probability of death in any one cycle, but eventually this matrix would end up as a set of rows:

Everyone in this process eventually dies.

Clinical Markov models offer interesting insights into the natural history of a process. If the top row of the transition matrix is taken at each cycle and graphed, Figure 4.2 results.

Figure 4.2 Absorbing Markov chaiatural history.

 

This graph can be interpreted as the fate of a cohort of patients beginning together at Well. The membership of the Well state decreases rapidly, as the forward transitions to Progressive and Dead overwhelm the back transition from Progressive to Well. The Progressive state grows at first, as it collects patients transitioning from Well, but soon the transitions to Dead, which, of course, are permanent, cause the state to lose members. The Progressive state peaks at Cycle 4, with 25.6% of the cohort. The Dead state actually is a sigmoid (S-shaped) curve, rising moderately for a few cycles because most people are Well, but as soon as the 28% mortality from the Progressive state takes effect, the curve gets steeper. Finally it flattens, as few people remain alive. This graph is typical of absorbing Markov process models.

 

4.2.3.2 Use of Absorbing Markov Models in Clinical Decision Analysis

The Markov formalism can substitute for an outcome in a typical decision tree. The simplest outcome structure is life expectancy. This has a natural expression in a Markov cohort model: Life expectancy is the summed experience of the cohort over time. If we assign credit for being in a state at the end of a cycle, the value of each state function in Figure 4.2 represents the probability of being alive in that state in that cycle. At the start of the process, all members of the cohort are in the Well state. At Cycle One (Table 4.5), 80% are still Well and 15% have progressive disease, so the cohort would have experienced 0.8 average cycles Well, and 0.15 cycles in Progressive disease. At Cycle Two (Table 4.6), 64.3% are Well and 22.5% have Progressive disease. So, after two cycles, the cohort experience is 0.8+0.643, or 1.443 cycles Well and 0.15+0.225, or 0.375 cycles in Progressive disease. Summing the process over 45 cycles, until all are in the Dead state, the results are 4.262 cycles Well and 2.630 cycles in Progressive. So the life expectancy of this cohort, transitioning according to the probability matrix in Table 4.5, is 6.892 cycles, roughly 2:1 in Well versus Progressive disease. Refinements to this approach, involving correction for initial state membership, can be found in Sonnenberg and Beck.

Whereas a traditional outcome node is assigned a value, or in Chapters 9 or 11 a utility, the Markov model is used to calculate the value by summing adjusted cohort membership. For this to work, each Markov state is assigned an incremental utility for being in that state for one model cycle. In the example above, the Well state might be given a value of 1, the Progressive state a value of 0.8. That is, the utility for being in the Progressive state is 80% of the value of the Well state for each cycle in it. In most models Dead is worth 0. Incremental costs can also be applied for Markov cost-effectiveness or cost-utility analysis. For this tutorial example, assume the costs of being in the Well state are $5000 per cycle, and in the Progressive state $8000 per cycle. Summing the cohort over 45 cycles leads to the results in Table 4.7. Thus, in this tutorial example, the cohort can expect to survive 6.892 cycles, or 6.366 quality-adjusted cycles, for a total cost of more than $42,000. These values would substitute for the outcomes at the terminal node of a decision tree model, and could be used for decision or cost-utility analysis.

An alternative way to use a Markov model is to simulate the behavior of a cohort of patients, one at a time. This approach is known as a Monte Carlo analysis. Each patient begins in the starting state (Well, in this example), and at the end of each cycle the patient is randomly allocated to a new state based on the transition probability matrix. Life expectancy and quality adjustments are handled as in the cohort solution. When the patient enters the Dead state, the simulation terminates and a new patient is queued. This process is repeated many times, and a distribution of survival, quality-adjusted survival, and costs results. Modern approaches to Monte Carlo analysis incorporate probability distributions on the transition probabilities, to enable statistical measures such as mean and variance to be determined.

Two enhancements to the Markov model render the formalism more realistic for clinical studies; both involve adding a time element. First, although the Markov property requires no memory of prior states, it is possible to superimpose a time function on a transition probability. The most obvious example of this is the risk of death, which rises over time regardless of other clinical conditions. This can be handled in a Markov model by modifying the transition probability to death using a function: in the tutorial example time could be incorporated as p (Well->Dead) = 0.05 + G(age), where G represents the Gompertz mortality function9 or another well-characterized actuarial model.

Second, standard practice in decision modeling discounts future costs and benefits to incorporate risk aversion and the decreasing value of assets and events in the future. Discounting (see Chapter 10) may be incorporated in Markov models as simply another function that can modify (i.e., reduce) the state-dependent incremental utilities.

 

4.3 Markov Model Example: Cervical Cancer

Figure 4.3 depicts a simplified model of the progression from mild CIN 1 to invasive cervical cancer or recovery to normal. This model and its attendant data are drawn from the Goldie et al. study of the costs and projected benefits of an HPV vaccine (2004), to which the reader is referred for the complete model and cost-effectiveness analysis. For this chapter the model and data are simplified in favor of didactic value.

Figure 4.3 Principal transitions in Cervical Cancer Model. Transitions to same state (e.g., Well–Well) not shown.

 

In Figure 4.3 states are represented for Well, persistent HPV infection, CIN 1, CIN 2,3, invasive cervical CA, and death. For clarity, arrows from states to themselves have not been drawn, and a few rare transitions and non-cancer deaths are also omitted. The figure thus depicts the principal transitions in the model. The largest state-to-state transition is from CIN 1 to HPV. The basic 1-year cycle transition probability matrix for a 35-year-old woman is presented in Table 4.8. In this table the baseline or favorable range estimates from Goldie et al. are used.

Note that from CIN 1, the most likely transition is to HPV, although remaining in CIN 1 is also frequent. Cervical Cancer (CVX CA) is reached only from CIN 2,3 and has an annual risk of death of 3.5%. If this table were used as depicted, both Well and Dead would be absorbing Markov states. Therefore, a time-dependent general population risk of death must be added. At 35, the annual risk of death is 0.17%, rising annually according to the Gompertz exponential function. At 84, the risk of death is 10%. Table 4.9 shows the experience over 10 years of 10,000 women aged 35 with CIN 1, according to the Markov model with the rising general death rate. In 1 year many women are well or have persistent HPV; none has cervical cancer because the model forces a transition to CIN 2,3 beforehand. CVX CA begins to appear at 37 and rises slowly.

Over an expected lifetime, the Markov model yields a probability of being in each state as shown in Figure 4.4. The Well cohort rises rapidly, and falls slowly as the natural death rate rises over time. The Well and Dead cohorts show the typical sigmoid functions. CIN 2,3 peaks at age 40, whereas CVX CA peaks at age 60 (149 prevalent cases). This is due to the relatively small excess mortality of CVX CA, and the structural assumptions in the Markov model in Figure 4.3 that has a patient remain in the CVX CA state until death. New cases of cervical cancer peak at age 41. One could extend the model by incorporating a state reflecting long-term survival from cervical cancer, but this would necessitate keeping track of how long each cohort member had had the cancer diagnosis. Modeling software can handle such issues, but the stochastic process becomes a semi-Markov model with attendant complexity.

Figure 4.4 Natural history of CIN 1 example.

 

Baseline results from this model are presented in Table 4.10. Averaged over a cohort, the patient with CIN 1 in this model can expect to live 1.63 years in that state, 33 years well, 3.56 years with persistent HPV, 3 years with CIN 2,3, and 0.55 years with CVX CA. Of course, no single patient has precisely this experience. A Monte Carlo simulation of 10,000 patients shows that the average number of cancer cases in this cohort is 356, with 95% of the simulations ranging between 335 and 378.

Sensitivity analysis can be conducted on Markov transition probabilities, and modern software easily supports this. A linked sensitivity analysis, moving probability estimates to the upper end (worst case) of their ranges, generates the results found in Table 4.11. In this formulation the transition from CIN 1 to CIN 2,3 is much higher, the 5-year survival from CVX CA is 63% vs. a baseline of 84%, and the transition from CIN to cancer is doubled. Monte Carlo analysis shows a mean 1147 invasive cancers (95% range 1126 to 1161).

Goldie et al.’s more complete Markov formulation incorporates quality adjustments, effects of screening, and a primary focus on the role of vaccination to prevent persistent HPV and resulting CIN and downstream sequelae. It also has an extensive cost model. Later chapters in this text will illustrate how costs and structural interventions can modify Markov and other stochastic models to generate sophisticated analyses of pharmacoeconomic problems.