HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Abedin, P. Articles by Khan, K. S Search for Related Content PubMed Articles by Abedin, P. Articles by Khan, K. S

Health technology assessment in obstetrics and gynaecology. Part 1: an overview of the process

^1. Birmingham Women's Hospital, Metchley Park Road, Edgbaston, Birmingham B15 2TG, UK Email: Pabedin{at}doctors.org.uk (corresponding author)
^2. Birmingham Women's Hospital, UK
^3. Birmingham Women's Hospital, UK

	Abstract

TOP Abstract Introduction Systematic reviews and meta... Randomised controlled trials Health economic evaluation References

 
Key content:

• A health technology is any method used to promote health; diagnose, prevent or treat disease; or improve long-term rehabilitation and care.
  
• Systematic reviews are summaries of the total evidence on a defined subject, using explicit techniques and critical appraisal of the relevant primary research to generate recommendations for practice and policy.
  
• ‘Bias’ exaggerates or underestimates the true effect of an intervention or exposure. There are several possible types of bias such as selection, performance, measurement and attrition.
  

Learning objectives:

• To learn about the role of health technology assessments (HTAs) in the formulation of health policy and the adoption of new technologies.
  
• To understand the methodology involved in conducting systematic reviews and economic evaluations in HTA.
  
• To learn about the role of institutions (for example, NICE) in utilising HTA to formulate health policy.
  

Ethical issues:

• Systematic reviews are the highest level of evidence as they incorporate the results of multiple studies but can be biased if individual studies are not published.
  
• Meta-analysis should only be conducted when there is sufficient heterogeneity (decided by statistical tests) between the studies under consideration in a systematic review.
  
• The investigators in a clinical trial should be transparent in any shortcomings of conduct or design that can question the conclusions.
  

Please cite this article as: Abedin P, Daniels JP, Khan KS. Health technology assessment in obstetrics and gynaecology. Part 1: an overview of the process. The Obstetrician & Gynaecologist 2007;9:109–115.

Keywords bias / clinical trials / health technology assessment / meta-analysis / systematic reviews

	Introduction

TOP Abstract Introduction Systematic reviews and meta... Randomised controlled trials Health economic evaluation References

 
Have you ever wondered how evidence-based health policy is made? Or what steps decision makers should take to support and enhance evidence-based practice? The process they go through is called health technology assessment (HTA). This is an internationally recognised term that covers any method used to promote health, prevent and treat disease and improve long-term rehabilitation and care.¹ ‘Technologies’ in this context are not confined to new drugs or pieces of sophisticated equipment but include procedures, devices, settings of care and screening programmes.

The evaluation of these technologies can be done through primary research, systematic reviews and model based syntheses of the scientific evidence. It is concerned with technologies and their costs in their widest sense and the judicious use of the information for policy and decision making, thereby showing the way forward for incorporating evidence into practice.

HTA considers the effectiveness, appropriateness and cost of technologies by asking four fundamental questions:

• Does the technology (treatment, diagnosis, surgical procedure, etc) work?
  
• For whom does it work?
  
• What is the cost?
  
• How does it compare with alternatives?
  

Two of the many national agencies involved in HTA are of particular interest: the National Institute for Health and Clinical Excellence (NICE) and the National Horizon Scanning programme. NICE is the independent organisation responsible for providing national guidance on the promotion of good health and the prevention and treatment of ill health in the UK. It does this by producing guidelines on public health, health technologies and clinical practice within the National Health Service (NHS) using the HTA process described in this article. NICE is an independent agency set up by the government that aims to promote clinical and cost effectiveness through the production of national guidance and audit. New and emerging technologies are assessed and all stakeholders, including industry, professional and patient groups are invited to participate in the decision making process before a health policy or guidance is decided upon. Once clinical guidance is published, health professionals are expected to take such recommendations into account when making decisions on the individual care of the patient.

National Horizon Scanning is an early warning system to identify new technologies that have the potential to impact on health services. It is designed to identify significant and urgent advances regardless of clinical specialty and includes regular scanning of primary, secondary and tertiary information sources. Advance notice is provided to the Department of Health about new and emerging technologies that need consideration of clinical and cost effectiveness, safety and efficacy and the cost impact about 2 to 3 years before their launch.

A quality HTA will use skills in systematic review techniques, health economics, statistics, modelling, clinical intuition and health expertise (Box 1). We provide an overview of some of these below.

View larger version (67K): [in this window] [in a new window]   Box 1 The 10 steps in health technology assessment2

	Systematic reviews and meta-analyses

TOP Abstract Introduction Systematic reviews and meta... Randomised controlled trials Health economic evaluation References

Steps in conducting a systematic review
Formulating the question
There are several key components to a well formulated question. A clearly defined question should specify the population under study, the intervention(s) or exposures and the types of outcomes that are of interest. In addition, the types of study designs that are relevant to answering the question should be specified.

Study designs are mainly divided into two broad categories:⁴

1. Experimental studies
   In this type of study, decisions concerning the allocation of participants to different intervention groups are under the control of an unbiased third party, not the investigator. The proportion of participants receiving the intervention is proscribed by the investigator.
   • Randomised controlled trial (RCT). This comprises a group of designs using methods whereby participants are randomly allocated to intervention or control groups, with a comparison of outcomes during an appropriate follow-up period. Randomisation avoids systematic biases because both known and unknown determinants of outcome are, on average, evenly distributed between intervention and control groups. However, selection biases can be introduced unless the group allocation is concealed until the participant is committed to the trial.
     
   
   
2. Observational studies
   These are studies in which natural variation in interventions or exposures among study participants is investigated to explore the effects of the interventions or exposures on health outcomes.
   • Cohort: a prospective comparison of outcomes following exposure and non-exposure to interventions.
     
   • Quasi-experimental: studies in which the allocation of participants to different intervention groups is controlled by the investigator but the method falls short of genuine randomisation, either because the allocation is not random (for example, alternating) or because it can be predicted. Although one can consider such studies to be experimental, they suffer from all the biases that threaten validity of observational studies, therefore, they should be interpreted with caution.
     
   • Case control: usually a retrospective comparison of exposure to interventions between participants with the outcome (cases) and those without the outcome (controls).
     
   • Cross-sectional: an examination of the relationship between disease and other variables of interest as they exist in a defined population at one particular time.
     
   • Before-and-after: the comparison of findings in study participants before and after an intervention.
     
   • Case series: the description of a number of cases of an intervention and outcome (without comparison with a control group).
     
   
   

Randomised controlled trials are considered the most reliable source of data when addressing questions regarding therapeutic efficacy, whereas other study designs are appropriate for addressing other types of questions. Questions relating to aetiology or risk factors can be addressed by case control and cohort studies. Questions relating to accuracy of tests are best addressed by cross-sectional design.

Searching the literature
The precision and validity of the findings of reviews are directly related to the comprehensiveness of the literature identification process. All important databases of reviews should be searched, along with conference proceedings and foreign language journals. To ensure there is no publication bias, unpublished papers should be searched for by contacting authors or searching in databases such as the SIGLE (System for Information on Grey Literature in Europe). The correct search term, incorporating the population, interventions, outcomes and study designs relevant to the review should be used.

Assessing the quality of the literature
The quality of the studies included in a systematic review will ultimately determine the impact it has and how reliable are its conclusions. This is dependent on the degree to which the included studies minimise bias and error in their design, conduct and analysis. Bias either exaggerates or underestimates the ‘true’ effect of an intervention or exposure. There are several possible types of bias, such as selection, performance, measurement and attrition.⁴

• Selection bias denotes systematic differences between comparison groups in prognosis or responsiveness to treatment. This can be minimised by randomisation and concealment of allocation from care providers and participants.
  
• Performance bias is difference in care provided apart from the intervention under evaluation. This is minimised by blinding.
  
• Measurement bias is where there are differences between comparison groups in how outcomes are assessed. This can also be reduced by adequate blinding and using objective outcomes where possible.
  
• Attrition bias denotes systematic differences between comparison group withdrawals from the study and can be addressed by the ‘intention to treat’ analysis: participant outcomes are analysed according to their initial group allocation, regardless of whether they complied fully with the intervention, changed their intervention group or dropped out of the study.
  

There are published quality assessment instruments or scales that can be used to weigh studies according to their methodological rigour. These are usually in the form of checklists.^5–6 One such scale was developed by Jadad,⁷ who validated a quality rating scale to evaluate RCTs, comprising adequacy of randomisation, double-blinding and description of withdrawals and dropouts.⁸ Other scales are also available, although some experts are of the opinion that, instead of using summary scores, it is preferable to use individual components of methodological quality to explore their influence upon the overall summary measure. The essential components of a quality RCT are discussed in more detail later in this article.

Summarising the evidence
A descriptive summary of the findings of studies included in a review is done by presenting the information about the study characteristics, their design and quality and their effects. A decision to carry out a meta-analysis, which is a statistical combination of results from two or more separate studies, must then be made. This increases the power or the chance of detecting a real effect as statistically significant if it exists. It also improves precision⁸ and answers questions not posed by the individual studies. However, if studies are clinically diverse, then a meta-analysis can be meaningless.

Meta-analysis should only be carried out when a group of trials is sufficiently homogenous in terms of participants, interventions and outcomes to provide a meaningful summary, although techniques are available to accommodate a degree of heterogeneity into meta-analyses.

Interpreting the results
The results of the individual studies and of the meta-analysis should be tabulated. The standard graphical output used in most systematic reviews, including Cochrane reviews, is the forest plot,⁹ although other formats are available.

The forest plot displays effect estimates and confidence intervals for both individual studies and meta-analysis (Figure 1): the area of the block indicates the weight assigned to the study in the meta-analysis, while the horizontal line depicts the confidence interval (this is usually 95%). If the summary effect estimates lie to the side of the vertical line denoting the intervention arm, it means there is a benefit of the experimental intervention. If, however, they lie towards the side of the control arm, there is no benefit. If the 95% confidence interval crosses the vertical line, there is not a statistically significant overall benefit of one intervention over the other.

View larger version (32K): [in this window] [in a new window]   Figure 1 Forest plot of relative risk of stroke9

	Randomised controlled trials

TOP Abstract Introduction Systematic reviews and meta... Randomised controlled trials Health economic evaluation References

 
There will be circumstances where there is insufficient previous research to synthesise in a meta-analysis, or that those that exist have been of poor quality or too disparate to try. Evidence must come from studies where biases and random errors have been minimised, which requires large scale RCTs. The methodology of RCTs should be relatively familiar;^10–12 however, the nuances needed to be considered when running or assessing a trial of an intervention in women's health are briefly discussed here.

The main advantage of randomisation is that it ensures that the groups of participants are comparable at the beginning of the trial, so that any difference seen later can be attributable either to the effect of the intervention or pure chance. However, the integrity of randomisation can be broken down if the randomisation sequence can be predicted (for example, by alternative or repeating allocations) or subverted. Once committed to the trial, participants should not be withdrawn in the hope of receiving a more favourable allocation on a second attempt. An independent randomisation service avoids the temptation to cheat. Furthermore, once allocated a treatment, the participants should be followed up and analysed in that group, irrespective of whether they actually received or completed the treatment course. Failure to perform this ‘intention to treat’ analysis negates all the gains of randomisation and can significantly bias results. Periodic checking of the baseline balance of prognostic factors between groups throughout the trial provides reassurance that strict randomisation is maintained, although performing statistical tests on multiple variables will produce significant differences through chance alone, so they should not be applied.

A clear definition of the eligibility criteria will establish the nature of the population under study, allowing clinicians to consider how applicable the evidence is to an individual patient situation. The calculated number of participants needed to demonstrate or refute an effect of an intervention should also be clearly stated to reassure the reader that the trial was large enough to detect a meaningful and realistic difference between the groups. Inevitably, not all participants will complete the trial, but dropouts and non-compliance should not be hidden. The CONSORT diagram is the favoured method of demonstrating the flow of participants through a trial.

A thorough discussion of the pitfalls of analysis is beyond the scope of this article: suffice to say there are many ways of distorting the results of the trial through inappropriate statistical analysis. Undue emphasis on particular subgroups should be regarded as suspicious, as such analyses will be underpowered and likely to produce significant results by chance. Ideally, results should be reported as an estimate of the likely treatment effect with a measure of variability, usually a 95% confidence interval, which allows the reader to gauge the clinical significance of the intervention.

	Health economic evaluation

TOP Abstract Introduction Systematic reviews and meta... Randomised controlled trials Health economic evaluation References

 
Clinicians often question why an economic evaluation is important. There are limited resources that can be spent on the healthcare system in most countries. The same is true for the NHS and this has been a much debated topic, with the observation that, in spite of increasing NHS budget allocations, the system still does not seem to be producing the desired results. In this atmosphere of demanding value for money, it is particularly important that the health programmes that are introduced are effective and efficient, with results superior to the existing alternatives.

An economic evaluation of a health technology is the comparative analysis of alternative courses of action by identifying, measuring, valuing and comparing both their costs and consequences. The latter information is obtained from systematic reviews and primary research, including RCTs.

There are essentially four types of economic evaluation.¹³ They differ with respect to the nature of the consequences being examined, although the method of cost evaluation remains essentially the same.

Cost minimisation: it is assumed that the consequences of alternative treatment options are the same and the least costly alternative is then adopted. An example would be day surgery versus inpatient surgery for the same procedure. It is assumed that the result of the surgery would be the same but the cost would be different, depending on whether the patient is a day or inpatient.

Cost effectiveness analysis: the cost is related to a single common effect, which can differ in magnitude between the alternative programmes. An example is a renal programme in which the cost effectiveness of renal transplantation is compared with that of dialysis. Here, the unit of comparison would be the cost per life-year saved.

Cost–benefit analysis: the outcomes of interest can be multiple and they need to be considered together to do a full evaluation. With reference to the above example, if home dialysis is also compared with hospital dialysis and quality of life, and medical complications considered in addition to the years of life gained, to do a cost–benefit analysis we would need to compute cost effectiveness ratios for the three effects.

Cost–utility analysis: utilities are employed as a measure of the value of health programmes. This is done is by calculating the quality-adjusted life-year (QALY), which takes into account both quality and quantity of life generated by healthcare interventions.¹⁴

A QALY places a weight on time in different health states. A year of perfect health is 1; however, a year of less than perfect health is less than 1. Death is considered to be 0; however, some health states can be considered worse than death and can have negative scores.

The cost–utility ratio is, thus, the additional cost required to generate a year of perfect health (one QALY). Comparisons can be made between interventions and priorities can be established based on those interventions that are relatively inexpensive (low cost per QALY) and those that are relatively expensive (high cost per QALY).

There are various instruments used to produce health utility values leading to a calculation of a QALY.¹⁵ These include the SF-36 (sickness impact profile/Nottingham health profile) and the EQ-5D (a health-related quality of life measure which includes social preferences) (Box 2).

View larger version (65K): [in this window] [in a new window]   Box 2 Using the EQ-5D

The main types of models are decision tree analyses (Figure 2) and Markov models (Figure 3).¹⁷ In the former, the ‘decision tree’ is structured around probabilities and outcomes, the latter being quantified in the form of cost, utilities, life-years and QALYs. Markov models on the other hand, represent disease processes that evolve over time. They assume a variety of health ‘states’ that individuals can have at a given time and that they move from one state to the other with a given probability. They are used to estimate long-term costs and life-years or QALYs gained.

View larger version (34K): [in this window] [in a new window]   Figure 2 Decision tree analysis of the use of erythropoetin (EPO) to treat anaemia

View larger version (27K): [in this window] [in a new window]   Figure 3 Markov model in decision analytic modelling

The theme is to be continued in the next issue, in which we will elaborate on health technology assessment as it is applied to obstetrics and gynaecology.

	References

TOP Abstract Introduction Systematic reviews and meta... Randomised controlled trials Health economic evaluation References

 

1. National Coordinating Centre for Health Technology Assessment. The NHS Health Technology Assessment Programme. [www.hta.nhsweb.nhs.uk/aboutHTA.htm].
2. Stevens A, Milne R. Evaluating health care technologies. In: Pencheon D, Guest C, Melzer D, Gray J M, editors. Oxford Handbook of Public Health Practice. 2nd ed. Oxford: Oxford University Press; 2006. p. 300–7.
3. Khan KS, Kunz R, Kleijnen J, Antes G. Systematic Reviews to Support Evidence-Based Medicine. London: Royal Society of Medicine Press Ltd; 2003
4. Khan KS, ter Riet G, Glanville J, Sowden AJ, Kleijnen J, editors. Undertaking Systematic Reviews of Research on Effectiveness: CRD's Guidance for Carrying Out or Commissioning Reviews. 2nd ed. York: NHS Centre for Reviews and Dissemination (CRD), University of York; 2001. CRD Report No. 4.
5. Shea B, Dube C, Moher D. Assessing the quality of reports of systematic reviews: the QUORUM statement compared to other tools. In: Eggar M, Smith GD, Altman DG, editors. Systematic Reviews in Healthcare, Meta-Analysis in Context. 2nd ed. London: BMJ Books; 2001. p. 122–3.
6. Shea B. Assessing the Quality of Reporting in Meta-Analysis of Randomised Control Trials. MSc thesis: University of Ottawa; 1999
7. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:1–12. doi:10.1016/0197-2456(95)00134-4[Medline]
8. Juni P, Witschi A, Bolsch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:1054–60. doi:10.1001/jama.282.11.1054[Abstract/Free Full Text]
9. Higgins JPT Green S, editors, Interpreting results. In: Cochrane Handbook for Systematic Reviews of Interventions 4.2.5. Chichester, UK: John Wiley & Sons, Ltd; 2005. [updated May 2005].
10. Daniels J, Hills RK. Designing clinical trials in women's health. Reviews in Gynaecological and Perinatal Practice 2006;6:33–9. doi:10.1016/j.rigapp.2006.02.002
11. Hills RK, Daniels J. Assessing new interventions in women's health. Best Pract Res Clin Obstet Gynaecol 2006;20:713–28. doi:10.1016/j.bpobgyn.2006.03.004[Medline]
12. Yeh J, D'Amico F. Forest plots: data summaries at a glance. J Fam Pract 2004;53:1007.[Medline]
13. Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL. Methods for the Economic Evaluation of Health Care Programmes. 3rd ed. Oxford: Oxford University Press; 2005
14. Phillips C, Thompson G. What is a QALY? London: Hayward Medical Communications; 1998
15. Billingham LJ, Abrams KR, Jones DR. Methods for the analysis of quality-of-life and survival data in health technology assessment. Health Technol Assess 1999;3:1–152.[Medline]
16. Naglie G, Krahn MD, Naimark D, Redelmeier DA, Detsky AS. Primer on medical decision analysis: Part 3 - Estimating probabilities and utilities. Med Decis Making 1997;17:136–41.[Abstract/Free Full Text]
17. Sonnenberg F, Beck JR. Markov models in decision making: a practical guide. Med Decis Making 1993;13:322–38.[Abstract/Free Full Text]
18. Buxton M, Drummond MF, Van Hout BA, Prince RL, Sheldon TA, Szucs T, et al. Modelling in economic evaluation: an unavoidable fact of life. In: Health Econ. 6, 1997. p. 217–27.doi:10.1002/(SICI)1099–1050(199705)6:3<217::AID-HEC267>3.0.CO;2-W.[Medline]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Abedin, P. Articles by Khan, K. S Search for Related Content PubMed Articles by Abedin, P. Articles by Khan, K. S