Coverage Decision-Making for Orthopedics Interventions in the Health Transformation Program in Iran: A Multiple Criteria Decision Analysis (MCDA)

authors:

avatar Efat Mohamadi 1 , avatar Seyed Mousa Tabatabaei 2 , avatar Alireza Olyaeemanesh 3 , * , avatar Seyede Fateme Sagha 4 , avatar Marziyeh Zanganeh 4 , avatar Majid Davari 5 , avatar Jalal Arabloo 6 , avatar Mani Yousefvand 1 , avatar Mohammadreza Mobinizadeh 7

PhD Student of Health Policy, Department of Health Management and Economics, School of Public Health, Tehran University of Medical Sciences, Tehran, IR Iran
MD, Director of Planning of Health Economic and Insurance Office, Ministry of Health, Tehran, IR Iran
Associate Professor, National Institute for Health Research, Tehran University of Medical Sciences, Tehran, IR Iran
Expert on Health Insurance, Ministry of Health, Tehran, IR Iran
Assistant Professor, Department of Pharmacoeconomics and Pharmaceutical Management, Tehran University of Medical Sciences, Tehran, IR Iran
Department of Health Management and Economics, School of Public Health, Tehran University of Medical Sciences, Tehran, IR Iran
Assistant Professor, National Institute for Health Research, Tehran University of Medical Sciences, Tehran, IR Iran

how to cite: Mohamadi E, Tabatabaei S M, Olyaeemanesh A, Sagha S F, Zanganeh M, et al. Coverage Decision-Making for Orthopedics Interventions in the Health Transformation Program in Iran: A Multiple Criteria Decision Analysis (MCDA). Shiraz E-Med J. 2016;17(12):e40920. https://doi.org/10.17795/semj40920.

Abstract

Introduction:

All governments, regardless of available resources, should move to establish a balance between the conflict of resource scarcity of health system and health care services required. This study applied a multi criteria decision analysis (MCDA) approach to contribute to priority-setting and the coverage decision-making on including uninsured orthopedics interventions in the healthcare transformation plan’s subsidized in Iran during year 2015.

Methods:

This study was conducted in four phases: a comprehensive review of studies related to the methods and criteria for prioritizing health services, identifying prioritization criteria, scoring and finalizing them, weighting of the criteria identified, and planning for a prioritized uninsured coverage for orthopedics intervention.

Results:

After screening the retrieved titles via PRISMA, from 350 papers, 12 studies were included. The main criteria used for the priority step in the health sector were as follows: safety, efficacy, need, existence of alternative procedures, life expectancy impact, cost, cost-effectiveness, catastrophic health expenditure, impact on the budget, acceptance of social/economic and equity in access. According to the viewpoint of the experts, the safety criteria had maximum weight (0.4) and equity in access had the least weight (0.03). Finally, ten uninsured orthopedics services were prioritized with a score of 9.01 to 5.01.

Conclusions:

This practical and real-life project significantly contributed to rational, apparent, and unbiased priority-setting practice by using the MCDA methodology. Prioritizing and weighting the criteria in this study indicated that the Iranian policymakers should pay more attention to clinical aspects and benefits of the service than financial issues. This could indicate that there are social perspective and health as the public right in the country.

1. Introduction

From the macroeconomic perspective and resource allocation, there are limitation resources in the health sector compared to other sectors. Considering the increase in the number, type and quality of health services, and the increasing demand for health services being much higher than the growth of public and private resources available, all governments, regardless of the amount of resources that are available, should consider the conflict between resource scarcity and required health care services and take steps to balance between the two. The need to “Choose” in the process of resources allocation among "competing demands" is the issue that occurs in all health systems (1).

In health care systems with public financing, this choice (tradeoff between interventions) reflects the relative priorities associated with the choice of interventions. These priorities may be known and formal, or not transparent; it depends on the method for selection and prioritization of health services. Different factors, including demographic and epidemiological changes, increased awareness, education, new technologies and increasing household incomes, have resulted in high demand for health services and lack of capacity of countries to increase resources at the same speed, and thus countries cannot secure all costs. Thus change and development of the health system is inevitable and unavoidable (2).

According to the mentioned reasons, if governments do not prioritize based on specific method, the selection processes are carried out spontaneously and without any interference. This is especially in the health sector that has market failure leads national resources to low-priority or unnecessary services. Solving this problem for low-income countries will be much more difficult and at the same time very important. One of the big problems of developing countries is mismatch allocation and distribution of resources (resources consumed) with the burden of diseases and health problems. Efforts to find reasons for this revealed two important issues: a) use of very simple methods and models for prioritizing health care services, and b) lack of appropriate quantitative information (3, 4).

Health service packages, according to the universal health insurance law in Iran, are categorized in several groups. The law does not support prevention and rehabilitation services. Regarding entering new services to benefit package, the Supreme Council of development pharmacopeia is stewardship to entry new drugs to health insurance benefit package, that they do not use evidence-based approach and focuses more on issues such as health and safety (5).

This study applied a multi criteria decision analysis (MCDA) approach for priority-setting and coverage decision-making, including uninsured orthopedics interventions, in the healthcare transformation plan’s subsidized in Iran during year 2015. Therefore, this study aimed to develop criteria for prioritization of service packages.

2. Methods

Prioritizing services was performed using methods of multi criteria decision analysis (MCDA). The model used in this study was the combination of analytical hierarchy process (AHP) and simple additive weighting (SAW). Analytical hierarchy process was used to derive an accurate weight, given the goal of the research. The disadvantage of this model was drawing multiple paired comparison matrixes that were sometimes causing confusion for experts. For this reason, the SAW model was used on the final phase of the prioritization. Evidence has shown that factors such as efficiency, equity, financial protection and political considerations appear to be important for policy-makers at the time of priorities setting (6-8). Neither of these criteria are worth paying attention to nor are all policy-makers good at gripping different types of information (4). This has opened an opportunity for the use of the MCDA as it covers a comprehensive set of criteria and examines the performance matrix qualitatively or quantitatively to rank order the interventions (9). In the former, policymakers interpret the performance matrix and implicitly judge the weights of the different criteria. In the latter, different criteria are weighted based on their relative importance, and are multiplied by the scores to attain weighed averages for all interventions. The method of this article was designed in four phases, including a comprehensive review of studies related to the methods and criteria for prioritizing health services, identification of the prioritization criteria, scoring and finalizing them, weighting of the criteria identified and planning for a prioritized uninsured coverage orthopedics intervention (Figure 1).

2.1. Phase 1: Comprehensive Review

In the first phase, to increase the sensitivity and specificity in the search, discovery and selection of related articles, the research team investigated several databases. Search terms were relatively common words and synonymous were used with the operator “OR” and “AND”. Search strategy was the selection of the most important keywords and databases such as PubMed and ProQuest, Cochrane library, ISI web of science, Scopus and Embase (1990 to present). The main inclusion criteria for retrieved papers was the existence of experiences of other countries in prioritize health services benefit package in both developed countries and developing countries in these papers. Key words used were: health package, health benefit package, essential health package, minimal health package, priority setting in health, criteria for priority-setting in health care, multi-criteria for priority-setting of health interventions.

2.2. Phase 2: Identify Prioritization Criteria, Scoring and Finalization

During the second phase, prioritized list of criteria and factors were developed to finalize the criteria. Therefore, based on the output of the first phase of the study and after reviewing internal and external sources and other similar studies of developed and developing countries, preliminary list of criteria needed to prioritize services were provided.

The selection list was assessed based on a Likert scale in dimensions of necessity, appropriateness and feasibility by expert opinion.

The primary identified themes were extracted based on their frequency in the studies, and were divided to three categories of service-oriented approach: clinical evaluation, economic assessment and social policy evaluation as checklist to obtain viewpoints of experts. There were 20 experts including top and middle managers, decision makers and policy makers with direct organizational connection who had experience with orthopedic interventions (in health and welfare ministries and health insurance organizations), and also experts and key informants at research centers, academic centers and related organizations. Checklists were sent to participant by email. Response rate was 70%. Data at this stage was analyzed using the SPSS software.

2.3. Phase 3: Weighting the Prioritization Criteria

After development of the criteria list, analytical hierarchy process (AHP) method was used for weighting. The AHP method is one of the most popular multi-functional decision-making techniques, which was invented by Thomas L. Saaty in the 1970s (10).

This method can be used when decision-making faced multi choice of competing and decision criteria weighting. The use of AHP in the group’s decision will keep benefits of group decision-making techniques and will resolve their disadvantages (such as speed, cost etc.), thus optimal decisions are taken with team members (11). The data from this phase were analyzed using the Expert choice software. A group of experts from the insurance organizations and orthopedic experts weighted the selected criteria. Agreement was achieved through geometric mean based on the Saaty scale.

In the AHP, relative weight was calculated by paired comparison matrix. In this research, the geometric mean method was applied. The geometric mean of elements of each row was calculated; then, the obtained vector was normalized to calculate the weight vector.

2.4. Phase 4: Prioritize Uninsured Coverage Orthopedics Interventions

At this stage, with the participation of experts, uninsured coverage orthopedic interventions were prioritized. According to lack of reliable information about interventions within the identified indicators studied in the country, an expert panel was used for prioritizing. An orthopedic, who was a member of orthopedics board of the country, was invited to participate in the expert panel. Panel of experts was held with eight people. At first, the purpose and procedure was explained to the participants. Then each member of the panel provided their opinions about services, according to specified criteria. The data from this phase was analyzed using Simple Additive weighting (SAW) method. In the SAW method, the total score for each alternative was calculated by multiplying the comparable rating for each attribute by the weight assigned to the attribute and then summing these values over all attributes.

3. Results

After screening the retrieved titles via PRISMA, from the 350 papers, 12 studies were included (Table 1 and Figure 2). The main criteria used for priority setting in the health sector were as follows: safety, efficacy, need, the existence of alternative procedures, life expectancy impact, cost, cost-effectiveness, catastrophic health expenditure, impact on the budget, acceptance of social/economic standards and equity in access (Table 2).

Table 1.

Comparison of the Criteria and Priorities Methods for Health Services in the Selected Countries

RowCountryHealth Financing SystemCriteria
1Norway (13)General taxSeverity of disease
The number of patients
Age of the target group
Individual health benefits
The desire for subsidies
Cost-effectiveness
2Thailand (14)Social health insuranceSeverity of disease
The average population covered
The possibility of
implementation
Burden of disease
Budget impact
Vulnerable populations
3Turkey (17)Social Health InsuranceTangibility
Accountability
Reliability
Confidence
Reliability
4Washington state (3)Private market + Medicaid and MedicareCost -utility index
Quality
The possibility of implementation
Access
Cost -effectiveness
5Oregon state (18)Private market + Medicaid and MedicareCost-utility
6UK (18)Specific taxNHS Clinical Practice Guidelines
Health Technology Assessment Guide
Interventional procedures
Public health guide
7Germany (1)Social Health Insurance (Sickness funds)Cost-Effectiveness
Public’s preferences
8Netherlands (18)Social health insurance (Multiple funds) + Private marketNecessity
Effectiveness
Efficiency
Self-sufficient
9Chile (19)Social SecurityCost-effectiveness
Quality
Access
Possibility
Financing
Fair and equitable coverage
10Iraq (20)CombinedPopulation needs
Fair and equitable coverage
Access
Cost-effectiveness
11South Korea (21)Social Insurance + private marketThe number of Severity of disease severity
Effectiveness
Variety of Services
The disease economic impact on households
Ethics and justice issues
12WHO map (18)-Safety
Burden of disease
Catastrophic health expenditure
Size of target population
Service requirements
Health workforce requirements
Information requirements
Unit costs
Budget impact
Cultural acceptability
Political acceptability
Article Search Flow Chart
Article Search Flow Chart
Table 2.

The Frequency of Prioritization Criteria in the Selected Countries

CriteriaDeveloped CountriesDeveloping Countries
Cost Effectiveness63
Vulnerable populations (need)03
Access12
Costs utility 20
Burden of disease22
Public's preferences30
The possibility of implementation12
The quality of service11
Severity of disease21
Clinical Practice Guidelines10
The disease economic impact on households10
Ethics and justice issues10
Individual health benefits10
Budget impact01
Financing01
Safetya--

According to studies, primary list of criteria was developed to prioritize services based on service-oriented approach in three categories; overall impact on life expectancy, economic evaluation, and political and social evaluation and was provided to experts as a checklist.

Mean and standard deviation of the necessity, appropriateness, feasibility and total score was calculated for each of the items (Table 3).

Table 3.

The Average of Total Score and Score of Three Aspects of Necessity, Appropriateness and Feasibility of the Criteria Identified by the Experts

Voted CriteriaNecessityOccasionsPossibilityTotal score
Safety1.1 ± 4.21.1 ± 3.70.4 ± 3.20.9 ± 3.7
Efficiency0 ± 50.3 ± 4.80.6 ± 4.10.4 ± 4.7
The need1.4 ± 41.4 ± 4.11.3 ± 3.51.8 ± 3.5
Alternative procedures0.7 ± 3.51.1 ± 3.71 ± 4.10.8 ± 4
Life expectancy impact1.5 ± 41.4 ± 3.71.6 ± 3.11.6 ± 3.7
Unit costs0.5 ± 4.40.5 ± 4.40.4 ± 4.20.4 ± 4.2
Cost Effectiveness0.7 ± 4.50.7 ± 4.51.4 ± 31 ± 4.1
Catastrophic health expenditure0.7 ± 4.20.8 ± 3.81.5 ± 30.9 ± 3.7
Budget impact0.8 ± 40.7 ± 3.51.4 ± 41 ± 4
Social acceptance / Economy0.8 ± 3.11.5 ± 3.41.1 ± 3.81.2 ±3.1
Equity in access0.7 ± 4.50.3 ± 3.41.7 ± 3.40.7 ± 4.2

The final criteria were selected by mean (average of 3.5 with a SD < 1), for weighting and applying them to prioritize the interventions. According to Table 3, the criteria were safety, efficiency, alternative procedures, life expectancy impact, unit cost, cost-effectiveness, catastrophic health expenditure, budget impact, and equity in access.

Output of expert choice software was weighted criteria. The criteria were weighted between 0 and 1 (Table 4).

Table 4.

Criteria Weights

RowCriteriaWeight
1Safety0.40
2Efficiency0.23
3Alternative procedures0.11
4Life expectancy impact0.10
5Unit costs0.02
6Cost effectiveness0.07
7Catastrophic health expenditures0.01
8Budget impact0.03
9Equity in access0.03

Prioritization the orthopedics intervention that was not reimbursed under health insurances in the country was conducted by weighting criteria. Policy makers can decide on support interventions according to financial burden of providing each service in the healthcare transformation plan’s subsidized in Iran (Table 5).

Table 5.

Prioritization Orthopedic Services

ServicesSafetyEfficiencyAlternative ProceduresLife Expectancy ImpactUnit CostsCost EffectivenessCatastrophic Health ExpenditureBudget ImpactEquity in AccessOutput
Osteochondral allograft, knee, open3.642.040.950.860.200.660.120.280.279.01
Excision of epiphyseal bar, with or without autogenous soft tissue graft obtained through same fascial incision3.642.040.950.860.200.660.120.280.279.01
Chemical synovectomy joint through the needle3.642.040.950.860.200.660.120.280.208.93
Osteotomy of spine, including discectomy, anterior approach, each additional vertebral segment3.642.040.950.860.200.660.120.270.208.92
Osteotomy of spine, including discectomy, anterior approach, single vertebral segment; cervical; thoracic; lumbar3.232.040.950.860.190.660.120.270.138.46
Autologous chondrocyte implantation, knee2.832.040.950.860.200.660.120.250.128.02
Ablation, bone tumors (eg osteoid osteoma, metastasis), radiofrequency, percutaneous; including imaging guidance2.832.040.880.860.190.660.110.270.137.97
Extracorporeal Shock Wave Therapy (ESWT) 3.290.960.200.490.100.220.050.140.065.51
Electrical stimulation to aid bone healing; invasive (operative)3.640.610.160.450.030.200.020.210.055.37
Low intensity ultrasound stimulation to aid bone healing, noninvasive.3.640.450.110.380.020.150.010.210.045.01

4. Discussion

This practical and real-life project, initiated by the ministry of health (MOH) in Iran, has significantly contributed to the rational, apparent and unbiased priority-setting practice by using the MCDA methodology. Prioritizing and weighting the criteria in this study showed Iranian policymakers pay more attention to clinical aspects and benefits of the service than financial issues. This could indicate that there are social perspectives and health as the public right in the country. Although this study used general priority criteria, two criteria were shown to be relevant to Iranian policymakers. The existing implementation of safety and efficiency are in line with the legal and hence societal values and reflect the Patients' Rights of Act. Priority rankings based on the same criteria showed how MCDA can be used and may invite a discussion on priority policies across major disease areas. The findings allow for a prioritization based on existing features of the modern health care complexities that policy makers are facing. Gress et al. (12) analyzed the processes and criteria for shaping benefit packages in England, Germany and Switzerland. According to this study, in Germany, stakeholders play a crucial role in improving the transparency of decision-making procedures. In Switzerland, there is practically no transparency in decision-making procedures at the moment. On the contrary, the costs of healthcare services affect the decision to include or exclude them in England (13). Guidelines and cost-effectiveness studies in Britain and Germany were carried out at independent organizations of the Ministry of Health, unlike Iran, gathering information related to the criteria for the prioritization of health services of the Ministry of Health. In a study carried out by Yang Kung using multi-criteria decision method, the MCDA was applied in four steps: 1) 17 interventions were selected for evaluation; 2) nine of them were nominated for further quantitative evaluation; 3) the interventions were then evaluated with consideration of their cost-effectiveness and budget impact; and finally 4) decision makers qualitatively measured them and came up with a consensus on which interventions should be put in the package (14). Kapiriri in a review study investigated several low-income countries, this study suggests and introduces five factors; effectiveness, transparency, purpose, accountability and fairness for setting priorities in health. Two important criteria expressed in this study (efficiency and equity) are similar with the criteria used in our study (15). Defechereux et al. conducted a four-stage study in Norway (13). These four steps are summarized as follows: identification of policy criteria, identification of different forms of alternatives based on policy criteria, assessment of alternatives using the criteria and determination of the preferred choice by rating them according to the criteria for each intervention (16). Baltussen and Niessen, in their study, introduced the MCDA method to prioritize health services. In this study, unlike our study, public's preferences were also considered (9). Howard et al. conducted a study in Afghanistan to explore stakeholder views of sexual and reproductive health (SRH) services delivered through the basic package of health services (BPHS). The study involved qualitative in-depth interviews. More than 50 percent of opinions about entry in the health service according to the basic benefit package were positive (16). The study used perspective of experts to make a decision about health services priority setting, similar to our study.

4.1. Conclusions

There were some limitations in our study. First of all, it was somewhat hard to define the scoring scales of some criteria including access equity and alternative procedures. Due to the lack of existence of a clear-cut definition for both of the preceding terminologies, experts’ opinions as well as other related articles were used to clarify the definition and scoring scales development. Secondly, we did not find enough evidence to compare each intervention based on the same criteria. Moreover, since implementing an empirical study would be costly and time-consuming for all proposed interventions, we used experts’ opinions along with some available information on this criterion. With consideration of this limitation, cautions should be taken for its future measurement. Thirdly, since non-academic people had some difficulties in understanding some criteria such as effectiveness, it was aimed to achieve a consensus in group discussions between laypeople and higher educated participants. Of course, this did not hinder us from involving all stakeholders in the process of priority setting. The present project was prospected to determine the relative weights of criteria, which may reveal the local values in reality. Although we have used the MCDA in the context of Iran, it is also possible to be applied for other settings. Meanwhile, this would require setting the criteria as well as scoring scales based on these setting conditions. This could then result in decisions that are more sensible, clear, and just. In addition, since this was not a comparative project, therefore, it does not clarify the question of whether or not the MCDA approach would have led to better decisions with respect to the allocation of resources to health interventions. It rather focused on the usage of MCDA and reflects its values that can be assessed through the framework. In spite of using general priority criteria in the present study, Iranian policy makers would use three criteria that are quite relevant to the context of Iran. Furthermore, having proper, clear, and fair national guidelines supports the legal and social values and addresses the Act of patient rights as well. Using various criteria to set the priorities reflects how MCDA can be applied for priority policies across major disease areas. The findings of the present study permit for a prioritization based on the current characteristics of the health care complexities that are of concern to policy makers.

References

  • 1.

    Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, et al. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012;10(1):9. [PubMed ID: 22808944]. https://doi.org/10.1186/1478-7547-10-9.

  • 2.

    Gonzalez-Pier E, Gutierrez-Delgado C, Stevens G, Barraza-Llorens M, Porras-Condey R, Carvalho N, et al. Priority setting for health interventions in Mexico's System of Social Protection in Health. Lancet. 2006;368(9547):1608-18. [PubMed ID: 17084761]. https://doi.org/10.1016/S0140-6736(06)69567-6.

  • 3.

    Bobadilla JL. Searching for essential health services in low and middle income countries. 1998. Available from: https://publications.iadb.org/handle/11319/5310.

  • 4.

    Bobadilla JL, Cowley P, Musgrove P, Saxenian H. Design, content and financing of an essential national package of health services. Bull World Health Organ. 1994;72(4):653-62. [PubMed ID: 7923544].

  • 5.

    Rahimi P, Tabatabaie H, Gouya MM, Zahraie M, Mahmudi M, Ziaie A, et al. Characterization of mutations in the VP(1) region of Sabin strain type 1 polioviruses isolated from vaccine-associated paralytic poliomyelitis cases in Iran. J Clin Virol. 2007;39(4):304-7. [PubMed ID: 17590391]. https://doi.org/10.1016/j.jcv.2007.04.017.

  • 6.

    Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health. 2009;14(8):930-9. [PubMed ID: 19563479]. https://doi.org/10.1111/j.1365-3156.2009.02311.x.

  • 7.

    Ubel PA. Pricing life: why it's time for health care rationing. MIT Press; 2001.

  • 8.

    Musgrove P. Public spending on health care: how are different criteria related? Health Policy. 1999;47(3):207-23. [PubMed ID: 10538919].

  • 9.

    Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006;4:14. [PubMed ID: 16923181]. https://doi.org/10.1186/1478-7547-4-14.

  • 10.

    Ju Y, Wang A. Emergency alternative evaluation under group decision makers: A method of incorporating DS/AHP with extended TOPSIS. Expert Systems Applicat. 2012;39(1):1315-23. https://doi.org/10.1016/j.eswa.2011.08.012.

  • 11.

    Wang Y, Jung KA, Yeo GT, Chou CC. Selecting a cruise port of call location using the fuzzy-AHP method: A case study in East Asia. Tourism Manag. 2014;42:262-70. https://doi.org/10.1016/j.tourman.2013.11.005.

  • 12.

    Gress S, Niebuhr D, Rothgang H, Wasem J. Criteria and procedures for determining benefit packages in health care. A comparative perspective. Health Policy. 2005;73(1):78-91. [PubMed ID: 15911059]. https://doi.org/10.1016/j.healthpol.2004.10.005.

  • 13.

    Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP, et al. Health care priority setting in Norway a multicriteria decision analysis. BMC Health Serv Res. 2012;12:39. [PubMed ID: 22335815]. https://doi.org/10.1186/1472-6963-12-39.

  • 14.

    Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health. 2012;15(6):961-70. [PubMed ID: 22999148]. https://doi.org/10.1016/j.jval.2012.06.006.

  • 15.

    Kapiriri L. How effective has the essential health package been in improving priority setting in low income countries? Soc Sci Med. 2013;85:38-42. [PubMed ID: 23540364]. https://doi.org/10.1016/j.socscimed.2013.02.024.

  • 16.

    Howard N, Woodward A, Patel D, Shafi A, Oddy L, ter Veen A, et al. Perspectives on reproductive healthcare delivered through a basic package of health services in Afghanistan: a qualitative study. BMC Health Serv Res. 2014;14:359. [PubMed ID: 25167872]. https://doi.org/10.1186/1472-6963-14-359.

  • 17.

    Atun R, Aydin S, Chakraborty S, Sumer S, Aran M, Gurol I, et al. Universal health coverage in Turkey: enhancement of equity. Lancet. 2013;382(9886):65-99. [PubMed ID: 23810020]. https://doi.org/10.1016/S0140-6736(13)61051-X.

  • 18.

    Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res. 2012;12:454. [PubMed ID: 23234463]. https://doi.org/10.1186/1472-6963-12-454.

  • 19.

    Moccero D. Delivering cost - efficient public services in health care, education and housing in Chile. Paris, Organization for Economic Co-operation and Development (Economics Department Working Papers No.606). 2008. Available from: http://www.eric.ed.gov/PDFS/ED503922.pdf.

  • 20.

    A Basic Health Services Package for Iraq. 2009. Available from: http://www.emro.who.int/iraq/pdf/basic_health_service_package_en.pdf.

  • 21.

    Ahn J, Kim G, Suh HS, Lee SM. Social values and healthcare priority setting in Korea. J Health Organ Manag. 2012;26(3):343-50. [PubMed ID: 22852456]. https://doi.org/10.1108/14777261211238981.