Designing a Performance Evaluation Model Based on Balanced Score Card and Analytic Hierarchy Process Methods: Montaserieh Hospital

1. Background

In the recent years, interest in health system performance measurement has grown rapidly (1, 2). With dramatic evolution of management knowledge, the existence of an effective evaluation system seems inevitable. In one way, the lack of evaluation in various aspects of organization, including assessment of the use of resources, personnel, goals, and strategies is considered as one of the symptoms and diseases of the organization (3-5). Among various methods that have been proposed for evaluating and directing the organization, balanced score card (BSC), which focuses on implementing strategies in action is the only method that reflects effective performance of parts of the organization on its performance as a whole (6). This technique is commonly used in the health systems of different countries, including the US and Canada, as an evaluation method, especially in the recent years (7). Balanced score card (BSC) was first introduced by Kaplan and Norton as a performance measurement tool, which translates the organization’s strategic plan to operational programs (1, 4) and a combination of financial and non-financial indices with four visions, including financial, growth and learning, processes, and customer (4, 8). Furthermore, AHP is one of the well-known techniques of multi-objective decision-making (9, 10), which is a way to prioritize (determining the degree of importance) with a strong scientific approach to decision-making (9, 10). In various studies, the use of BSC has caused enhancement of patients and community satisfaction, increase in productivity and hospital sources, improvement in performance in managing health care, controlling the costs of the hospital as well as careful comparison of the performance of the Iranian public hospitals (9, 11-13), and it is suggested to apply the BSC approach to improve hospital performance evaluation (13, 14).

Using BSC-AHP integrative approach for strategic assessment allows policy-makers to identify a set of performance indices with prioritization and weighting of indices that are consistent with the strategic objectives (15). Kazemzadeh et al. (2010) and Chan’s findings shows that the composition of the BSC-AHP could help hospitals increase the quality of their services (16).

Currently, there is no integrated system for evaluating the performance of public hospitals, therefore, establishing a system of evaluation and performance measurement at the hospitals seems essential (8, 12, 14).

2. Objectives

Given the importance of performance evaluation in the health care system and recommendations of the hospitals’ new evaluation system on the use of BSC and AHP in determining indicators and prioritizing them (17, 18) and the lack of studies in the field of performance evaluation of hospitals in Iran, this study aimed at evaluating the performance of Montaserieh hospital with BSC approach, during year 2015.

3. Methods

This research was an applied study using a hybrid approach (quantitative-qualitative) that was conducted in 2015, at Montaserieh hospital of Mashhad, Iran, as a pilot study. Montaserieh is located in the center of the city of Mashhad, Khorasan Razavi, in the East of Iran as the only specialized governmental referral educational hospital providing organ transplantation and stem cell separation affiliated to Mashhad University of Medical Sciences.

In order to implement the balanced score card approach, the following steps were carried out at the hospital:

- The first stage: formation of the BSC team

- The second stage: reviewing the hospital’s strategic plan, including strategic map of objectives and strategic themes and objectives of each vision and their relationships

- The third stage: identifying primary indices of hospital performance evaluation (financial, customer, processes, learning, and development)

- The fourth stage: the final selection of criteria using the Delphi method

- The fifth stage: determining the weight of each vision and index using AHP

- The sixth stage: measuring the realization percentage of each index and the importance factor or weight of that index.

- The seventh stage: determining the overall performance of the hospital.

Implementation of these steps at the Montaserieh hospital started with a briefing session; after that, the strategic plan of the hospital was determined by the BSC team in focused group discussion sessions (four sessions of four hours). The team was selected based on expert sampling, and included the director of the hospital, head nurses, training supervisors, and the head of quality improvement sector, safety manager, administrative affairs manager, finance manager, and the person in charge of keeping medical records. To determine the indices, library documents, internet sites, and documents available at the hospital were studied and about 150 primary indices were proposed and then by forming focused group discussions (6 sessions of 3 hours) with the BSC team, 68 indexes in the four visions of BSC were determined based on significance. Eventually, using the classic Delphi method in two rounds, 34 indices were finalized in two stages by the BSC team. These indices were presented in the form of a checklist and their formal and content validity were revised several times by the review team and its criterion validity was confirmed by categorizing the indices. In addition to the key people of the hospital, four professors of the management and health economics department of the health school, as people familiar with the process of balanced scorecard and creation of the inter-sectoral team nature, were added to BSC team members to choose a more accurate and realistic indicator. In order to achieve final selection of indicators identified in the previous step, two meetings were held with the members of the BSC team and the experts and professors and finally, according to the experts and to achieve the quality results, the Delphi consensus method was used in two steps to select the final indicators.

For this purpose, a form of basic indicators with four perspectives of the balanced score card was developed, which was completed in two stages by the members of the BSC team. Team members were asked to complete this form for each index by giving a score of one to ten. After collecting forms distributed in the first phase, the mean score for each indicator was calculated from the sum of the scores by each member of the BSC team and the indicators with more than 7.5 points were selected and indicators with a score below 2.5 were excluded. The indicators with scores between 2.5 and 7.5 were also modified and made available for the BSC team members again in the corresponding form for the second stage in order to rate them. At the end of the second stage, the indices with 7.5 points were selected and added to the first stage indices. Then, at meetings (two sessions) with BSC team members, the experts were asked to express their final comments, and ultimately, the most important performance evaluation indicator was selected based on the ability to collect and perform strategic planning in Montaserieh hospital in the four perspectives of BSC.

Selected indicators at each stage of the research were regularly reviewed and modified in case of necessity by the team members; the collection and analysis of this research was conducted at the same time.

In the next step, measuring hospital performance was done by index weighting and prioritization based on analytical hierarchy process and collecting data on the indices in different parts of the hospital; thereafter, by calculating the score for each index and vision, the rate of hospital performance could be achieved. At this stage, to determine the performance of each vision, first, the percentage of realization of each index (measured value of index divided by the value of the expected goal multiplied by 100), then the final weight of each index (by multiplying the weight of each index in weight of its corresponding vision), and the percentage of realization of each index (by multiplying the percentage of realization of each index in the final weight of each index) was determined. Finally, the score of a vision was calculated by summing weighted percentage of realization of all indices of a vision, and then to calculate the score of hospital performance, the performance of the four visions was added together (Figure 1).

conceptual framework for balanced score card

Figure 1.

conceptual framework for balanced score card

For this purpose, at the Montaserieh hospital, the questionnaire of AHP was designed in the form of pair wise comparisons and was distributed between BSC team members (14 people). After entering the questionnaires’ information for all members of the group of experts (n = 14) and calculating the perspectives and indicators weights from the perspective of each of these members, all these ideas were combined and weight of each of the perspectives and indicators were calculated according to the experts group (n = 14) in the next stage.

For each indicator, the following items were determined: the unit of measurement (number, percentage, etc.), the index responsible (quality improvement, infection control, medical records, accounting, etc.), the expected goal, the expected realization value of the Index (the computed quantitative value). In this study, the expected targets were determined according to the hospital’s strategic plan by managers and employees.

Experts participating in the study had complete freedom to choose to participate in the study. In addition, informed consent was obtained from all these people, yet they were paid no money for the collaboration.

4. Results

In order to determine the indices, the hierarchical tree of strategic map was drawn as shown in Figure 2.

The hierarchical tree of strategic mapping

Figure 2.

The hierarchical tree of strategic mapping

Therefore, 68 indices were identified for this purpose in the first stage, of which 26 indices in the vision of processes, 20 indices in the financial vision, 14 indices in growth and learning vision, and eight indices in the customer vision were put. Then, among these indices, 34 indexes were selected in a two stage process using the Delphi method, according to the ideas of the BSC team: Seven indices in growth and learning vision, 13 indices in the processes vision, seven indices in the customer vision, and seven indices in the financial vision. To achieve a single interpretation of any index, definitions should be provided for each index based on the databases of the Ministry of Health, articles, and opinion of experts, and this matter was achieved by the BSC team at Montaserieh hospital.

Based on the analysis of the questionnaire obtained from AHP pair-wise comparison, the importance factor of customer vision was 65.7% (the highest), and importance factors of financial vision, process vision, and growth and learning vision were 19.7%, 9%, and 5.6% (the lowest), respectively.

In assessment of indexes of the customer vision, it was found that the index of percentage of responding to patients weighing 39% had the highest importance; and index of ratio of kidney transplant operations of Montaserieh hospital to the total kidney transplant operations across the country weighing 4%, has the least importance at customer vision. This prioritization in other visions concludes that in the financial vision, increased revenue percentage compared to the past year, and the index of medical tourism earnings of hospital over the previous year had the highest (32.5%) and lowest (5%) importance, respectively. The most and least important indexes in processes vision are nosocomial infection index and success of internal departments and units in achieving the total audit score of the same departments and units with scores 21.1% and 2.2%, respectively. Also, in terms of growth and learning, the index of needed nursing resources based on the standards had the highest importance by 41.6% and the index of decisions of hospital committees, which was not implemented by 3.1%, had the least importance (Table 1).

In relation to prioritization of all of the indices based on the final weight, it was found that the response percentage to the patients with an overall share of 25.6%, Hoteling index with 11.8 %, and employee satisfaction index with 9.9% were more important at Montaserieh hospital. On the other hand, the percentage of the hospital executive committee decisions, which were implemented, the percentage of success of departments and units to achieve the total audit score of the same departments and units, and the percentage of approved processes obtained as 0.17%. 0.19%, and 0.22%, respectively, were the least important among all indices of four visions of BSC.

In regards to Ratings the visions, customer vision obtained a score of 58.26% (the highest), financial vision 17.71%, process vision 8.24%, and growth and learning vision 5.06% (the lowest). Review of the scores of performance of visions’ indices showed that at the customer vision, percentage response to the patients by 19.76%, at financial vision the percentage of hospital revenue increased compared to the previous year by 9.65 percent, and at the processes vision the percentage of nosocomial infection with 1.76%, at the growth and learning vision the index of nursing staff required number based on the standards by 2.33% achieved greatest weighted realization. The final score of Montaserieh hospital performance in 2015 was equal to 89.26% (Table 2).

Also, in relation to the achievement of the objectives at each vision (the sum of realized percentage of all indexes of a vision and dividing it by the number of indexes), the results showed that Montaserieh hospital in Mashhad, during year 2015, achieved 80%, 93.7%, 78%, and 90% of its objectives in the customer, internal processes, financial, and growth and learning visions, respectively. In general, it could be said that Montaserieh hospital in Mashhad achieved 85.4% of its goals during year 2015.

5. Discussion

According to the results obtained from AHP, it was found that the highest weight (importance factor) among the perspectives of the Balanced Score card was related to the customer perspective with 65.8%, which indicates that this perspective had the highest priority in BSC model among the four perspectives followed by financial perspective with 19.7%, processes perspective with 9% and finally, growth and learning had the lowest priority (5.6%) in measuring performance based on four perspectives of the BSC.

In order to show the effect of weighting the indexes and perspectives (using AHP), by comparing hospital performance rating, without weighting the indices and perspectives, it was found that in case of weighting indices and perspectives, calculating hospital performance in every aspect will be more accurate and therefore, the hospital performance score will be higher.

According to the results, the coefficients of customer, financial, processes and growth, and learning perspectives were calculated as 65.7, 19.7, 9, and 5.6, respectively. Therefore, among the BSC perspectives in Montaserieh hospital of Mashhad, the customer perspective had the greater priority than the other perspectives. The findings are consistent with the findings of Ebrahimi and Vatankhah and Salemi, Chan, and Wu et al. (17, 19-21).

Growth and learning perspective was calculated as 5.6%, and had the least importance; this is while the growth and learning perspective was a base for other perspectives of BSC and therefore, one should be careful about the selection of its indicators (22). In the present study, seven indicators were selected in this perspective, among them, education per capita, the number of nurses needed, the number of committees and time to access information indices were consistent with the selected indicators of Iravani et al. and Raeisi et al. study on this perspective (9, 23).

Regarding the scores obtained from the indicators performance, the customer perspective also had the highest score in the hospital performance with 58.26% followed by financial perspective with 17.71%, processes perspective with 8.24% and learning and growth perspective with 5.06% of the total 100% of Montaserieh hospital performance score. This finding is consistent with Iravani et al. and Azar and Mohammadi findings (9, 12).

5.1. Limitation

Although studies around the world confirm the effectiveness of BSC in the private and public sector, yet in general, there are also disadvantages in applying it. Firstly, there are no perspectives or scales appropriate for all the organizations. Secondly, BSC has not relatively or absolutely provided a technique for estimating the importance of each perspective and even indicators of one perspective. In this regard, to overcome the first obstacle, a review should be conducted in levelling perspectives when mapping, due to the nature of the organization (24). To overcome the second obstacle, it is suggested to use AHP and weighting perspectives and indicators of BSC model to make better performance evaluation by determining model parameters and the priority of each indicator (19).

This study facilitated achieving the objectives through better connection among the goals of strategic planning with operational programs activities of each unit, in a way that by a better score has helped the hospital evaluation compared to the past.