1. Background
Today, information technology in healthcare centers, especially in hospitals, has a strong potential for improving the quality of healthcare services, enhancing the productivity and effectiveness of programs, and reducing organizational costs (1-3). Thus, in many hospitals and other healthcare centres, a wide range of such technologies are used (4) for various purposes. As prescribing is a vital process in healthcare providing, EPS is one of such information technology to be efficiently adopted (5). Electronic prescription refers to the application of electronic systems for facilitating and improving communications related to the prescription process. This system also assists in the selection, consumption, preparation, and supply of medications through supporting decision-making and providing access to knowledge at the site of care. Also, it offers the possibility of precisely auditing the entire process of medication consumption (6).
Thus, implementing an EPS can overcome many paper-based prescription problems and medical errors, such as illegal actions and mistakes, and offer innumerable opportunities for a more effective and beneficial prescription process (7-10). Implementation of electronic prescribing systems is an irreversible intervention in the prescription process. This system is an interdisciplinary socio-technical information system with different specifications, including a wide range of users and domains of expertise; a high degree of complexity; numerous users and subsystems; different implementation processes; and special technical solutions in every country (11-14).
The use of information systems in healthcare, especially electronic prescribing, has increased in the past decades, and one of their main objectives is reducing human errors (15). If they have weak performance, they will have a negative impact on the healthcare providing process. Thus, information systems in healthcare, especially EPS must be rigorously evaluated. This evaluation ensures their performance and quality and encourages users and policy-makers to use them (16, 17).
Electronic prescribing is a recommended solution for enhancing patient security and satisfaction (18), and the aims of this solution are to reduce medicinal errors and prescription time and improve medication adherence in patients (19).
Evaluation of health information systems is essential to ensure its proper performance and minimize potential errors (15, 20). However, despite a large number of published evaluation studies in this area, many authors have reported different problems about the evaluation of this system, classified in three main areas: the complexity of the evaluation subject, the complexity of an evaluation project, and the motivation for evaluation (20, 21). Also, Ammenwerth et al. (2003) declared that there is no standard method for evaluating an integrated information system, and the success of system evaluation is a complex phenomenon (16).
Several studies have proposed and applied tools for the EPS evaluation (20-23). Kaufmann et al. (2014) conducted a comprehensive overview of existing tools to assess inappropriate prescribing and revealed the characteristics of the assessment tools to assist readers in choosing the appropriate tool (24). In addition, Bell (2004) developed a conceptual framework for evaluating the EPS (25). In another study, a rational multidimensional workflow was provided to develop and implement EPS in clinical settings (26). Barber et al. (2007) provided an evaluation framework and described the advantages of approaching the evaluation of integrated electronic prescribing from a socio-technical perspective (27).
In another related study, Devine et al. (2010) conducted a qualitative evaluation method and proposed a tool for assessing the EPS perceived by those who write the prescriptions and implement this system (28). In Iran, a study was carried by Ahmadi et al. (2014) to model the current business process of outpatient prescribing and clarify various actions during this process. This modeling could provide a gateway toward the future EPS (29).
The literature review revealed that previous studies examined only a few dimensions of EPS evaluation. Therefore, a comprehensive and practical evaluation tool is needed for the critical appraisal of features of this system at the national level. Due to the lack of a standard and multidimensional tool for national EPS evaluation, the present study aimed to develop a multifaceted evaluation tool covering all dimensions of this system.
2. Methods
This study comprised a literature review and a qualitative study conducted in two main steps in 2018 (Figure 1).
Figure 1.
The process of developing the final tool for EPS evaluation
In the first step, a literature review was carried out in PubMed, Web of Science, Scopus, Embase databases, Google Scholar, and Google search engines from 2000 to 2018 to identify the main features and capabilities of the prosperous EPS. The following corresponding search terms and synonyms were used, adapted based on each database: ("electronic prescription" OR "electronic prescribing" OR "electronic Prescriptions" OR eprescri* OR e-prescri* OR e-Rx OR "electronic transmission of prescription OR "medical order entry systems" OR eDispensing OR "electronic dispensing" OR "two-way electronic order system" OR "computerized physician order entry" OR CPOE OR "prescription routing services") AND (assessment OR evaluation OR characteristics OR specification OR criteria OR feature OR capabilities OR "conceptual framework" OR recommendations OR guide OR manual OR "practical resource" OR blueprint OR handbook OR Overview). The English language was set as a search limit. Finally, from 1893 retrieved documents (research articles, guidelines, reports, manuals, handbook), 14 documents were selected based on the objectives of the study. A narrative content analysis was applied to find the main features and capabilities of the prosperous EPS.
Based on the extracted features and capabilities of the EPS from the literature review, a preliminary questionnaire was designed for Delphi rounds. The clarity and understandability of the questions determined by the experts in the related fields, including medical informatics and health information management, pharmacists, and physicians. All of the participated experts expressed that the items were easy to understand and that no modification was required. The questionnaire had a five-point Likert scale, 1 (completely disagree) to 3 (neutral) to 5 (completely agree), for assessing the level of agreement of the experts about the questions used for evaluating the electronic prescription system.
The second step was a qualitative study using a Delphi methodology. Recruitment using purposive sampling and snowballing techniques identified eligible experts within related fields of the subject with at least three years of work experience, and 30 experts were invited for the Delphi round, of whom 27 agreed to participate. The Delphi encompassed the professionals from the department of drug and narcotic monitoring of the ministry of health and medical education (N = 3), insurance specialists (N = 6), general practitioners (N = 5), pharmacists (N = 6), and experts in medical informatics (N = 3), and health information management field (N = 4). The number of participants in Delphi rounds remained constant, with a 100% level of participation in all three rounds. Data were collected by the Delphi method until reaching the consensus of experts on evaluation questions and domains.
In the first round, the experts were allowed to suggest new items and write comments on the questionnaire. In the second round, the experts were allowed to change their scores by mentioning their reasons. Delphi rounds continued until reaching the consensus of the experts, defined as over 75% agreement.
3. Results
In the first step, criteria for evaluating EPS were extracted from 14 selected records (five original research articles, four guidelines, four research articles, and one manual) (14, 23, 30-41). Based on the results of this step, a 79-question questionnaire in 12 main domains was prepared for use in the Delphi rounds of the second step.
These domains included practical capabilities of the process/user and patient safety, transfer and storage of data, prescription control and renewal, technical functions, user interfaces, security and privacy, reporting, legality, scalability, portability, hardware and infrastructure, and system failure/recovery. In the first Delphi round, 12 items were added to the questionnaire (total question number = 91) based on the experts' comments and suggestions. In round 2, a questionnaire with 91 questions in 12 main dimensions was applied (Figure 1). Based on the agreement rate of the second round, 30 questions and two domains (legality and scalability) were deleted (Figure 1). In the third round of Delphi, the experts' consensus was reached about all 61 remaining questions. The result of the third round of Delphi is presented in Table 1.
Table 1.
The Agreement Rate of the Items for EPS Evaluation (Third Delphi Round)
| Row | System Capabilities | Median | Agreement Percentage |
|---|---|---|---|
| Practical Capabilities of Process/User and Patient Safety | |||
| 1 | Is the system part of the electronic health record system? | 5 | 91 |
| 2 | Does the system provide the possibility of selecting medications appropriate for the diagnosis? | 5 | 95 |
| 3 | Does the system provide access to the 'patients' medical history? | 5 | 98 |
| 4 | Does the system provide access to the 'patients' current and previous medications? | 5 | 96 |
| 5 | Does the system display a set of patient demographic information at the time of prescription for patient identification? | 5 | 90 |
| 6 | Is it possible for the system administrator to combine multiple records created for the same patient? | 5 | 92 |
| 7 | Can the system help patients control their costs, and does it enable the provider to determine the 'patients' actual costs for selection of medication based on the pharmacopoeia and insurance coverage? | 5 | 99 |
| 8 | Can the system offer to users alternative drugs based on patients' clinical and insurance information? | 5 | 97 |
| *9 | Can the system provide optimal medication use instruction and provide the patients with information on how to use the prescript drugs and why there were prescribed? | 5 | 93 |
| 10 | Does the system control drug contraindication? | 5 | 95 |
| 11 | Does the system control drug-disease interactions? | 5 | 97 |
| 12 | Does the system control drug-disease interactions? | 5 | 93 |
| 13 | Does the system control drug allergies? | 5 | 93 |
| 14 | Can the system control drug contraindications at the time of drug selection? | 5 | 95 |
| 15 | Does the system allow the patients to view their drug history? | 4 | 82 |
| 16 | Can the system control repeat treatments? | 5 | 85 |
| 17 | Can the system print out a complete drug list for the patient? | 5 | 77 |
| 18 | Can the system examine all non-prescribed drugs, e.g. OTCs and alternatives, for the doctor writing the prescription? | 5 | 86 |
| 19 | Can the system control the drug at the time of pregnancy or lactation? | 5 | 81 |
| 20 | Can the system calculate drug dosage? | 5 | 89 |
| 21 | Can the system control drug dosage for the elderly? | 5 | 88 |
| 22 | Can the system control drug dosage for children? | 5 | 84 |
| 23 | Does the system allow the modification of a prescription before sending it without having to create a new prescription? | 5 | 83 |
| 24 | Does the system interact with external information systems and various databases and can extract patient data from external sources, e.g. Hospital, laboratory, and electronic health record systems, in order to support informed decision-making? | 5 | 95 |
| 25 | Does the system provide alerts on prescription renewal and repeat? | 4 | 80 |
| 26 | Does the system provide training for patients? | 5 | 95 |
| 27 | Does the system support drug withdrawal guidelines and prescription changes by the doctor? | 5 | 83 |
| 28 | Does the system provide doctor-level feedback through doctor's access to their prescription pattern? | 4 | 80 |
| Data Gathering and Transferring | |||
| 29 | Can the prescription be sent to the pharmacy chosen by the patient or the central electronic prescription database? | 5 | 98 |
| 30 | Can the electronic prescription be retrieved via different methods (scanning prescription barcode, using the patient smart card, entering prescription code, and etc.) at the pharmacy? | 5 | 99 |
| 31 | Is clinical data transfer among systems based on the final edition of HL7 or NCPDP? | 4 | 79 |
| 32 | Does the system use a single provider ID? | 5 | 97 |
| 33 | Does the system use a single patient ID? | 5 | 98 |
| 34 | Is the doctor informed of the unsuccessful transfer of the prescription to the pharmacy? | 5 | 93 |
| Prescription Renewal and Control | |||
| 35 | Does the system notify the doctor on prescriptions and renewal of prescriptions not prescribed in a period specified in the prescription? | 4 | 82 |
| 36 | Does the system remind the clinical specialist of the results of tests and controls recommended by drug producers? | 5 | 85 |
| 37 | Does the system notify the doctor of laboratory results that require measures to be taken? | 4 | 85 |
| 38 | Does the system provide access to laboratory results? | 5 | 81 |
| 39 | Does the system receive and store data on drug delivery to the patient from the pharmacy? | 5 | 96 |
| 40 | Can the system control the permission for repeating the prescription in terms of time interval and number of permissible repeat times? | 5 | 85 |
| 41 | Can the system send a request for prescription renewal and repeat from the pharmacy to the doctor? | 5 | 88 |
| 42 | Can the system send a confirmation for prescription repeating requests from the doctor to the pharmacy? | 5 | 96 |
| Technical Function | |||
| 43 | Does the system support emergency clinical controls? | 4 | 77 |
| 44 | Does the system support emergency controls for patient eligibility? | 4 | 78 |
| 45 | Can the system identify the user? | 4 | 95 |
| 46 | Does the system have valid credentials? | 5 | 99 |
| User Interfaces (UI) | |||
| 47 | Is the alert message clearly displayed on the computer screen? | 4 | 81 |
| 48 | Is there a mechanism for the prevention of errors while selecting a drug from the drop-down menu? | 5 | 89 |
| 49 | Can the system receive the necessary data from the pharmacy's computer system? | 5 | 92 |
| 50 | Can the system receive the necessary data from the payment system or insurance companies (managers of medical insurances)? | 5 | 97 |
| Security and Privacy | |||
| 51 | Does the system enjoy optimal computer program security? | 5 | 80 |
| 52 | Does the system enjoy optimal transaction security? | 5 | 85 |
| 53 | Does the system support privacy and confidentiality standards? | 5 | 90 |
| 54 | Can any user be separately identified in the system and enjoys role-based access? | 4 | 79 |
| 55 | Does the system support the electronic signature? | 5 | 98 |
| Reporting | |||
| 56 | Is it possible to receive various reports from the system? | 5 | 0.88 |
| Portability | |||
| 57 | Does the system support personal digital assistants, smartphones, and tablets? | 5 | 0.94 |
| Hardware and Infrastructure | |||
| 58 | Is the system independent of the platform? | 5 | 98 |
| 59 | Can the system be accessed in a wireless environment? | 5 | 86 |
| System failure/recovery | |||
| 60 | Can the system recover damaged or lost files? | 5 | 89 |
| 61 | Can the system automatically recover data? | 5 | 92 |
All items with an acceptable significance level were regarded as necessary and important for the EPS evaluation tool in the third Delphi round. Finally, the electronic prescribing evaluation tool had 10 main dimensions (Table 1).
4. Discussion
The evaluation of health information systems, including electronic prescription systems, provides valuable information for system developers, helping them develop the system based on 'users' operational needs and thereby leading to the efficient and effective application of these systems (16, 38). Appropriate health information system evaluation tools is vital for achieving valuable and useful results (42). Devine et al. evaluated the EPS by using a qualitative evaluation method and proposed a tool with ten main criteria for assessing the EPS (28).
The present study proposed a tool for electronic prescription evaluation with the main dimensions of practical capabilities of the process/user and patient safety, transfer and storage of data, prescription control and renewal, technical functions, user interfaces, security and privacy, reporting, portability, hardware and infrastructure, and system failure/recovery.
Based on the findings, the practical capabilities of the process/user and patient safety is one of the main items in this tool, comprising one-third of the items in the questionnaire. According to this dimension, the EPS must be able to integrate with EHR systems, detect all kinds of drug interactions at the time of prescription, and offer to users alternative drugs based on patients' clinical and insurance information.
Samadbeik et al. stated that the progress and success of electronic prescription systems depend on integrating electronic health records (EHR) and the cooperation of all stakeholders (43). Abramson et al. expressed that implementing and using an electronic prescription system integrated with EHR reduce medicine-related errors and increase patient safety (44). Also, problems with the design and safety of the EPS were stated as a cause for the unsuccessful implementation of this system in a Brazilian general hospital in the study by Joia and Magalhães (45).
Bell et al. made recommendations for patient safety and health outcomes in the electronic prescription system based on expert consensus. The main capabilities agreed upon were diagnosis-based medicine menus, safety alerts, current medication lists, and system integrity (25). Therefore, in system evaluation, the practical capabilities of process/user and patient safety must receive considerable attention.
The transfer and storage of data was another main criterion for system evaluation. Several studies have emphasized the transferability and storage of data in electronic prescription (43, 44, 46, 47). A major role of health information systems is the enhancement of data transfer and exchange in the healthcare setting and the facilitation of data storage. One must, therefore, pay attention to this feature as a major criterion in system evaluation.
The results of this study revealed that prescription renewal and control is another important criterion in system evaluation. Due to the high sensitivity of the treatment process, prescription control through access to other data, e.g., results of tests, seems to be essential. In some cases, the prescription may be renewed or modified based on patient conditions. Suna regarded accessibility to patient data in the electronic prescription system as a vital matter. This feature has been included in the electronic prescription program of Finland (48). This feature is also included in the electronic prescription system of Denmark as a pioneering country (49). This issue, which directly affects patient health, must be considered in the electronic prescription evaluation. Systems without this ability are considered to be low inefficiency.
The technical functions of the system must also be included in its evaluation. The electronic prescription system must be able to identify the patients and be accredited by credible sources.
Our results also showed that user interfaces of the system merit attention from developers and evaluators. Avery et al. regarded user interface as an important feature of computer systems used by general practitioners (32). Due to the high importance of user interface in the successful implementation of a health information system, various studies have evaluated the user interface of these systems (50, 51). The suitable user interface design is an integral part of electronic prescription system development, which merits the attention of developers and evaluators. This is why many studies have performed user interface evaluation independently of other components because it directly affects the continuation of use and motivation for the use of the system.
Other results indicated that the other main component of electronic prescription system evaluation is its privacy. Cochran et al. (2015) evaluated security and privacy as the most important issues in implementing healthcare systems and regarded them as influential on the quality of implementation (52). Secure information exchange and preventing the access of unauthorized persons to patient data contribute to the progress of health information systems, thus ensuring system success and popularity among patients and healthcare providers. The development of a secure prescription system assists integration with an EHR (43). Also, Zarour et al. proposed an architecture for a national EPS guaranteed patient privacy and sufficient interoperability of e-prescription system with other e-health services for the developing country of Algeria (53). Therefore, security and privacy are of paramount importance in electronic prescription system evaluation, and the status of this component affects the evaluation results.
The other main components specified in this study as the main criteria for electronic prescription system evaluation were reporting, transportability, hardware and infrastructure, and system failure and recovery. One feature which must be taken into consideration while developing information systems is reporting for various purposes. For instance, the system must provide reports which may be required by managers and doctors for different reasons. Transportability depends on the development of infrastructures and the method of design and development and can be regarded as a value-add for the system. This feature must be examined by system evaluators.
In terms of hardware and infrastructure, a brief examination suffices because the electronic prescription system cannot be implemented in the first place if appropriate infrastructure is not available. The system must recover data because they may be lost for different reasons (14, 54), and the system designers and developers must pay attention to this point. Also, the study of Joia and Magalhães about the implementation of an EPS in a Brazilian general hospital showed that one of the main motives for resistance to the system was inadequate technological infrastructure (45).
4.1. Conclusion
The electronic prescribing evaluation tool developed in this study can be used for the critical appraisal of features of this system. It is recommended that this multifaceted evaluation tool be employed to help buyers compare different systems and assist EPS software vendors in prioritizing their activities regarding the system development. By application of this tool, health care provider organizations can also choose a system that improves many aspects of health care.
Based on the results of this study, it is recommended that the formal evaluation of the electronic prescription system in use in healthcare centers be performed. In order to improve the status of the electronic prescription system, comprehensive and complete evaluation tools which cover all dimensions of the prescription must be employed. The evaluation tool proposed here considers all dimensions of the electronic prescription system, i.e., practical capabilities of the process/user and patient safety, transfer and storage of data, prescription control and renewal, technical functions, user interfaces, security and privacy, reporting, portability, hardware and infrastructure, and system failure/recovery, and can thus be a suitable tool for electronic prescription system evaluation.
