Abstract
Background:
Clinical decision support (CDS) functionalities in Computerized Provider Order Entry system (CPOE) need to be identified by the institutional healthcare providers in developing countries. In this regard, CDS functionalities should be a priority for the execution in CPOE.Objectives:
Thus, our study was done to identify and prioritize the CDS functionalities in CPOE.Methods:
A Two-round Modified Delphi process was used. Firstly, a systematic search was conducted in electronic databases from the date of database inception to February 2019 for identifying CDS functionalities integrated into CPOE. Studies were retrieved from databases, including PubMed, Embase, ProQuest, Scopus, Web of Science, Cochrane, Science Direct, ACM digital library, and IEEE Xplore Digital Library. Secondly, the Modified Delphi method was carried out in 2019 to provide contextual priorities regarding CDS functionalities in CPOE in two iterative rounds. A total of 12 experts working in the three Intensive Care Units (ICUs) with more than three years of experience with homegrown CPOE in Shiraz Nemazee Teaching Hospital, including two clinical pharmacists, two health information management faculty member, four cases with critical care fellowship, and four critical care experts participated in the study. The ≥ 66.6% agreement was considered as the consensus level. SPSS software version 24 was used for statistical analysis.Results:
Totally, 327 studies, which met eligibility criteria were found. A number of 60 potential CDS functionalities in CPOE were identified from eligible studies. Also, 13 out of 60 CDS functionalities reached high priority consensus after 2 iterative Delphi rounds, including drug-allergy checking (83.3%), basic dosing guidance (75%), single dosing checking (66.7%), duplicate therapy checking (66.7%), drug-pregnancy alerts (75%), time-based alerts (66.7%), alert for deep vein thrombosis prophylaxis (66.7%), alerts for duplicate medication order checking (66.7%), drug-drug interaction checking (75%), intelligent dosing guidance based on the patients’ characteristics (66.7%), renal-drug problems checking (83.3%), drug-disease interaction checking (66.7%), and displaying medication/test cost (75%).Conclusions:
Our study identified high-priority CDS functionalities to be considered in the CPOE system from the viewpoint of multidisciplinary experts, especially in Iran. Results of this study may be beneficial to plan, design, and implement CDS functionalities in CPOE in the ICU.Keywords
Decision Support Systems Computerized Provider Order Entry CPOE Medical Order Entry Systems Intensive Care Units
1. Background
Today, the health care system is encountered with increasing health care service demands and reduced resources. High-quality information technology contributes to health care transformation through a reduction in utilizing unnecessary services and medication errors (1). Some studies addressed patient safety as one of the main health care challenges in each country (2, 3). The most frequent medication errors occur at prescribing medication stage in the Intensive Care Unit (ICU) (4). Computerized Provider Order Entry (CPOE) is a component of health information technology that can contribute to improving care quality by decreasing medication errors and reducing resource overutilization (5-7).
The main aim of CPOE is assurance regarding the provision of standardized, legible, and complete orders by the providers (8). However, health care organizations can achieve benefits of CPOE after integrating Clinical Decision Support (CDS) functionalities into CPOE (9-11). Nevertheless, the implementation of CDS with CPOE is imperfect in developing countries (12). Advanced and basic CDS functionalities in CPOE can improve care quality. Unfortunately, despite the potential for integration of the CDS into CPOE to improve the care process, patients’ outcome, and economic outcome, CDS functionalities in CPOE are not fully employed in healthcare organizations in developing countries (13, 14). So far, CDS has been increasingly built in CPOE because the CDS system can help the provider to avoid forgetting, avert harm to the patient, and increase adherence to standard care (2, 15).
Detecting the requirements regarding design, development, testing, and deployment of CDS functionalities in CPOE is costly and time-consuming and also requires vast resources and access to the patients’ longitudinal information and comprehensive medical record (16). Moreover, the barriers related to clinical software development, such as limited resources, inadequate budget, and insufficiently skilled programmers, among others, make prioritization of clinical software requirements important (17). Post-implementation of CDS in CPOE needs updating the knowledge base, upgrading the infrastructures, changing the policies, and training the end-users (clinicians), which in turn limits the implementation of the CDS system. Providers in Nemazee Teaching Hospital have used the CPOE to enter all medical orders, including medication orders, radiology imaging orders, laboratory orders, blood bank orders, and general orders in the computer by the providers since October 2015 (18). CDS functionalities in CPOE need to be identified by the institutional healthcare providers in developing countries. In this regard, CDS functionalities should be a priority for the execution of CPOE. On the other hand, the Agile method is used to develop a CDS system; determining the priority of the requirements is an important step in the agile process.
2. Objectives
Thus, the primary aim of this study was to identify CDS functionalities in CPOE. Our secondary purpose was prioritizing CDS functionalities to implement in CPOE based on the provider’s viewpoint.
3. Methods
To obtain a consensus from the experts about the high priority of CDS functionality into CPOE, a two-round modified Delphi process was applied. Firstly, we conducted a systematic review to identify CDS functionalities integrated into CPOE. Studies about CDS functionalities integrated into CPOE published in English until February 2019 were included in this study. On the other hand, theses/dissertations, proceeding papers, conference papers, unpublished papers, and those written in non-English languages were excluded.
A systematic search was conducted in electronic databases, including PubMed, Embase, ProQuest, Scopus, Web of Science, Cochrane, Science Direct, ACM Digital Library, and IEEE Xplore Digital Library from February 2019. Table 1 presents the search strategy of this review. The Endnote software was used to manage the references.
Search Strategy for Identifying Clinical Decision Functionalities on A Computerized Provider Order Entry
| Search Strategy | |
|---|---|
| Information databases | Pubmed, Embase, Scopus, Web of Science (WOS), Cochrane, Proquest, ACM Digital Library, IEEE Xplore, Science Direct |
| Search strategy | |
| #1 | “Medical order entry systems” OR “CPOE” OR “Computerized order entry” OR “Computerized prescriber order entry” OR “Computerized provider order entry” OR “Electronic order” OR “Electronic prescribing” OR “Electronic physician order entry” OR “Computerized physician order entry” |
| #2 | “clinical decision support systems” OR “Clinical computerized decision support systems” OR “Decision-support systems” OR “Reminder systems” OR “Computer-assisted decision-making” OR “computer-assisted therapy” OR “Expert systems” OR “Alert system” OR “clinical decision support alerts” OR “formulary decision support” |
Two authors screened the titles and abstracts of the studies. After the first stage of screening, papers were retrieved and reviewed. Relevant information of each study was extracted based on a collection form, which included the journal name, title of the paper, study design, study date, study sample, data collection tools, CDS functionalities integrated into CPOE, and conclusion. All forms were synthesized item by item. In the extraction stage, one of the authors inserted the data into the form and the other authors reexamined them. In the case of disagreements between the two authors, a co- author was asked to resolve them. The extracted data were categorized according to the type of medical orders. The CDS functionalities integrated into CPOE were synthesized through a narrative review.
Secondly, a modified Delphi method was conducted to provide contextual priorities regarding CDS functionalities in CPOE in order to address the second aim of the study. This method has been applied in many studies to rank patient safety according to the scope and status (2, 19, 20). It has also been used to develop a framework in order to support the implementation of the Qatar National Vision 2030 (21). The setting of the study was ICU in the large 700 -bed Nemazee Teaching Hospital, which is the largest tertiary hospital in the south of Iran. Providers have used the homegrown Electronic Medical Record (EMR) in the three ICUs at Shiraz Nemazee Teaching Hospital since October 2015. This homegrown EMR software is a web-based system. CPOE component was developed by involving the end-users, attending physicians, residents, nurses, and security staff of the unit. This EMR software encompasses demographic information, problem lists, nursing assessment, electronic medication administration record, test results, and CPOE components. CPOE module included all orders in the electronic structured format: medication orders, general orders, clinical imaging orders, test orders, and blood bank orders, but CDS functionalities have not been combined in the CPOE (8, 13). The ICU personnel in Nemazee Hospital are familiar with the CPOE system. Thus, researchers decided to select the personnel of these three ICUs as research participants to determine the priorities regarding the integration of CDS functionalities into the CPOE. This study was carried out in 2019.
The participants were selected from different stakeholders with experience in CPOE and ICU. They were identified based on the CPOE steering committee (22), including (1) critical care academic members and with intensive care unit fellowship who had experience in CPOE and ICU; (2) clinical pharmacists who had experience in CPOE and ICU; and (3) health information management academic members who had experience in CPOE. With regard to the criteria, a list of 12 experts was identified based on the feasibility of access. The samples of this study included twelve cases working in the three ICUs at Nemazee Hospital with more than 3 years of experience in homegrown CPOE in Shiraz Nemazee Teaching Hospital who met the inclusion criteria. Seven out of 12 participants were male, and the mean work experience of the subjects was 16 years.
The study was performed in the following steps:
1) To identify CDS functionalities in a CPOE system, a comprehensive review was carried out using electronic information databases, and the search strategy is presented in Table 1.
2) The list of CDS functionalities was developed by MK, as shown in Table 2. The list of CDS functionalities was classified based on the type of medical orders. All authors reviewed and approved the questionnaires. CDS functionalities in CPOE identified in the previous stage were used by assigning numbers to them in order to evaluate the perceptions about the priority of requirements in the implementation of CDS functionalities with CPOE, and a simple method was used for prioritization of CDS functionalities in the CPOE system.
Clinical Decision Support Functionalities in Computerized Provider Order Entry From Literature Review
| Number | Functionalities |
|---|---|
| Medication Order Decision Support | |
| 1 | Drug-allergy checking |
| 2 | Basic dosing guidance |
| 3 | Single dose range checking |
| 4 | Maximum daily dose checking |
| 5 | Maximum lifetime dose checking |
| 6 | Default doses/pick lists |
| 7 | Indication-based dosing (7.5 mg methotrexate once weekly for rheumatoid) |
| 8 | decision support for the recommended route of administration (oral or intravenous routes) |
| 9 | Formulary decision support (drugs covered by hospital or patient’s insurance) |
| 10 | Duplicate therapy checking |
| 11 | Drug-drug interaction checking, drug-herb interaction checking |
| 12 | Intelligent dosing guidance (based on patient’s characteristics) |
| 13 | renal-drug problems checking |
| 14 | Drug-food interaction checking |
| 15 | Drug-lab alert |
| 16 | drug-disease interactions and contraindications checking |
| 17 | Drug-pregnancy checking |
| 18 | Decision support supporting drug prescription during breastfeeding |
| 19 | Drug-patient age checking |
| 20 | Guiding drug selection or dosing based on genetic profiles |
| 21 | Support for optimal drug selection based on the indication |
| 22 | Plan of care alerts (reminders to reassess the need for restraints and reorder if necessary at least every 24 h). |
| 23 | Reminders to order a diagnostic or therapeutic procedure based on patient’s parameters |
| 24 | Look-alike/sound-alike medication warning |
| 25 | Time-based alerts that an order has not been fully carried out |
| 26 | Problem list management |
| 27 | High-risk state monitoring |
| 28 | Polypharmacy alerts (suggesting consultant pharmacist) |
| 29 | Chemotherapy prescription clinical decision-support systems |
| 30 | computerized decision support on antibiotic use |
| 31 | Alert for the use of thrombolytic prophylaxis |
| Order Facilitator | |
| 32 | Medication order sentences/medication or test order set |
| 33 | Subsequent or corollary orders |
| 34 | Service-specific order sets |
| 35 | Condition-specific order sets |
| 36 | Procedure-specific order sets |
| 37 | Condition-specific treatment protocol |
| 38 | Transfer order set |
| Relevant Information Display | |
| 39 | Context-sensitive information retrieval |
| 40 | Patient’s specific relevant data display |
| 41 | Medication/test cost display |
| 42 | Tall man lettering |
| 43 | Context-sensitive user interface |
| Laboratory Test Order Decision Support | |
| 44 | Decision support for detecting unnecessary laboratory test order |
| 45 | Prediction of test abnormalities |
| 46 | Test cost display |
| 47 | Decision support for detecting duplicated test orders |
| Clinical Decision Support for Imaging Ordering | |
| 48 | Indicating Appropriateness of imaging order |
| 49 | Diagnostic imaging cost display |
| 50 | Clinical decision-making tools for exam selection |
| 51 | real-time computerized duplicate diagnostic imaging order alert |
| 52 | Displaying radiation dose during order |
| Blood product order (transfusion service) | |
| 53 | Real-time clinical decision support for red blood cell |
| 54 | Real-time clinical decision support for decreased inappropriate plasma transfusion |
| 55 | Computerized decision support systems to promote the appropriate use of blood products |
| 56 | Real-time clinical decision support systems for platelet and cryoprecipitate order |
| Expert System | |
| 57 | Antibiotic suggestions based on patient history, Gram stain results on antimicrobial therapy, culture results, and patient characteristics |
| 58 | Ventilator suggestions based on patient-specific blood gas readings and current condition |
| 59 | Recommendations regarding the appropriateness of transfusion and suggested products and dosing based on clinical indications |
| 60 | Tools, calculators, guidelines, and protocols for ordering total parenteral nutrition (TPN), enteral nutrition or other alimentation procedures |
| Workflow Support | |
| 61 | Automatic termination of orders after a fixed period of time |
| 62 | Applying logic and route orders for special approval based on order type, ordering provider, or patient characteristics. |
| 63 | Parsing tools to translate free-text orders into structure representations |
| Others | |
| 64 | Providers were prompted at order entry to specify the indication for urinary catheter insertion. |
3) The list of CDS functionalities was approved by the research team, and a pilot survey was conducted, in which the physicians’ feedbacks were obtained to investigate face validity. In the final stage, the main list of papers was sent to the study participants.
4) Demographic information, including the participant’s job, s' specialties, and prior experience with CPOE were considered as the inclusion criteria.
Participants were asked to rate their agreement with each CDS functionality on a 3-point Likert scale, which ranged from “low priority” to “high priority.” Each question also consisted of open-ended questions, and the participants were asked to provide their comments. Additionally, they were asked to recommend any CDS functionality if needed.
The authors achieved consensus through a modified Delphi ranking method in 2 iterative rounds. A questionnaire was sent to all the experts in the first round. The research team members were requested to complete and return the questionnaire within 2 weeks for rounds 1 and 2; the questionnaire consisted of a cover letter, participants’ demographic data, and a list of 60 CDS functionalities in CPOE. CDS functionalities were divided into medication-related decision support, medical imaging order decision support, laboratory test orders decision support, transfusion orders decision support, expert system integrated into CPOE, and other CDS functionalities in the CPOE system, such as urinary catheter reminder. Also, 43 out of 60 CDS functionalities belonged to electronic medication orders.
Descriptive statistics, including mean, median, mode, and variability (standard deviation, mean, and range) were done to assess the participants' demographic information. Moreover, descriptive statistics were used to evaluate the consensus agreement on the importance of CDS functionalities, including measures of agreement percentage. An agreement of ≥ 66.6 was considered as the consensus level. SPSS software version 24 was used for statistical analysis.
Ethical approval for this study was obtained from the Ethics Committee of Shiraz University of Medical Sciences (approval ID: IR.SUMS.REC.1398.1046).
4. Results
The literature search was conducted in February 2019, by which a total of 10715 potential relevant papers were initially identified; of which 7904 records remained after the removal of duplicates. The three-step review process was conducted (title screening, abstract screening, and full-text screening). In the first and second stages, 2 reviewers screened the titles and abstracts of all the papers for relevant studies, and studies that did not meet the inclusion criteria were eliminated. After the first and second stages of screening, 701 papers were retrieved and reviewed, and then, the full-text of the papers was screened, and finally, 327 studies remained. Figure 1 shows the flow diagram of the study selection.
PRISMA flow diagram. CPOEm Computerized Provider Order Entry system; CDSS, Clinical Decision Support system.
Mostly addressed CDS functionalities in CPOE belonged to medication order decision supports, including drug-drug interaction checking (n = 59), basic dosing guidance (n = 37), renal-drug problems checking (n = 36), decision support for antibiotic use (n = 27), order set (n = 24), indicating the appropriateness of imaging orders (n = 23), drug-lab alert (n = 22), drug-allergy checking (n = 21), formulary decision support (n = 19), decision support systems to promote the appropriate use of blood bank (n = 17), and drug-disease interaction checking (n = 16). Table 2 presents potential CDS functionalities in CPOE.
The survey was sent to 12 participants who met the inclusion criteria. Three professors at Shiraz University of Medical Sciences (one ICU professor and 2 professors in the field of health information technology) were asked to revise and evaluate the prioritization tools. The 3-point Likert scale questionnaires were employed to determine the priority of 60 potential alerts.
Respondents were asked to score 60 potential CDS functionalities in medical orders using a Likert-type scale in the first round, and there was a suggestion section in the prioritization tools. The mean score for each CDS functionality in medical orders ranged from 1.33 to 2.75. Table 3 presents the frequency of answers regarding prioritization of each CDS functionality for all participants. In round 1, 8 out of 60 CDS functionalities reached high priority consensus (> 66.6%), including drug-allergy checking (83.3%), basic dosing guidance (75%), single dosing checking (66.7%), duplicate therapy (66.7%), drug-pregnancy alerts (75%), tickers (time-based alerts indicating that an order has not been fully carried out) (66.7%), alert for deep vein thrombosis (DVT) prophylaxis (66.7%), and alerts used for the duplicate test (66.7%) in the first round (more than 66.6% of the responders scored high priority). However, drug selection guidance or dosing guidance based on genetic profiles reached a consensus low priority (75%) in the first round (Table 3).
Clinical Decision Support Survey Results (Round 1)
| Number | Potential Clinical Decision Support for Medication Orders | Low Priority, No. (%) | Medium Priority, No. (%) | High Priority, No. (%) | Total Priority Score, Mean ± SD | Decision |
|---|---|---|---|---|---|---|
| 1 | Drug-allergy checking | 1 (8.3) | 1 (8.3) | 10 (83.3) | 2.75 ± 0.62 | Selected |
| 2 | Basic dosing guidance | 1 (8.3) | 2 (16.7) | 9 (75) | 2.66 ± 0.65 | Selected |
| 3 | Single dose range checking | 2 (16.7) | 2 (16.7) | 8 (66.7) | 2.50 ± 0.79 | Selected |
| 4 | Maximum daily dose checking | 2 (16.7) | 4 (33.3) | 6 (50) | 2.33 ± 0.77 | Discuss |
| 5 | Maximum lifetime dose checking | 2 (18.2) | 5 (45.5) | 4 (36.4) | 2.18 ± 0.75 | Discuss |
| 6 | Default doses/pick lists | 4 (33.3) | 3 (25) | 5 (41.7) | 2.08 ± 0.90 | Discuss |
| 7 | Indication-based dosing (7.5 mg methotrexate once weekly for rheumatoid) | 3 (25) | 2 (16.7) | 7 (58.3) | 2.33 ± 0.88 | Discuss |
| 8 | Decision support supporting route of administration (IV to PO) | 1 (8.3) | 4 (33.3) | 7 (58.3) | 2.50 ± 0.67 | Discuss |
| 9 | Formulary decision support (drugs covered by hospital or patient’s insurance) | 2 (16.7) | 4 (33.3) | 6 (50) | 2.33 ± 0.77 | Discuss |
| 10 | Duplicate therapy checking | 1 (8.3) | 3 (25) | 8 (66.7) | 2.58 ± 0.66 | Selected |
| 11 | Drug-drug interaction checking | 1 (8.3) | 5 (41.7) | 6 (50) | 2.41 ± 0.66 | Discuss |
| 12 | Drug-herb interaction checking | 5 (41.7) | 2 (16.7) | 5 (41.7) | 2.00 ± 0.95 | Discuss |
| 13 | Intelligent dosing guidance (based on patient characteristics) | 2 (16.7) | 5 (41.7) | 5 (41.7) | 2.25 ± 0.75 | Discuss |
| 14 | Renal-drug problem checking | 2 (16.7) | 3 (25) | 7 (58.3) | 2.41 ± 0.79 | Discuss |
| 15 | Drug-food interaction checking | 3 (25) | 5 (41.7) | 4 (33.3) | 2.08 ± 0.79 | Discuss |
| 16 | Drug-lab alert | 2 (16.7) | 3 (25) | 7 (58.3) | 2.41 ± 0.79 | Discuss |
| 17 | Drug-disease interactions and contraindications checking | - | 5 (41.7) | 7 (58.3) | 2.58 ± 0.51 | Discuss |
| 18 | Drug-pregnancy alert | 2 (16.7) | 1 (8.3) | 9 (75) | 2.58 ± 0.79 | Selected |
| 19 | Decision support for drug prescription during breastfeeding | 3 (25) | 2 (16.7) | 7 (58.3) | 2.33 ± 0.88 | Discuss |
| 20 | Drug-patient age checking | 1 (8.3) | 7 (58.3) | 4 (33.3) | 2.25 ± 0.62 | Discuss |
| 21 | Guiding drug selection or dosing based on genetic profiles | 9 (75) | 2 (16.7) | 1 (8.3) | 1.33 ± 0.65 | Excluded |
| 22 | Support for optimal drug selection based on indication | 4 (33.3) | 4 (33.3) | 4 (33.3) | 2.00 ± 0.85 | Discuss |
| 23 | Plan of care alerts | 3 (25) | 3 (25) | 6 (50) | 2.25 ± 0.86 | Discuss |
| 24 | Care reminder (reminders to order a diagnostic or therapeutic procedure based on patient’s parameters) | 3 (25) | 7 (58.3) | 2 (16.7) | 1.91 ± 0.66 | Discuss |
| 25 | Look-alike/sound-alike medication warning | 2 (16.7) | 5 (41.7) | 5 (41.7) | 2.25 ± 0.75 | Discuss |
| 26 | Time-based alerts that an order has not been fully carried out | 1 (8.3) | 3 (25) | 8 (66.7) | 2.58 ± 0.66 | Selected |
| 27 | Problem list management | 5 (41.7) | 4 (33.3) | 3 (25) | 1.83 ± 0.83 | Discuss |
| 28 | Polypharmacy alerts (suggesting consultant pharmacist) | 5 (41.7) | 5 (41.7) | 2 (16.7) | 1.75 ± 0.75 | Discuss |
| 29 | Chemotherapy prescription clinical decision-support systems | 7 (58.3) | 3 (25) | 2 (16.7) | 1.58 ± 0.79 | Discuss |
| 30 | Computerized decision support on antibiotic use | 3 (25) | 2 (16.7) | 7 (58.3) | 2.33 ± 0.88 | Discuss |
| 31 | Alert for the use of thrombolytic prophylaxis | 1 (8.3) | 3 (25) | 8 (66.7) | 2.58 ± 0.66 | Selected |
| Order Facilitator | ||||||
| 32 | Medication order sentences. medication or test order set | - | 5 (45.5) | 6 (54.5) | 2.54 ± 0.52 | Discuss |
| 33 | Subsequent or corollary orders | 1 (9.1) | 3 (27.3) | 7 (63.6) | 2.54 ± 0.68 | Discuss |
| 34 | Service-specific order sets | 4 (36.4) | 3 (27.3) | 4 (36.4) | 2.00 ± 0.89 | Discuss |
| 35 | Condition-specific order sets | 3 (27.3) | 4 (36.4) | 4 (36.4) | 2.09 ± 0.83 | Discuss |
| 36 | Procedure-specific order sets | 2 (16.7) | 5 (41.7) | 5 (41.7) | 2.25 ± 0.75 | Discuss |
| 37 | Condition-specific treatment protocol | 1 (8.3) | 7 (58.3) | 4 (33.3) | 2.25 ± 0.62 | Discuss |
| 38 | Admission/transfer order set | 4 (33.3) | 3 (25) | 5 (41.7) | 2.08 ± 0.90 | Discuss |
| Relevant Information Display | ||||||
| 39 | Context sensitive information retrieval | 2 (16.7) | 4 (33.3) | 6 (50) | 2.33 ± 0.77 | Discuss |
| 40 | Patient specific relevant data display | 2 (16.7) | 3 (25) | 7 (58.3) | 2.41 ± 0.79 | Discuss |
| 41 | Medication/test cost display | 4 (33.3) | 5 (41.7) | 3 (25) | 1.91 ± 0.79 | Discuss |
| 42 | Decision for detecting unnecessary Laboratory test order | 6 (50) | 2 (16.7) | 4 (33.3) | 1.83 ± 0.93 | Discuss |
| 43 | Prediction of test abnormalities | 3 (25) | 5 (41.7) | 4 (33.3) | 2.08 ± 0.79 | Discuss |
| 44 | Test cost display | 5 (41.7) | 5 (41.7) | 2 (16.7) | 1.75 ± 0.75 | Discuss |
| 45 | decision for detecting duplicated test orders | 3 (25) | 1 (8.3) | 8 (66.7) | 2.41 ± 0.90 | Selected |
| CDSS During Imaging Order | ||||||
| 46 | Indication appropriateness imaging order | 4 (33.3) | 4 (33.3) | 4 (33.3) | 2.00 ± 0.85 | Discuss |
| 47 | Diagnostic imaging cost display | 5 (41.7) | 6 (50) | 1 (8.3) | 1.66 ± 0.65 | Discuss |
| 48 | Clinical decision-making tools for exam selection | 5 (41.7) | 5 (41.7) | 2 (16.7) | 1.75 ± 0.75 | Discuss |
| 49 | Real-time computerized duplicate Diagnostic imaging order alert | 3 (25) | 2 (16.7) | 7 (58.3) | 2.33 ± 0.88 | Discuss |
| 50 | Display radiation dose during order | 4 (33.3) | 4 (33.3) | 4 (33.3) | 2.00 ± 0.85 | Discuss |
| CDSS During Blood Bank Ordering | ||||||
| 51 | Real-time clinical decision support for red blood cell | 5 (41.7) | 2 (16.7) | 5 (41.7) | 2.00 ± 0.95 | Discuss |
| 52 | Real-time clinical decision support for decreased inappropriate plasma transfusion | 5 (41.7) | 2 (16.7) | 5 (41.7) | 2.00 ± 0.95 | Discuss |
| 53 | Real-time clinical decision support systems for platelet and cryoprecipitate order | 4 (33.3) | 2 (16.7) | 6 (50) | 2.16 ± 0.93 | Discuss |
| Potential Expert System During Medical Order | ||||||
| 54 | Antibiotic suggestions based on patient’s history, Gram stain results on antimicrobial therapy, culture results, and patient’s characteristics | 3 (25) | 4 (33.3) | 5 (41.7) | 2.16 ± 0.83 | Discuss |
| 55 | Ventilator suggestions based on patient-specific blood gas readings and current condition | 5 (41.7) | 4 (33.3) | 3 (25) | 1.83 ± 0.83 | Discuss |
| 56 | Tools, calculators, guidelines, and protocols for ordering total parenteral nutrition (TPN), enteral nutrition or other alimentation procedures | 1 (8.3) | 5 (41.7) | 6 (50) | 2.41 ± 0.66 | Discuss |
| Potential Workflow Support | ||||||
| 57 | Automatic termination of orders after a fixed period of time | 3 (25) | 5 (41.7) | 4 (33.3) | 2.08 ± 0.79 | Discuss |
| 58 | Logic Applying and route orders for special approval based on order type, order provider, or patient’s characteristics | 2 (16.7) | 4 (33.3) | 6 (50) | 2.33 ± 0.77 | Discuss |
| 59 | Parsing tools to translate free-text orders into structure representations | 5 (41.7) | 2 (16.7) | 5 (41.7) | 2.00 ± 0.95 | Discuss |
| Others | ||||||
| 60 | Providers were prompted at order entry to specify the indication for urinary catheter insertion | 2 (16.7) | 3 (25) | 7 (58.3) | 2.41 ± 0.79 | Discuss |
Feedbacks and mean total priority scores of each potential CDS functionality that did not achieve agreement were sent to the experts. A 2-week period was provided for the participants to repeat the survey, choosing whether to change their priority scores based on mean total priority scores of each CDS or keep their mean scores. Also, 51 out of 60 CDS functionalities in medical orders that did not reach consensus were presented again in round 2. In round 2, 5 out of 51 CDS functionalities in medical orders achieved high priority consensus, including drug-drug interaction checking (75%), intelligent dosing guidance (based on patients’ characteristics) (66.7%), renal-drug problems checking (83.3%), drug-disease interaction (66.7%), and displaying medication/test cost (75%) (Table 4).
Clinical Decision Support Survey Results (Round 2)
| Potential Clinical Decision Support for Medication Orders | Low Priority, No. (%) | Medium Priority, No. (%) | High Priority, No. (%) | Total Priority Score, Mean ± SD | Sum Response | |
|---|---|---|---|---|---|---|
| 1 | Maximum daily dose checking | 1 (8.3) | 5 (41.7) | 6 (50) | 2.41 ± 0.66 | 29 |
| 2 | Maximum lifetime dose checking | 1 (8.3) | 4 (33.3) | 7 (58.3) | 2.50 ± 0.67 | 30 |
| 3 | Default doses/pick lists | 2 (16.7) | 4 (33.3) | 6 (50) | 2.33 ± 0.77 | 28 |
| 4 | Indication-based dosing (7.5 mg methotrexate once weekly for rheumatoid) | 3 (25) | 3 (25) | 6 (50) | 2.25 ± 0.86 | 27 |
| 5 | decision supporting for recommended route of administration (oral or intravenous routes) | 3 (25) | 4 (33.3) | 5 (41.7) | 2.16 ± 0.83 | 26 |
| 6 | Formulary decision support (drugs covered by hospital or patient’s insurance) | 4 (33.3) | 3 (25) | 5 (41.7) | 2.08 ± 0.90 | 25 |
| 7 | Drug-drug interaction checking | - | 3 (25) | 9 (75) | 2.75 ± 0.45 | 33 |
| 8 | Drug-herb interaction checking | 2 (16.7) | 6 (50) | 4 (33.3) | 2.16 ± 0.71 | 26 |
| 9 | Intelligent dosing guidance (based on patient’s characteristics) | 2 (16.7) | 2 (16.7) | 8 (66.7) | 2.50 ± 0.79 | 30 |
| 10 | Renal-drug problem checking | 1 (8.3) | 1 (8.3) | 10 (83.3) | 2.75 ± 0.62 | 33 |
| 11 | Drug-food interaction checking | 2 (16.7) | 9 (75) | 1 (8.3) | 1.91 ± 0.51 | 23 |
| 12 | Drug-lab alert | 2 (16.7) | 5 (41.7) | 5 (41.7) | 2.25 ± 0.75 | 27 |
| 13 | Drug-disease interactions and contraindications checking | 3 (25) | 1 (8.3) | 8 (66.7) | 2.41 ± 0.90 | 29 |
| 14 | Decision supporting drug prescription during breastfeeding | 2 (16.7) | 3 (25) | 7 (58.3) | 2.41 ± 0.79 | 29 |
| 15 | Drug-patient age checking | 1 (8.3) | 9 (75) | 2 (16.7) | 2.08 ± 0.51 | 25 |
| 16 | Support for optimal drug selection based on indication | 2 (16.7) | 8 (66.7) | 2 (16.7) | 2.00 ± 0.60 | 24 |
| 17 | Plan of care alerts (reminders to reassess the need for restraints and reorder if necessary at least every 24 h). | 1 (8.3) | 6 (50) | 5 (41.7) | 2.33 ± 0.65 | 28 |
| 18 | Reminders to order a diagnostic or therapeutic procedure based on patient’s parameters | 1 (9.1) | 6 (54.5) | 4 (036.4) | 2.27 ± 0.64 | 25 |
| 19 | Look-alike/sound-alike medication warning | 1 (8.3) | 4 (33.3) | 7 (58.3) | 2.50 ± 0.67 | 30 |
| 20 | Problem list management | 2 (16.7) | 5 (41.7) | 5 (41.7) | 2.25 ± 0.75 | 27 |
| 21 | Polypharmacy alerts (suggest consultant pharmacist) | - | 7 (58.3) | 5 (41.7) | 2.41 ± 0.51 | 29 |
| 22 | Chemotherapy prescription clinical decision-support systems | 3 (25) | 5 (41.7) | 4 (33.3) | 2.08 ± 0.79 | 25 |
| 23 | Computerized decision support on antibiotic use | - | 7 (58.3) | 5 (41.7) | 2.41 ± 0.51 | 29 |
| Order Facilitator | ||||||
| 24 | Medication order sentences/medication or test order set | 3 (25) | 5 (41.7) | 4 (33.3) | 2.08±0.79 | 25 |
| 25 | Subsequent or corollary orders | 1 (8.3) | 5 (41.7) | 6 (50) | 2.41 ± 0.66 | 29 |
| 26 | Service-specific order sets | (25)3 | 5 (41.7) | 4 (33.3) | 2.08 ± 0.79 | 25 |
| 27 | Condition-specific order sets | 2 (16.7) | 5 (41.7) | 5 (41.7) | 2.25 ± 0.75 | 27 |
| 28 | Procedure-specific order sets | 1 (8.3) | 6 (50) | 5 (41.7) | 2.33 ± 0.65 | 28 |
| 29 | Condition-specific treatment protocol | 3 (25) | 6 (50) | 3 (25) | 2.00 ± 0.73 | 24 |
| 30 | Transfer order set | 2 (16.7) | 8 (66.7) | 2 (16.7) | 2.00 ± 0.60 | 24 |
| Relevant Information Display | ||||||
| 31 | Context sensitive information retrieval | 2 (16.7) | 7 (58.3) | 3 (25) | 2.08 ± 0.66 | 25 |
| 32 | Patient specific relevant data display | 1 (8.3) | 7 (58.3) | 4 (33.3) | 2.25 ± 0.62 | 27 |
| 33 | Medication/test cost display | 1 (8.3) | 2 (16.7) | 9 (75) | 2.66 ± 0.65 | 32 |
| 34 | Decision support for detecting unnecessary laboratory test order | 2 (16.7) | 3 (25) | 7 (58.3) | 2.41 ± 0.79 | 29 |
| 35 | Prediction of test abnormalities | 2 (16.7) | 7 (58.3) | 3 (25) | 2.08 ± 0.66 | 25 |
| 36 | Test cost display | 2 (16.7) | 5 (41.7) | 5 (41.7) | 2.25 ± 0.75 | 27 |
| CDSS During Image Orders | ||||||
| 37 | Indication appropriateness imaging order | 3 (25) | 7 (58.3) | 2 (16.7) | 1.91 ± 0.66 | 23 |
| 38 | Diagnostic imaging cost display | 5 (41.7) | 5 (41.7) | 2 (16.7) | 1.75 ± 0.75 | 21 |
| 39 | Clinical decision-making tools for exam selection | 5 (41.7) | 5 (41.7) | 2 (16.7) | 1.75 ± 0.75 | 21 |
| 40 | Real-time computerized duplicate diagnostic imaging order alert | 5 (41.7) | 2 (16.7) | 5 (41.7) | 2.00 ± 0.95 | 24 |
| 41 | Display radiation dose during order | 5 (41.7) | 5 (41.7) | 2 (16.7) | 1.75 ± 0.75 | 21 |
| CDSS During Blood Bank Order | ||||||
| 42 | Real-time clinical decision support for red blood cell | 3 (25) | 7 (58.3) | 2 (16.7) | 1.91 ± 0.66 | 23 |
| 43 | Real-time clinical decision support decreases inappropriate plasma transfusion | 3 (25) | 7 (58.3) | 2 (16.7) | 1.91 ± 0.66 | 23 |
| 44 | Real-time clinical decision support systems for platelet and cryoprecipitate order | 2 (16.7) | 7 (58.3) | 3 (25) | 2.08 ± 0.67 | 25 |
| Expert System | ||||||
| 45 | Antibiotic suggestions based on patient history, hospital antibiotic gram, culture results, and patient characteristics | 2 (16.7) | 3 (25) | 7 (58.3) | 2.41 ± 0.79 | 29 |
| 46 | Ventilator suggestions based on patient-specific blood gas readings and current condition | 6 (50) | 4 (33.3) | 2 (16.7) | 1.66 ± 0.77 | 20 |
| 47 | Tools, calculators, guidelines, and protocols for ordering total parenteral nutrition (TPN), enteral nutrition or other alimentation procedures) | 2 (16.7) | 7 (58.3) | 3 (25) | 2.08 ± 0.66 | 25 |
| Workflow Support | ||||||
| 48 | Automatic termination of orders after a set period of time | 1 (8.3) | 5 (41.7) | 6 (50) | 2.41 ± 0.66 | 29 |
| 49 | Applying logic and route orders for special approval based on order type, ordering provider, or patient’s characteristics. | 1 (8.3) | 6 (50) | 5 (41.7) | 2.33 ± 0.65 | 28 |
| 50 | Parsing tools to translate free-text orders into structure representations | 2 (16.7) | 9 (75) | 1 (8.3) | 1.91 ± 0.51 | 23 |
| Others | ||||||
| 51 | Providers were prompted at order entry to specify the indication for urinary catheter insertion | - | 8 (66.7) | 4 (33.3) | 2.33 ± 0.49 | 28 |
5. Discussion
In this study, CDS functionalities in CPOE in ICU were identified and prioritized. The most important CDS functionalities for increasing the patient’s safety were identified by consensus with the group of experts in a modified Delphi rating process. High priority CDS functionality that needs to be embedded into CPOE was obtained considering end-users’ opinion about CDS priority in CPOE to acknowledge the end-users’ voice concerning their task and role. This is in line with other instances of design and implementation of CDS, which have been successful as they have considered the provider involvement in the identification of requirements, design, development, and evaluation of the system.
In this study, 13 out of 60 CDS functionalities reached high priority consensus, including drug-allergy checking, basic dosing guidance, single dosing checking, duplicate therapy, drug-pregnancy alerts, tickers (time-based alerts indicating that an order has not been fully carried out), alert for DVT prophylaxis, alerts for the duplicate test, drug-drug interaction checking, intelligent dosing guidance (based on patients’ characteristics), renal-drug problems checking , drug-disease interaction alerts, and displaying medication/test cost. Our study results are consistent with those of Kuperman et al. (23) and Tolley et al. (24). Kuperman et al. (23) introduced basic and advanced medication CDS in CPOE so that basic decision support included drug-allergy checking, basic dosing guidance, formulary decision support, duplicate therapy checking, and drug–drug interaction checking. Advanced decision support included dosing support for renal failure, drug-lab checking, drug-pregnancy checking, and drug-disease contraindication. Although developing countries have implemented CPOE with CDSS, many researchers have reported that CPOE has not been integrated into the CDS system in Iran. For example, Rabiei et al. (14) addressed this issue in Iran. In relation to the CDS functionalities of CPOE, they indicated that the current EMR software in Iran lacks those functionalities (14). They highlighted the need for developing the CDS functionalities in the CPOE in Iran. In addition, Eslami et al. showed that CPOE did not consist of CDS functionalities. However, Ferranez et al. (25) showed that medication clinical decision support integrated into CPOE increases the inpatient safety and quality of care.
CDS functionalities integrated into the CPOE in ICU can decrease the medication error, ICU mortality rate, hospital mortality rate, length of stay in ICU, and length of stay in the hospital so that these functionalities increase the patient safety, quality of care and decrease inappropriate services (26-28). In this study, most CDS functionalities that reached high priority consensus from the viewpoint of intensivists belonged to the medication-related decision support system. Hospitalization of the patients in ICU is at high risk of medication errors and adverse outcomes (2, 29). Thus, in this study, the experts were asked to highlight important medication-related CDS functionalities in ICU. CDS integrated into the CPOE system is one of the interventions used to prevent medication error and inappropriate order at prescribing stage in hospitals.
Most CDS functionalities reached a high priority consensus that belonged to medication orders and were attributed to the effect of medication CDS on clinical outcomes, such as medication errors and adverse drug outcomes. However, lab test orders-related decision support and medical imaging orders decision support did not achieve high priority consensus because they have significant effects on the cost, process, and outcome of care while their use may contribute to the transformation of health care. In addition, some CDS functionalities related to medication orders, such as drug-patient age checking, drug-food interaction checking, and polypharmacy alerts, did not reach a consensus since they are not essential in ICU.
This study is the first modified Delphi study in Iran, which investigated the prioritization of CDS functionalities built within CPOE from the viewpoint of multidisciplinary experts. End-user involvement in the identification of requirements, design, and development of CDS functionalities is indispensable to achieve success in CDS implementation. Thus, high-priority CDS functionalities may be useful to plan for CDS implementation. CDS designers and managers in the field of health information technology should pay attention to priorities proposed by the end-users in ICU. However, there were several limitations in this study. First, the sample size of the study was small because there was no other hospital expert to meet the inclusion criteria related to prior experience with CPOE in this region. There was individual selection bias because the included experts were not randomly selected. There are different prioritization methods, but we used the modified Delphi method for ranking CDS in ICU. Thus, further studies are recommended for prioritization of CDS functionalities using other methods, such as Analytic Hierarchy process (AHP), Machine Learning, Top-10 Selection, WinWin, etc. The results of this study may be beneficial to plan, design, and implement clinical decision support functionalities in CPOE.
Acknowledgements
References
-
1.
Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742-52. [PubMed ID: 16702590]. https://doi.org/10.7326/0003-4819-144-10-200605160-00125.
-
2.
Ravaghi H, Sajadi HS. Research priorities in the field of patient safety in iran: Results of a delphi study. J Patient Saf. 2019;15(2):166-71. [PubMed ID: 26076078]. https://doi.org/10.1097/PTS.0000000000000215.
-
3.
Zegers M, Hanskamp‐Sebregts M, Wollersheim H, Vincent C. Patient safety strategies. In: Wensing M, Grol R, Grimshaw J, editors. Improving Patient Care. 3rd ed. Wiley-Blackwell; 2020. p. 275-98. https://doi.org/10.1002/9781119488620.ch16.
-
4.
Vazin A, Fereidooni M. Determining frequency of prescription, administration and transcription errors in internal intensive care unit of Shahid Faghihi Hospital in Shiraz with direct observation approach. Iran J Pharm Sci. 2012;8(3):189-94.
-
5.
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163(12):1409-16. [PubMed ID: 12824090]. https://doi.org/10.1001/archinte.163.12.1409.
-
6.
Silvestri MT, Bongiovanni TR, Glover JG, Gross CP. Impact of price display on provider ordering: A systematic review. J Hosp Med. 2016;11(1):65-76. [PubMed ID: 26498736]. https://doi.org/10.1002/jhm.2500.
-
7.
Huckvale C, Car J, Akiyama M, Jaafar S, Khoja T, Bin Khalid A, et al. Information technology for patient safety. Qual Saf Health Care. 2010;19 Suppl 2:i25-33. [PubMed ID: 20693213]. https://doi.org/10.1136/qshc.2009.038497.
-
8.
Khammarnia M, Sharifian R, Zand F, Keshtkaran A, Barati O. Designing computerized provider order entry software in iran: The nurses' and physicians' viewpoints. Comput Inform Nurs. 2016;34(9):413-20. [PubMed ID: 27270630]. https://doi.org/10.1097/CIN.0000000000000250.
-
9.
Sheikh A. Realising the potential of health information technology to enhance medication safety. BMJ Qual Saf. 2020;29(1):7-9. [PubMed ID: 31519731]. https://doi.org/10.1136/bmjqs-2019-010018.
-
10.
Helmons PJ, Grouls RJ, Roos AN, Bindels AJ, Wessels-Basten SJ, Ackerman EW, et al. Using a clinical decision support system to determine the quality of antimicrobial dosing in intensive care patients with renal insufficiency. Qual Saf Health Care. 2010;19(1):22-6. [PubMed ID: 20172878]. https://doi.org/10.1136/qshc.2007.025700.
-
11.
Pontefract SK, Hodson J, Slee A, Shah S, Girling AJ, Williams R, et al. Impact of a commercial order entry system on prescribing errors amenable to computerised decision support in the hospital setting: a prospective pre-post study. BMJ Qual Saf. 2018;27(9):725-36. [PubMed ID: 29572298]. [PubMed Central ID: PMC6109251]. https://doi.org/10.1136/bmjqs-2017-007135.
-
12.
Niknam F, Khammarnia M, Zand F, Fallahnejad E, Sharifian R. Evaluating physicians and nurses’ satisfaction with computerized physician order entry system in Iran. J Health Inf. 2018;5(2):51-6.
-
13.
Khammarnia M, Sharifian R, Zand F, Barati O, Keshtkaran A, Sabetian G, et al. The impact of computerized physician order entry on prescription orders: A quasi-experimental study in Iran. Med J Islam Repub Iran. 2017;31:69. [PubMed ID: 29445698]. [PubMed Central ID: PMC5804463]. https://doi.org/10.14196/mjiri.31.69.
-
14.
Rabiei R, Moghaddasi H, Asadi F, Heydari M. Evaluation of computerized provider order entry systems: assessing the usability of systems for electronic prescription. Electron Physician. 2018;10(8):7196-204. [PubMed ID: 30214702]. [PubMed Central ID: PMC6122865]. https://doi.org/10.19082/7196.
-
15.
Medlock S, Eslami S, Askari M, Arts DL, van de Glind EM, Brouwer HJ, et al. For which clinical rules do doctors want decision support, and why? A survey of Dutch general practitioners. Health Inf J. 2019;25(3):1076-90. [PubMed ID: 29148311]. [PubMed Central ID: PMC6769284]. https://doi.org/10.1177/1460458217740407.
-
16.
Wolf M, Miller S, DeJong D, House JA, Dirks C, Beasley B. The process of development of a prioritization tool for a clinical decision support build within a computerized provider order entry system: Experiences from St Luke's Health System. Health Inf J. 2016;22(3):579-93. [PubMed ID: 25814483]. https://doi.org/10.1177/1460458215571769.
-
17.
Achimugu P, Selamat A, Ibrahim R, Mahrin MN. A systematic literature review of software requirements prioritization research. Inf Software Technol. 2014;56(6):568-85. https://doi.org/10.1016/j.infsof.2014.02.001.
-
18.
Khammarnia M, Setoodezadeh F. Electronically prescribing: A new policy in Iranian hospitals. Iran J Pharm Sci. 2017;13(1):1-6.
-
19.
Dewa LH, Murray K, Thibaut B, Ramtale SC, Adam S, Darzi A, et al. Identifying research priorities for patient safety in mental health: an international expert Delphi study. BMJ Open. 2018;8(3). e021361. [PubMed ID: 29502096]. [PubMed Central ID: PMC5855203]. https://doi.org/10.1136/bmjopen-2017-021361.
-
20.
Elmontsri M, Banarsee R, Majeed A. Key priority areas for patient safety improvement strategy in Libya: a protocol for a modified Delphi study. BMJ Open. 2017;7(6). e014770. [PubMed ID: 28674137]. [PubMed Central ID: PMC5734567]. https://doi.org/10.1136/bmjopen-2016-014770.
-
21.
Jebara T, Cunningham S, MacLure K, Pallivalapila A, Awaisu A, Al Hail M, et al. A modified-Delphi study of a framework to support the potential implementation of pharmacist prescribing. Res Social Adm Pharm. 2020;16(6):812-8. [PubMed ID: 31522998]. https://doi.org/10.1016/j.sapharm.2019.09.005.
-
22.
Dixon BE, Zafar A. Inpatient Computerized Provider Order Entry (CPOE)-Findings from the AHRQ Health IT Portfolio. Rockville, MD: Agency for Healthcare Research and Quality; 2009. Report No.: 09-0031-EF. Contract No.: 09-0031-EF.
-
23.
Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14(1):29-40. [PubMed ID: 17068355]. [PubMed Central ID: PMC2215064]. https://doi.org/10.1197/jamia.M2170.
-
24.
Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving medication-related clinical decision support. Am J Health Syst Pharm. 2018;75(4):239-46. [PubMed ID: 29436470]. https://doi.org/10.2146/ajhp160830.
-
25.
Ferrandez O, Urbina O, Grau S, Mateu-de-Antonio J, Marin-Casino M, Portabella J, et al. Computerized pharmacy surveillance and alert system for drug-related problems. J Clin Pharm Ther. 2017;42(2):201-8. [PubMed ID: 28078665]. https://doi.org/10.1111/jcpt.12495.
-
26.
Prgomet M, Li L, Niazkhani Z, Georgiou A, Westbrook JI. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: A systematic review and meta-analysis. J Am Med Inform Assoc. 2017;24(2):413-22. [PubMed ID: 28395016]. [PubMed Central ID: PMC7651905]. https://doi.org/10.1093/jamia/ocw145.
-
27.
Alanazi A. The effect of computerized physician order entry on mortality rates in pediatric and neonatal care setting: Meta-analysis. Inf Med Unlocked. 2020;19:100308. https://doi.org/10.1016/j.imu.2020.100308.
-
28.
Abraham J, Kitsiou S, Meng A, Burton S, Vatani H, Kannampallil T. Effects of CPOE-based medication ordering on outcomes: An overview of systematic reviews. BMJ Quality & Safety. 2020;29(10):1-2. [PubMed ID: 32371457]. https://doi.org/10.1136/bmjqs-2019-010436.
-
29.
Kaboodmehri R, Hasavari F, Adib M, Khaleghdoost Mohammadi T, Kazemnejhad Leili E. Environmental factors contributing to medication errors in intensive care units. J Holist Nurs Midwifery. 2019;29(2):57-64. https://doi.org/10.32598/jhnm.29.2.57.