Application of Electrosurgical Units by Operating Room Personnel: Development and Psychometric Testing of an Instrument

1. Background

Electrosurgical units (ESUs) are among the most frequently-used devices in surgeries since they conduct electricity through a surgical tool to the patient’s body (1-4). By creating hemostasis during surgery, ESUs enable the surgical team to make a surgical incision without causing bleeding and, thus, have a better view of the area under operation (1). On the other hand, incorrect use of ESUs can have many serious consequences for both the patient and surgical team, among them burning the patient’s skin, starting a fire in the operating room, inhaling the smokes from ESU, and disrupting cardiac pacemakers. Inhalation of the smokes made by ESUs can cause gene mutation in the personnel in the long run (5).

The American College of Surgeons reported that 54% of every 506 surgeons suffer from side effects of ESUs (6). According to the statistics released by the Association of perioperative registered nurses in 1999, every year 40,000 patients suffer from skin burns due to inappropriate use of ESUs. Moreover, the treatment of such burns costs nearly US$600 million, which imposes a huge financial burden on hospitals (7). In 2009, the American Emergency Care Research Institute reported 550 to 650 cases of fire in operating rooms, 68% of which were caused by ESUs (8). In recent years, patient safety in the operating room is a cause for concern (9). The Association of perioperative registered nurses suggests that instruments designed to evaluate the performance of surgical teams can result in their better adherence to instructions to apply ESUs (10).

In Iran, electrosurgical unit is called cautery or electro-cautery. There is not a specialized checklist for the use of ESUs to assess the performance of the personnel with regard to their application of such units. The medical equipment workshops for operating room personnel include a yearly workshop on ESUs. However, the workshops do not address all the specific details about electrosurgery and only contain summary presentations about the settings of ESUs. The personnel are not educated about such issues as the side effects of ESUs and their occupational health; e.g., the hazards of inhaling electrosurgical smoke. On many occasions in the present study, the research team witnessed incorrect practices in the personnel’s use of ESUs; e.g., failure to shave the area where the plate was to be attached, using a scalpel to clean the tip of the electrosurgical pencil, and failure to check the settings of the unit before application.

The manual for the application of ESUs was published by the American Association of Surgical Technologists and the Association of perioperative registered nurses in 2012. Since there is a large number of instructions in the manual, many of them are likely to be forgotten in practice. Also, the administrative instructions are presented as routine recommendations and there is no guarantee that they are observed by the operating room personnel. Evaluation of the performance of the personnel and stressing the importance of correct application of ESUs can play a major role in efficient use and extending the life of these devices, reducing occupational hazards for the surgical team, and enhancing the safety of the patient. Employment of psychometric instruments to evaluate the personnel’s practice can facilitate and promote adherence to the instructions.

2. Objectives

The present study aimed at developing and measuring the psychometric properties of an instrument to evaluate the application of ESUs by operating room personnel.

3. Methods

3.1. Design and Sample

The current methodological study was conducted in two stages: first, the various aspects of the principles of ESUs application were established based on the results of a review of literature (articles, theses, and books) and data available on the websites of related institutes including the Association of Surgical Technologists, Association of Perioperative Registered Nurses, and American National Standard Institute. Also, a thorough search was conducted in the databases of PubMed, Science Direct, Google Scholar, Scopus, Medline, Elsevier, CINAHL, ProQuest, and Thomson Reuters with the keywords related to the subject in question. Subsequently, to develop the items of the checklist, the researchers held several meetings with nursing, operating room, instrument development, and medical equipment experts.

In the second stage, the validity and reliability of the instrument were measured. Validity was measured according to the face validity, content validity, and construct validity of the checklist; the reliability of the checklist was measured according to its internal consistency (the Kuder-Richardson coefficient) and inter-rater or observer reliability. The face validity of the checklist was evaluated both qualitatively and quantitatively: for the qualitative evaluation of face validity, 20 operating room nurses and medical equipment experts were interviewed face-to-face and the relevance of the items to the subject in question and ambiguities in the meaning of the terms or statements were examined. The quantitative method of item impact score was used to eliminate inappropriate items: accordingly, 20 experts were asked to score each item on the checklist based on a five-point Likert scale. The preliminary version of the checklist came to consist of 32 items answered on a yes-no basis: 18 items addressed the preoperative, 11 items the operative, and three items the postoperative performance of the personnel.

The content validity of the checklist was also evaluated both qualitatively and quantitatively. In the qualitative stage, 20 experts were interviewed and asked to assess the items in terms of composition and significance. The two measures of content validity ratio (CVR) and content validity index (CVI) were used to evaluate content validity, quantitatively. Subsequently, the average CVI (S-CVI/Ave) of the checklist was calculated.

The construct validity of the checklist was determined using the known group comparison technique. In this approach, to measure the capacity of an instrument to discriminate between groups, the researcher uses groups, which are expected to be different in a certain characteristic (11). To apply the known group comparison technique, the researchers observed the performance of 20 members of surgical teams in 20 surgeries. Initially, 20 members of the surgical teams of the largest hospitals affiliated to a university of medical sciences in the South of Iran were randomly divided into a control and an intervention group (10 individuals in each group) based on the simple sampling method (selected from a random number table). The manner of application of ESUs by both groups in 20 orthopedic and cerebral surgeries was observed and recorded in the checklist by one of the researchers. In the next stage, the 10 individuals in the intervention group were given face-to-face education about the principles of correct application of ESUs. The control group did not receive any education; 14 days after the intervention, the performance of the personnel in both groups in 20 similar surgeries was observed and recorded in the checklist again. To avoid bias, the same researcher educated all the personnel in the intervention group. Also, to prevent the exchange of information between the intervention and control groups, the subjects were selected from two different hospitals. Both the pretest and posttest observations were executed by the same researcher. In the statistical analysis step of the current study, items 5, 6, 9, 10, 12, 16, and 25 were recoded. Finally, scores of the two groups for their performance in the application of ESUs were compared using the statistical tests of the Mann-Whitney and the Wilcoxon.

To determine the reliability of the checklist, the researchers employed methods to measure internal consistency and inter-rater reliability. The internal consistency of the instrument was calculated using the Kuder-Richardson coefficient. Accordingly, the performance of 40 members of surgical teams with regard to their application of ESUs in 40 surgeries (20 cerebral and 20 orthopedic surgeries) was observed and recorded in the designed checklist by one of the researchers. In this method, the test or instrument is said to be reliable only when its Kuder-Richardson reliability coefficient is at least 0.64 (12). To evaluate inter-rater reliability, the checklist was completed by one of the researchers and a trained co-researcher in their simultaneous observations of 10 real surgeries. Subsequently, the data recorded in the checklists were compared in terms of kappa inter-rater agreement. The minimum acceptable value for kappa coefficient is 0.6, and values above 0.8 indicate satisfactory inter-observer or inter-rater agreement (12). In the present study, P < 0.05 was considered the level of significance.

3.2. Statistical Analysis

The Mann-Whitney nonparametric test was employed to compare the intervention and control groups. Also, the Wilcoxon nonparametric test was employed to compare the mean of two dependent groups in the pre- and post-intervention stages. The collected data were analyzed using SPSS version 20.

4. Results

In the initial stage of the study, data collected from a review of literature (books, theses, and articles) and search in related websites and databases (10, 13-18) were employed to develop items to evaluate the application of ESUs by operating room personnel.

The preliminary version of the checklist consisted of 85 items. In the qualitative evaluation of the face validity of the checklist, the research team revised 20 items and added five more items to the checklist. In this stage, the number of items was reduced to 37. In the quantitative evaluation of face validity, scores obtained for all the items were above 1.5. In the qualitative evaluation of the content validity of the checklist, based on the views of the consulted experts and the research team, items 9 and 28, 15 and 18, 21 and 22, and 35 and 37 were merged due to overlap; items 7 and 32 were eliminated; items 27, 30, and 31 were revised; and item 19 was rewritten as two separate items. Eventually, the checklist consisted of 32 items. The results of the quantitative evaluation of content validity showed that the CVR of all the 32 items were above 0.42. As prescribed by Lawshe table, items with CVRs of 0.42 or above are retained (19). The S-CVI/Ave of the checklist was 0.97. Table 1 shows the results of the evaluation of the checklist in terms of CVI.

The pretest and posttest scores of the control and intervention groups as obtained by the checklist to evaluate the performance of the subjects were compared using the statistical tests of Mann-Whitney and Wilcoxon. The results of the evaluation of construct validity based on the Mann-Whitney test showed that the mean score of the subjects in the intervention group was significantly higher than that of the ones in the control group (P value = 0.001). Moreover, the results of the Wilcoxon test showed that the posttest performance scores of the subjects in the intervention group were significantly higher than their pretest scores (P value = 0.005). The difference between the results of the control and intervention groups proves the construct validity of the checklist (Table 2).

The internal consistency (the Kuder-Richardson coefficient) of the checklist was 0.66. The inter-rater reliability of the checklist was calculated using the kappa percent agreement. The minimum percentage of agreement among the experts was 0.6, which was the minimum acceptable value, and the percentage of agreement of 22 of the 32 items was above 0.8, indicating satisfactory reliability. Table 3 shows the kappa coefficient values for agreement between the two raters.

5. Discussion

ESUs are among the most frequently-used and helpful pieces of equipment in surgeries. These units can play a key role in a successful surgery (20); however, incorrect application of ESUs can have many serious consequences for both the patient and the surgical team (5). It is essential that operating room personnel provide care and employ medical equipment based on established codes (21). Thus, employment of instruments to evaluate the performance of personnel with regard to their application of ESUs can encourage their adherence to instructions in manuals, which in turn reduces the side effects and occupational hazards associated with incorrect use of ESUs, and guarantees the safety of the surgical team and the patient (10).

In the checklist developed in the present study, items 23 and 24 stressed the significance of the use of high filtration masks and extractors to remove electrosurgical smoke. According to many studies, the smokes created by vaporization of tissue contain toxic chemicals, which are extremely harmful to both the patient and the surgeon. Smokes caused by vaporization of tissue of patients with a chronic viral disease, such as hepatitis and AIDS, even contain traces of the virus (22, 23). The carcinogenic properties of these smokes, which mostly contain benzene are proven. Since the smokes caused by vaporization of tissue are both infectious and carcinogenic, they need to be effectively extracted from the environment of surgery before they are inhaled (24). Therefore, the utilization of smoke extraction facilities and high filtration masks in the operating room are highly recommended (25, 26).

Items 11, 13, 14, 15, 16, 17, 18, 19, 28, and 32 are related to application instructions meant to minimize the risk of burning the patient’s skin; the most serious side effect of ESUs. In a case study in Pakistan, Saaiq et al. (5), reported three burn cases caused by incorrect application of ESUs; they emphasized that the surgical team should make sure that the patient is not wearing any jewelry or other metallic objects and check the contact of the return electrode with the patient’s skin before surgery and examine the condition of the patient’s skin both before and after operation.

Item 12 is about avoiding the employment of monopolar electrodes for patients with pacemakers. The interference of monopolar ESUs with the functioning of cardiac pacemakers leads to cardiac arrhythmia (27). Such an event can change a sinus ventricular rhythm to ventricular fibrillation or ventricular tachycardia (28).

Items 1, 2, 3, 4, 5, 6, 7, and 8 are related to application instructions intended to extend the life of ESUs and reduce the risk of electric shock, which in turn reduce the financial burden of fixing or replacing ESUs on hospitals. The content of the checklist developed in the present study falls into four categories: operating room safety and environmental factors, application and maintenance of the electrosurgical unit, safety of the patient and personnel, and troubleshooting in the course of application; eight items were related to the safety of the personnel and their occupational health; 13 items were related to patient safety; four items were about extending the life of the unit, and seven items addressed the safety of the patient and the personnel together. The items on the checklist are to be answered in three stages: 18 items before operation, 11 items in the course of operation, and three items after operation. Answers to the items are on a yes-no basis. In terms of scoring, each item is worth 1 point. The total score ranges 0 (minimum) to 32 (maximum); higher scores indicate the better performance of personnel in the application of ESUs. A score of 1 means that the practice of personnel with regard to the aspect in the item is correct; otherwise, a score of 0 is assigned. The sum of the scores shows the overall performance of the operating room personnel in the use of ESUs. Scores 0 - 10 indicate poor performance, 11 - 22 average performance, and 23 - 32 satisfactory performance.

The present study was an innovation in Iran and worldwide in the development and evaluation of psychometric properties of an instrument to assess the application of ESUs by operating room personnel. The CVI of all the items was above 0.79, which confirmed the simplicity, clarity, and relevance of the items. The S-CVI/Ave of the checklist with 32 items was 0.97; according to experts, the minimum acceptable score is 0.8. A significant difference between the results for the two groups (control and intervention) proved that the checklist possessed acceptable construct validity. The Kuder-Richardson reliability coefficient of the checklist was above the minimum acceptable value of 0.64, which demonstrated the satisfactory internal consistency of the instrument. The results of kappa coefficient calculation proved the satisfactory reliability of the checklist and agreement between the observers. The present instrument can help ensure safe application of ESUs.

5.1. Conclusions

The results of the present study demonstrated that the developed instrument was sufficiently valid and reliable to evaluate the application of ESUs by operating room personnel. Also, the present checklist can be easily employed by both external evaluators, on a self-administered basis. The checklist addresses the various aspects of the application of ESUs and can be helpful in identifying errors by operating room personnel. Identification of problems can guide policy-makers and administrators to plan more fruitful workshops. The present instrument can also be employed by researchers in future studies in this area.