1. Background
In the last decades, bipolar disorder (BD) treatments have undergone a shift away from relying on clinical expertise to adopting evidence-based practice guidelines (PGs) and expert consensus (1), leading to an improvement in medical care safety, patient performance, and quality of life (2). Nonetheless, suboptimal prescribing has remained an issue, and there is an excellent opportunity to improve the care standards for reducing the recurrence of symptoms, frequent hospitalizations, and costs (3). Assessing the current practices is a vital prerequisite for addressing this problem and implementing planned healthcare changes (4). Proposing a valid and reliable quality measure derived from PGs recommendations as well as adopting effective measures to deal with regional or national conditions prevailed in routine practices are appropriate starting points for finding the variations occurred in practice across settings and geographic areas and improving the quality of mental healthcare. The development of medication assessment tools (MATs) is a possible method for assessing medication therapy in BD (5-7). Medication assessment tools have been applied to evaluate the medication therapy management of many illnesses (8, 9). An essential step toward designing an MAT is to determine its validity and reliability (9).
So far, numerous studies have been conducted to investigate the adherence of psychiatrists to the PGs in different bipolar phases in the inpatient and outpatient settings (1, 10-15), and the adherence rate has been found to vary between 27 and 83% (8). The inconsistency observed in the results from these studies has been due to the adoption of different measurement methods. The failure to evaluate the multiple aspects of care, perform comorbidity evaluation, and use explicit criteria to measure adherence are the methodological weaknesses of these studies.
Only the study by Al-Taweel and Alsuwaidan investigated the psychiatrists’ adherence to PGs in BD using MAT and explicit criteria for the multiple aspects of care as well as considering comorbidities (8). This study, however, suffers from some weaknesses, including a lack of reliability tests and posing too many questions.
To the best of our knowledge, moreover, no study had designed a reliable and valid tool for comprehensive evaluation of the clinicians’ prescribed practices for pharmacotherapy in patients with BD in Iran.
2. Objectives
The present study aimed to develop and validate a MAT for evaluating the prescribers’ adherence to pharmacotherapy recommendations outlined in PGs during the acute phase of bipolar disorder (APBD) in Iran (MATAPBD).
3. Methods
In this study, Iranian psychiatrists and 4th-year postgraduate psychiatric assistants (PGY-4 psychiatry residents) working in the nationally-recognized institutions in Iran were investigated from August 2021 to May 2022. The study was conducted in two phases: (1) item generation by performing literature review and research group discussion; and (2) item reduction to assess the psychometric properties of the developed tool.
3.1. Phase 1: Item Generation
The item generation process was completed by reviewing the literature extensively and holding research group discussion to produce an initial item pool.
The Iranian Ministry of Health and Medical Education attempted to formulate a clinical PG for managing adult patients with bipolar disorder (IGB) as well as pharmacological and instrumental treatments in 2020 (16).
Due to the focus of IGB on the treatment regimens of BD (16), two other guidelines [i.e., Canadian Network for Mood and Anxiety Treatments (CANMAT) guideline (17) and International Society for BD (ISBD) consensus guidelines for the safety monitoring of BD] (18) were also used. Both guidelines were selected based on the opinion of the expert panel members. The CANMAT guideline, one of the four reference guidelines, was also used as a reference for developing the Iranian guideline.
Canadian Network for Mood and Anxiety Treatments guidelines were used to derive recommendations about the factors influencing the selection of treatment regimens, treatment response evaluation intervals, and decision-making approaches concerned with the continuation or discontinuation of a treatment regimen.
International Society for BD consensus guideline for the safety monitoring of BD (18) was used to derive recommendations about physical and laboratory monitoring before and during treatment.
When the data were insufficient or vague in the previous sources, Kaplan and Sadock’s Comprehensive Textbook of Psychiatry (10th edition) (19) as well as Kaplan and Sadock’s Synopsis of Psychiatry Behavioral Sciences/Clinical Psychiatry (11th edition) (20) were used, which are reference books for psychiatrists in Iran. Nine members of the expert panels were contacted individually for an unstructured meeting, and all of them approved the above-mentioned references.
All recommendations of the user PGs were extracted. According to the expert panel’s opinion, then, the recommendations were selected based on factors such as the frequency of bedside use and the possibility of use in Iran.
3.2. Phase 2: Item Reduction
In this phase, the psychometric features of the 54-item MATAPBD were evaluated in terms of face, content and construct validity, and reliability.
3.2.1. Face Validity
Quantitative and qualitative face validity were employed for validating the MATAPBD. To this end, ten participants (six psychiatrists and four PGY-4 psychiatry residents) were asked to assess the face validity of the 54-item tool. To evaluate the qualitative face validity, the items were surveyed for relevancy, difficulty, and ambiguity (21).
To measure quantitative face validity, the same ten participants were asked to assess the items in terms of suitability using a five-point Likert scale. The impact score was calculated for each item, and scores above 1.5 were regarded as acceptable [impact score = frequency (%) × suitability] (22).
3.2.2. Content Validity
An expert panel was employed to evaluate the content validity of the MATAPBD by adopting qualitative and quantitative approaches. Our expert panel consisted of six clinical psychiatrist professionals with over 15 years of work experience in the psychiatry department, three clinical pharmacist professionals, and one professional psychometrician. Our experts bore the professional title of at least, associate professor and were experienced in developing and assessing the psychometric scales. To implement the qualitative approach, the expert panel evaluated the wording and grammar of items along with item scaling and allocating (21). After evaluating each item, space was considered for the experts to provide scientific suggestions and improve the items or make comments. As for the quantitative content validity, the scale was assessed by content validity ratio (CVR) and modified kappa coefficient (K) to ensure that the construct of interest was measured by the scale. The content validity ratio (CVR) was checked by the experts in order to assess items’ essentiality on a three-point Likert scale. When an answer was considered as "not essential" or "useful but not essential" by the experts, they were asked to offer justification for their decisions. The results were interpreted using the Lawshe rule (22). Since there were ten experts involved in the evaluation, the lowest acceptable CVR score was considered equal to 0.62 based on Lawshe (23). To assess K and eliminate the chance of the existence of irrelevant factors affecting each item, ten experts were asked to assess the item relevancy of the scale with 54 items. Modified Kappa was measured for all items, and the excellent value of kappa was considered > 0.75 (24). The results and comments produced adopting each of the qualitative and quantitative approaches to content validity were analyzed and summarized by the research team in order to guide the revision of the instrument. Doubts about comments or suggestions were cleared by experts through face-to-face communication, phone calls, or email communication.
3.2.3. Item Analysis
Before evaluating the construct validity, an item analysis was carried out to identify the possible problems of items by examining the corrected item-total correlation. To this end, 30 psychiatrists were selected using convenience sampling, and were asked to complete the online form of MATAPBD. The correlation coefficients lower than 0.3 or above 0.9 were considered as the criteria for removing the items (24).
3.2.4. Construct Validity
The inclusion criteria in this study were: Iranian psychiatrists or PGY-4 psychiatry residents working in all nationally recognized institutions involved in direct patient care with acute BD and able to use social networks such as Telegram and WhatsApp. The exclusion criteria, on the other hand, were: Iranian psychiatrists or PGY-4 psychiatry residents who were unwilling to participate in this study or failed to thoroughly answer all questions in the questionnaire. The sample size was measured using the rule of thumb that regards 200 subjects as sufficient size for responding to the MATAPBD (25). The participants were selected using convenience sampling method and from among those working in medical schools or the Iranian Psychiatric Association (IPA) across the different provinces. Psychiatrists and psychiatric trainees whose addresses and contact information were available in the medical schools or provincial branches of IPA as well as those who were introduced by colleagues were also included.
The required data were collected using an online questionnaire. To this end, the online questionnaire was first developed using electronic polling (Epoll) survey software, and then its URL link was shared with the participants via email or social networking applications such as Telegram or WhatsApp. Only those questionnaires fully completed by the participants based on the instruction included in the developed questionnaire were reviewed. Our corresponding author was given a username and password for checking the numbers of the completed questionnaires. The questionnaire used for evaluating construct validity consisted of two sections. The first section was intended to elicit information on demographic characteristics. Sociodemographic questions were included in the MATAPBD for collecting information about age and gender, academic position, and years of experience in managing patients with BD. In the second section, the participants were asked to respond to all items of MATAPBD based on a five-point Likert-type scale (i.e., "always," "often," "sometimes," "rarely", and "never") and according to their clinical practice in the management of patients with APBD.
The construct validity of the MATAPBD was assessed by adopting the maximum-likelihood exploratory factor analysis (MLEFA) method and Promax Rotation. The Kaiser–Meyer–Olkin (KMO) and Bartlett’s tests were performed to evaluate the sample’s adequacy, and KMOs with values greater than 0.7 were considered good (26). Horn’s parallel analysis was carried out to calculate the number of latent factors (27). The existence of an item in a latent factor was established based on a factor loading of about 0.35, which was calculated based on the following formula:
3.3. Reliability
Reliability was measured using internal consistency and stability. Cronbach’s alpha (α) and McDonald’s omega (Ω) were used to measure internal consistency. Coefficient’s α and Ω values greater than 0.7 were regarded as acceptable internal consistency (31-33). The stability was assessed by measuring the intraclass correlation coefficients (ICC) of the MATAPBD with a two-way random effects mode. To this end, the test-retest method with a 2-week interval in 21 was employed for investigating the participants (20 psychiatrists and 6 PGY-4 psychiatry residents). It is worth mentioning that an ICC value > 0.8 is an acceptable value of stability (34). Furthermore, the standard error of measurement (SEM) was computed for determining the scale as below, which evaluates the errors of the tool score (22).
3.4. Responsiveness
The responsiveness or sensitivity of a tool shows its ability to detect changes over time (21). In this study, the responsiveness of the tool was assessed by examining the minimum detectable change (MDC) and using the following formula (35, 36):
MDC 30% is acceptable, and that less than 10% is considered excellent (21).
3.5. Multivariate Normality and Outliers
Skewness, kurtosis, and outliers were used to evaluate univariate distributions. Multivariate distributions were evaluated for normality and outliers. Items with a Mahalanobis distance of P < 0.001 were considered multivariate outliers (37). Descriptive statistics (i.e., means and standard deviations for quantitative data, as well as frequencies and percentages for qualitative data) were used to describe the demographic characteristics. All statistical procedures were analyzed by SPSS26 and JASP 0.15.0.0.
4. Results
4.1. Phase 1: Item Generation
The results of the literature review and research group discussion were combined. Three main aspects of the treatment were defined based on the results obtained in this phase as: "Physical and laboratory assessments of patients before treatment initiation", "general pharmacotherapy approaches and evaluation of the treatment response", and "physical and laboratory assessments for patients during treatment". The item pool with 54 items was generated.
4.2. Phase 2: Item Reduction
Four items were found ambiguous to the participants according to the feedback from them and, therefore, were rewritten and modified in the qualitative face validity step. When assessing the quantitative face validity, all items included in this section were assigned an impact score greater than 1.5, which was acceptable. In the quantitative content validity step, the CVRs of 11 items were < 0.62 and, therefore, were removed, and the number of items decreased from 54 to 43. According to the results of the kappa value, the kappa for all items was > 0.75, which was an excellent value. Then, 23 items were merged into six items based on the expert group’s opinion, and the number of items decreased from 43 to 26. Following the item analysis, five items were removed, and 21 items entered the factor analysis step (Figure 1).
Figure 1.
Flow chart of two phases of the study (abbreviations: CVR, content validity ratio; CVI, content validity index; EFA, exploratory factor analysis).
The questionnaire was sent to 400 people meeting the inclusion criteria and was completed by 216 of them. Thus, only 216 completed questionnaires were used for assessing construct validity. The demographic characteristics of the participants responding to the MATAPBD are presented in Table 1.
Table 1.Demographic Characteristics of PATAPBD Respondents
| Variable | Values (N = 216) |
|---|---|
| Gender, No. (%) | |
| Female | 149 (69) |
| Male | 67 (31) |
| Age (y), mean ± SD, (95% CI) | 38.43 ± 6.78 (31.65, 45.21) |
| Academic position, No. (%) | |
| Yes | 53 (31) |
| No | 163 (69) |
| Year of experience in managing patients with bipolar disorder | |
| Mean ± SD (95% CI) | 6.02 ± 5.82 (0.2, 11.84) |
| Range (Min, Max) | 1 (1, 30) |
Taking into account the results of KMO (0.778) and Bartlett’s value of 1954 (P < 0.001), the sample was found adequate and suitable for the construct validity step. In this step, four items were removed since their communality values were less than 0.2, and the factor loadings were less than 0.3. After performing Promax Rotation, four factors, including "laboratory test before treatment initiation" (8 items), "monitoring during treatment" (4 items), "first line regimes" (3 items), and "time interval of patient evaluation" (2 items) were extracted (17 items total) (see Appendix 1 for more information on four factors of the MATAPBP and their factor loadings).
Internal consistency of the scale was indicative of the fact that Cronbach’s alpha and McDonald’s omega of all factors were > 0.7 (Table 2), which confirmed the acceptability of the scale. The overall ICC for MATAPBD was 0.914 (CI 95%: 0.73 - 0.96), which showed strong stability of the scale over time. The value of SEM for the scale was ± 2.76, suggesting that the individuals’ scores for the same scale tended to be distributed a 2.76 value around their "true" score. The MDC % for the tool was calculated to be 7.65%, which is considered excellent.
Table 2.The Indices of Internal Consistency of the MATAPBP (N = 216)
| Factors | Cronbach’s Alpha | McDonald’s Omega |
|---|---|---|
| 1 | 0.899 | 0.816 |
| 2 | 0.816 | 0.821 |
| 3 | 0.734 | 0.796 |
| 4 | 0.879 | 0.923 |
5. Discussion
To the best of our knowledge, this study was the first one to examine the psychometric properties of MAT by using recommendations extracted from national and international PGs outlined for the evaluation of the prescribers’ practices for managing the patients during the acute phase of BD in Iran.
According to our findings, the final MATAPBD had an appropriate level of validity and reliability. Final MATAPBD included 17 items and four factors, namely "laboratory tests before treatment initiation", "monitoring during treatment", "first line regimes", and "time interval of patient evaluation". These factors explained 57.97% of the total extracted variance. Factor extraction is carried out to maximize the explained variance (38). The highest value for the explained variance was recorded for the "laboratory tests before treatment initiation factor" (23.086).
According to the results from Cronbach’s alpha and McDonald’s omega, the MATAPBD revealed strong and excellent internal consistency. In addition, this scale had strong stability with the acceptable value of ICC. The scale’s SEM was estimated, and the smaller value of SEM was significant. SEM measures the accuracy of the score of any participant. Moreover, the evaluation of the responsiveness of the tool produced a desirable result. These assessments are essential and required areas of consensus-based standards for selecting health measurement instruments (COSMIN) (21, 39).
To our knowledge, only one study was already conducted to develop and validate an MAT for evaluating the adherence of psychiatrists to international PGs outlined for bipolar patients in a depressive state. A total of 49 criteria were used in the given study for assessing the final tool (8). Al-Taweel and Alsuwaidan’s tool, similar to ours, examined psychiatrists’ adherence to PG in four areas, but there were also differences between these two studies in terms of the investigated states (8).
First, the tool developed by Al-Taweel and Alsuwaidan examined the adherence to PG in the depressive state of BD, while our research investigated the adherence to PG in the acute phase of BD, both in the manic and depressive state (8).
Second, construct validity was performed in our study using EFA, leading to the reduction of items, while this validity was not performed in Al-Taweel and Alsuwaidan study (8). The number of items in the questionnaire plays a critical role in questionnaire-based studies since it directly affects the time a respondent needs to complete a questionnaire, the response rate, and the quality of the obtained data (40).
Third, reliability was assessed in our study, but it was not measured in Al-Taweel and Alsuwaidan study (8). Reliability is used to evaluate the stability of the measures administered at different times to the same individuals and the equivalence of the sets of items from the same test. The higher the reliability, the more accurate the results, which increases the chance of making the correct decision in research (41).
The first and second extracted factors were labeled "laboratory test before treatment initiation" and "monitoring during treatment", which comprised eight and four items, respectively. Measuring fasting blood sugar before starting atypical antipsychotics accounted for the most factor load of these 12 items. Diabetes mellitus (DM) is a significant risk factor for cardiovascular disease, and patients with BD are two to three times at greater risk of DM developing when compared to healthy controls of the same age and gender (35). Furthermore, individuals taking either first-generation or second-generation antipsychotics face a greater risk of developing diabetes than the general population. Diabetes progresses rapidly after initiation of the treatment, suggesting that antipsychotics may be implicated. Therefore, it is recommended that the patients with diabetes should be screened when the treatment is initiated or when the treatment is changed (36).
The third factor was "first-line regimes", with three items evaluating the choice of first-line treatment regimens by psychiatrists in APBD. When choosing the treatment regimen, factors such as the patient’s previous and current medications, side effects, and patient’s clinical features should be considered (17).
The last factor was labeled as "time interval of patient evaluation" with two items. This factor was associated with the intervals considered when evaluating the bipolar patients in the acute phase and the treatment approach in these patients. Although the duration of an adequate medication trial for patients with bipolar major depression is usually between six and eight weeks, it is not feasible to apply this duration to hospitalized patients. The minimal improvement (reduction of baseline symptoms < 20 percent) within the first few weeks of treatment with a specific medication determines the probability of the drug’s effectiveness in the future. Therefore, it is necessary to evaluate the patients regularly during the first weeks (42-45).
This pioneering study may have formed the scientific base for further studies aiming at evaluating individual patients, providing feedback to clinicians, identifying barriers, and constantly monitoring the quality improvement in the pharmacotherapy of patients in the acute phase of BD. Moreover, the MATAPBD may have been employed as a valuable tool to examine the association between treatment recommendation conformance to patient outcomes and cost impact in future studies.
The present study faced some limitations. First, since this tool was validated by psychiatrists working in Iran, our study results may not have been generalizable to other countries. Second, an online questionnaire was used in our study to collect the data, which was another area of improvement. Although online survey has some benefits, the lack of face-to-face communication, the inability to verify the participant’s status, and the accuracy of their answers are limitations of online surveys.
5.1. Conclusions
The tool developed in this study consisted of four factors with 17 items that were found to have favorable validity and reliability to evaluate prescribers’ adherence to PGs for managing patients with APBD. MATAPBD was determined a novel assessment tool for measuring the quality of BD patients’ management. This tool was revealed to help psychiatrists ensure the compliance of their evaluation and treatment procedures with the most recent evidence-based recommendations. Moreover, it may have been used by healthcare professionals for performing peer review or undertaking organized efforts aimed at improving the evaluation and treatment of patients with BD.
