1. Background
Schizophrenia is a complicated, persistent mental health illness, inflicting several self-care, occupational, and cognitive damages with the expected prevalence of 4.6 - 7.2 per 1000 general population (1). It is the fifth leading cause of disability in the world (2). According to the Global Burden of Disease Study 2017 (GBD 2017), 19.78 million patients were suffering schizophrenia, which represents a high global burden, i.e., 0.51% of the all-cause disability-adjusted life years (DALYs) were due to schizophrenia (3).
People who suffer from schizophrenia experience increased rates of morbidity from general medical conditions and a higher probability of premature death than the general population (3-5). Antipsychotic medications are the cornerstone of evidence-based treatment to reduce symptoms and improve the condition of schizophrenic patients (6, 7). Treatment of schizophrenia can be extremely difficult since patients vary significantly in terms of the intensity, type, and frequency of symptoms (8, 9). Therefore, it is necessary to optimize the care of people with schizophrenia to reduce the significant economic and human costs of these disorders (10, 11).
Recent studies show that the quality of care for mental disorders is suboptimal (4, 12), particularly in low-and-middle-income countries (LMICs) (13, 14). It is estimated that approximately 70% of people with schizophrenia do not have access to effective mental health care in these countries (14, 15). These studies reveal that there is a persistent quality gap between the evidence-based practice standards and the care provided in these countries (4, 13, 14). They point to the lack of standardized infrastructure, measures, and policies to consistently measure the quality of care as the most important reasons for this persistent gap in mental health care quality (4, 12).
The assessment of the clinical quality of care requires an intense document base which can act as a measurement principle to evaluate the treatment processes (13, 16, 17). Practice guidelines (PGs) are evidence-based recommendations for treating the patients with specific problems (18). Medication assessment tools (MATs) which are, according to standards, suggested by practice guidelines, have been applied in the evaluation of the medication therapy management of numerous illnesses (19) and considered as clinical audit tools to compare routine care of patients with PGs recommendations (10, 17, 18, 20, 21). Providing a valid and reliable quality measure derived from PGs recommendations, and adopting the measures to regional or national conditions for application in routine practices is a critical starting point in detecting the variations in practice across settings and geographic areas and improving the quality of mental health care (4, 10, 17). As far as we know, studies that have assessed psychiatrists’ adherence to PGs to manage the schizophrenia fall into two categories (22-27). First are studies that used medical record data to evaluate guideline adherence (24, 26). Documentation in medical records was assumed to accurately reflect clinical practice in these studies. However, this is an accepted assumption, differences in the results of these studies could reflect real variation in adherence, as well as differences in the adequacy of medical documentation by physicians, especially in developing countries, that medical records aren’t often kept completely and may not reflect reality (17, 21, 26). In the second group of studies, in addition to using medical records, interviews were also conducted with the patients to collect data (22, 23, 25, 27). The lack of using a valid and reliable tool for assessing the clinicians’ adherence rate and the lack of assessment of care’s multiple aspects are the weaknesses of these studies. Further, data from some studies of the first and second groups were collected when many atypical antipsychotic drugs had not yet been introduced (25-27), which limits the ability of their results to evaluate whether atypical agents were prescribed based on PGs.
According to the most recent population-based survey in Iran, the prevalence of mental diseases is 29.7%, (28) and roughly 3% of health care spending is allocated to mental health (29). In 2001, the prevalence of schizophrenia and emotional disorders in Iran was 17% (30). In Iran, as in many other developing countries, there is a lack of appropriately designed national guidelines to help healthcare providers in their care of patients in psychiatry; that causes healthcare workers to use different types of international guidelines in the treatment of their patients (17, 31).
To the best of our knowledge, no study has so far designed a reliable and valid tool to evaluate the clinicians’ prescribed practices for pharmacotherapy of schizophrenia in Iran. This tool can be used as a self-assessment tool by psychiatrists as well as for peer review or for organized efforts by health care professionals to improve the care of patients with schizophrenia.
2. Objectives
This study aims to develop and validate a MAT to evaluate prescribers’ adherence to pharmacotherapy recommendations in PGs during the acute phase of schizophrenia (APSCZ) in Iran (MATAPSCZ).
3. Methods
This cross-sectional study was conducted among Iranian psychiatrists and psychiatric trainees studying the final year of residency [final-year or postgraduate year 4 (PGY-4) psychiatry residents] in all the nationally recognized institutions from August 2021 to April 2022. In this study, the term participants refer to both psychiatrists and psychiatric trainees.
The research was divided into two phases: (1) item generation and (2) item reduction.
3.1. Phase 1: Item Generation
The item generation process was developed through an extensive review of literature and research group discussion to produce an initial item pool.
Due to the lack of a national PG for the treatment of schizophrenia in Iran, a variety of recognized psychiatric references are used to treat patients with schizophrenia. Therefore, some exploratory work was done prior to the initiation of the study to determine which PGs could be used in this study to measure psychiatrists’ adherence to pharmacotherapy recommendations in the management of patients during the acute phase of schizophrenia. According to the literature review, high-quality, and up-to-date English PGs for the treatment of schizophrenia were found (32-34). Nine of members of the expert panels were contacted individually for an unstructured meeting and all agreed that among these PGs, the American Psychiatric Association (APA) Practice Guideline third edition (35), and the psychosis and schizophrenia in adults, the National Institute for Health and Care Excellence (NICE) Guideline on Treatment and Management edition 2014 (36) are the most followed guidelines in Iran. If the data were insufficient or vague in the previous sources, Kaplan and Sadock’s Comprehensive Textbook of Psychiatry tenth edition (37), and Kaplan and Sadock’s Synopsis of Psychiatry Behavioral Sciences/Clinical Psychiatry eleventh edition (38) were used, which are reference books for psychiatrists while studying in Iran. In addition, Applied Therapeutics the Clinical Use of Drugs eleventh edition (39) textbook, and the Maudsley Prescribing Guidelines in Psychiatry thirteenth edition (40) were utilized. The expert panels also agreed with all the mentioned references. Following the literature review, an initial item pool was created. The research team checked and modified items based on their grammar, writing, overlap, and similarity at frequent meetings. Next, the MATAPSCZ was prepared for psychometric evaluation.
3.2. Phase 2: Item Reduction
At this phase, the psychometric features of the 91-item MATAPSCZ were evaluated regarding face, content and construct validity, and reliability.
3.2.1. Face Validity
Quantitative and qualitative face validity were employed for the MATAPSCZ. Ten participants (six psychiatrists and four PGY-4 psychiatry residents) assessed the face validity of the 91-item tool. To conduct qualitative face validity, items were surveyed with respect to their relevancy, difficulty, and ambiguity (41).
Quantitative face validity was assessed by asking the same ten participants to evaluate items on a five-point Likert scale for their suitability. Each item’s impact score was calculated, and it was considered acceptable if the score was above 1.5 (impact score = frequency (%) × suitability) (41).
3.2.2. Content Validity
Qualitative and quantitative evaluations of the content validity of the MATAPSCZ were conducted by an expert panel. Our expert panel consisted of six clinical psychiatrist professionals with over 15 years of work experience in the psychiatry department, three clinical pharmacist professionals, and one professional psychometrician. At least the professional title of our experts was associate professor and they all had experience in the development and psychometric assessment of a scale. In the qualitative approach, the expert panel evaluated wording and grammar of the items along with item scaling and allocating (42). Also, after each item, a space was considered for the experts to provide scientific suggestions to improve the items or make comments. The Content validity ratio (CVR) and modified kappa coefficient (K) were evaluated in quantitative content validity (41). The experts checked content validity ratio (CVR) to assess items’ essentiality in a three-point Likert scale (41). If experts consider an answer as "not essential," or "useful but not essential", they must justify their decisions. The result was interpreted using the Lawshe rule. As experts were ten persons, the lowest acceptable CVR score was equal to 0.62 according to Lawshe (43).
The item relevancy for the K assessment was evaluated by ten experts to eliminate the effect of chance for each item. To assess K to eliminate the effect of chance for each item. Modified kappa was calculated for every item. Kappa values greater than 0.75 were considered excellent (41).
The results and comments of each of the qualitative and quantitative approaches of content validity were analyzed and summarized by the research team in order to guide the revision of the tool. Doubts about comments or suggestions were cleared with experts face-to-face, by email, or by phone.
3.2.3. Item Analysis
An item analysis was performed before determining the construct validity, in order to detect the probable problems with items based on the corrected item-total correlation. Using convenience sampling, 30 psychiatrists were chosen and fulfilled the MATAPSCZ online form. The correlation coefficient between items less than 0.32 or higher than 0.9 was considered to exclude items (41).
3.2.4. Construct Validity
The inclusion criteria were: Iranian psychiatrists or PGY-4 psychiatry residents in all the nationally recognized institutions who were involved in direct patient care with acute schizophrenia and able to use email or social networks like WhatsApp or Telegram. The exclusion criteria were Iranian psychiatrists or PGY-4 psychiatry residents who didn’t completely answer all questions of the questionnaire.
The sample size was measured using the rule of thumb that regards 200 participants were considered sufficient for answering the MATAPSCZ (44). The participants were obtained using convenience sampling through medical schools or the Iranian Psychiatric Association (IPA) across the different provinces. Psychiatrists and psychiatric trainees whose addresses and contact information were available in the medical schools or provincial branches or who were introduced through colleagues were included.
The data were collected through an online questionnaire. Electronic polling (Epoll) survey software was used in creating the online questionnaire, and the participants received its URL link via email or social networks, including Telegram or WhatsApp. Only the questionnaires that were filled until the end stage were considered completed. It was possible for the corresponding author to check the number of completed questionnaires by entering a username and password.
The questionnaire used for construct validity had two parts. Participants’ demographic characteristics were the subject of the first part. Sociodemographic questions were included in the MATAPSCZ for collecting information about age and gender, academic position and year of experience in managing patients with schizophrenia. In the second part, participants were asked to respond to every item of MATAPSCZ on a five-point Likert-type scale (“always,” “often,” “sometimes,” “rarely,” and “never”) according to their clinical practice in the management of patients with APSCZ. A maximum-likelihood exploratory factor analysis (MLEFA) method and a varimax rotation were used to evaluate the construct validity of the MATAPSCZ. In order to evaluate the sample adequacy, Kaiser-Meyer-Olkin (KMO) and Bartlett’s tests were conducted. Kaiser-Meyer-Olkin with values greater than 0.7 were considered good (45). Horn’s parallel analysis was used to identify the number of latent factors (41). A factor loading of 0.3 was used for determining whether items exist in latent factors, as calculated by the following formula:
3.2.5. Reliability
Reliability was evaluated using internal consistency and stability. Internal consistency was assessed by Cronbach’s alpha (α), McDonald’s omega (Ω), and the average inter-item correlation (AIC). Coefficient’s α and Ω values greater than 0.7 and AIC between 0.2 - 0.4 were regarded as acceptable internal consistency (41). Evaluation of the intraclass correlation coefficients (ICC) of the MATAPSCZ with a two-way random effects model was used to measure the stability. To this aim, the test-retest method was used on 26 participants (20 psychiatrists and 6 PGY-4 psychiatry residents) with a time interval of 2 weeks. An ICC value more than 0.8 is an acceptable value of stability (41). Furthermore, the standard error of measurement (SEM) was computed for the scale (SEM = SD Pooled × √1 – ICC), which evaluates the errors of the tool score (46).
3.2.6. Responsiveness
The responsiveness or sensitivity of a tool shows its ability to detect changes over time (42). The following formula was used to calculate the minimum detectable change (MDC) to assess responsiveness in this study (41, 42):
MDC 30% is acceptable, and less than 10% is considered as excellent (42).
3.2.7. Multivariate Normality and Outliers
Skewness, kurtosis, and Outliers were used to evaluate univariate distributions. Multivariate normality was also assessed (46). Mahalanobis distance evaluation was used to assess the Multivariate outliers. Items with a Mahalanobis distance of P < 0.001 were considered to be multivariate outliers (41). Demographic characteristics were described using descriptive statistics (means and standard deviations for quantitative data, frequencies, and percentages for qualitative data). We used SPSS26 and JASP 0.15.0.0 for statistical analysis.
3.3. Ethical Considerations
This study was approved by the Research Ethics Committee of Mazandaran University of Medical Sciences (IR.MAZUMS.REC.1400.373) (Approval Date: 2021-07-28). In all stages, participants were informed of the purpose of the study. Moreover, participants gave oral informed consent before filling out the face and content validity. Participation or non-participation of the residents of psychiatry in the study did not affect the approval score of their residency period. The participants could leave the study whenever they wanted, and they were assured of anonymity and data confidentiality.
In addition, in the online survey scale, all information included in the consent form was provided in the initial part of the questionnaire. The research’s purpose, the number of items, the affiliation of researcher, email for questions, and study’s ethical code were included as well. Further, participants were informed about their voluntary participation and anonymity of their responses in the form of group data. The online survey items could only be viewed by participants who accepted to take part and clicked the “next button”.
4. Results
4.1. Phase 1: Item Generation
Following the review of literature and research group discussion, in order to compare care provided to schizophrenic patients with guideline-recommended care, we defined three main aspects of pharmacotherapy: “Physical and laboratory assessments for patients before drug treatment initiation,” “General principles of drug treatment,” and “Physical and laboratory assessments for patients during drug treatment.” There were 100 items in the initial item pool. Of these, 91 items were included as items of the MATAPSCZ.
4.2. Phase 2: Item Reduction
4.2.1. Face Validity, Content Validity, and Item Analysis
In the qualitative face validity step, according to the participant’s feedback, ten items were ambiguous to the participants, which were rewritten and modified. In the quantitative face validity, all items obtained an impact score higher than 1.5, which was acceptable. During the assessment of content validity, according to the expert panel suggestion in the qualitative approach, eleven items were merged, and nine items were rewritten; so, the items were reduced to 80 items. In the quantitative content validity step, we removed 16 items with CVR less than 0.62. The total number of MATAPSCZ was reduced from 80 to 64 items. Then the experts had suggestions for editing the order of the items, which were applied after a decision by the research team. According to the results of the kappa value, the kappa for all items was is > 0.75 which was an excellent value. During the item analysis step, six items had a correlation coefficient of less than 0.32 and were removed, and factor analysis was conducted on the MATAPSCZ with 58 items (Figure 1).
Figure 1.
Flow chart of two phases of the study (CVR, content validity ratio; CVI, content validity index; EFA, exploratory factor analysis).
4.2.2. Construct Validity
Out of the 320 distributed MATAPSCZ survey, 200 were collected back and checked for completion and validity. Of these, 17% were PGY-4 psychiatry residents (see Appendix 1 in supplementary file, which shows the demographic characteristics of the participants who responded to the MATAPSCZ).
The sample was adequate for the construct validity step according to the results of KMO (0.825) and Bartlett’s value 6873.456 (P < 0.001). In this step, 10 items were eliminated since their communality values, and factor loadings were below 0.2 and 0.3, respectively, and after varimax rotation, six factors were extracted (48 items) such as “physical and laboratory assessments for a patient before starting an antipsychotic drug” (15 items), “general pharmacotherapy approaches and evaluation of the treatment response” (12 items), “general principles for selection of an antipsychotic drug” (6 items), “physical and laboratory assessments for a patient taking an antipsychotic drug” (8 items) “indications for the administration of an injectable antipsychotic drug” (4 items) and “Indications for the prescription of an antipsychotic drug in a lower dose” (3items) were extracted. These six factors explained 50.1% of the total variance of prescribers’ adherence to evidence-based PGs in the management of patients with APSCZ (see Appendix 2 in supplementary file, which shows the details of factor analysis results of the six factors of the MATAPSCZ).
4.2.3. Reliability and Responsiveness
Internal consistency showed that Cronbach’s alpha, and McDonald’s omega of all factors were higher than 0.7 and AIC was 0.2 to 0.4 for all factors. Therefore, the internal consistency of the scale was acceptable (see Appendix 3 in Supplementary file, which shows. The indices of internal consistency of the MATAPSCZ). The overall ICC for MATAPSCZ was 0.914 (CI 95: 0.792 to 0.965), that showed strong stability of the scale over time. For the scale, the SEM was ± 4.5, indicating that the participants’ scores tend to have a 4.5 value distribution around their "true" score. The MDC% for the tool was calculated to be 6.15%, which indicates its excellent responsiveness.
5. Discussion
To the best of our knowledge, this is the first effort to develop and evaluate the psychometric properties of MAT using pharmacotherapy recommendations extracted from international PGs to assess the prescribing practices in managing patients during the acute phase of schizophrenia. Our findings showed that the final MATAPSCZ had an acceptable level of validity and reliability, included 48 items and six factors, namely “physical and laboratory assessments for a patient before starting an antipsychotic drug,” “general pharmacotherapy approaches and evaluation of the treatment response,” “general principles for selection of an antipsychotic drug,” “physical and laboratory assessments for a patient taking an antipsychotic drug,” “indications for the administration of an injectable antipsychotic drug” and “indications for the prescription of an antipsychotic drug in a lower dose” which explained 50.1% of the total extracted variance.
Factor extraction aims to maximize the explained variance (46). The highest values for the explained variance were related to “physical and laboratory assessments for a patient before starting an antipsychotic drug” (15.4%) and “general pharmacotherapy approaches and evaluation of the treatment response” factors (10.2%).
Based on the findings of Cronbach’s alpha, AIC, and McDonald’s omega, the MATAPSCZ revealed excellent internal consistency. Besides, this scale has strong stability with the acceptable value of ICC. The scale’s SEM was estimated. It is extremely important for SEM to be smaller. In fact, SEM measures the score’s accuracy of any participant. Moreover, the evaluation of the responsiveness of the tool showed a desirable result. These assessments are an essential area of consensus-based standards for the selection of health measurement instruments (COSMIN) (42) which were not reported in previous studies evaluating the adherence of psychiatrists to PGs for the management of schizophrenia.
The first and fourth extracted factors were labelled “physical and laboratory assessments for a patient before starting an antipsychotic drug” and “physical and laboratory assessments for a patient taking an antipsychotic drug,” which comprised 15 items and 8 items, respectively.
Side effects are of clinical importance because they are associated with reduced quality of life and treatment adherence which may lead to the recurrence of the underlying psychiatric disorder (47). Therefore, medical and laboratory evaluations are important in ensuring the safe initiation and monitoring of suitable treatments (38). PGs have no definite standard for the frequency of monitoring, thus, judgments regarding monitoring individuals for side effects, physical conditions, or abnormities in laboratory tests are essentially related to the clinical circumstances (35, 48).
The second and sixth extracted factors were “general pharmacotherapy approaches and evaluation of the treatment response” and “indications for the prescription of an antipsychotic drug in a lower dose” with 12 items and 3 items, respectively.
The main goal of the management of APSCZ with antipsychotic medication is to decrease acute symptoms to bring the individuals back to their baseline function levels (35). Evidence-based standards for schizophrenia have suggested antipsychotic dosage ranges, including the initial, typical, and maximum daily doses (35, 37, 39), which should be individualized depending on the illness stage (e.g., first episode vs. later stages) (33, 35), age (33, 35), mental condition (35, 49), concomitant physical health issues, receiving multiple medications (35), somatic condition with special attention to side effects (35, 50), need for urgent treatment (51), and the medication formulation (35). Determining the ideal dose of antipsychotic during acute treatment is both difficult and important because there is generally a delay between the start of treatment and complete therapeutic response (35, 37). The existing evidence proposes that patients who have not shown minimal improvement (at least a 20% reduction) in symptoms by around two weeks on a therapeutic dose are less likely to exhibit much improvement (at least a 50% reduction) in symptoms at 4 - 6 weeks (35, 52). As a result, checking the patient’s clinical status for 2 - 4 weeks is reasonable on a therapeutic dose unless the patient exhibits unpleasant side effects (35, 37).
According to PGs, clozapine is recommended for patients suffering from treatment-resistant schizophrenia (TRS), but there is a substantial difference in definitions provided for TRS in practice and clinical trials (35, 53). A prevalent definition regarding clinical purposes is that a patient’s symptoms have exhibited no response or inadequate response to two antipsychotics trials from different classes over at least six weeks with a therapeutic and tolerable dose of each antipsychotic (35, 53). At least 30% of patients with TRS respond to clozapine (54). American Psychiatric Association (35) and Comprehensive Textbook of Psychiatry, tenth edition (37) also recommend clozapine for patients with schizophrenia who are prone to suicide attempts or patients whose risk of suicide remains substantial despite other treatments.
Monitoring of clozapine plasma levels is not recommended routinely, but it is mentioned in some of the PGs as a suggested method for the optimization of clozapine treatment (55). In Iran, the plasma level of clozapine is not measured routinely, and if no response is evident after receiving an adequate duration and target dose (typically 300 - 450 mg/day) and clozapine is well-tolerated, psychiatrists increase the dose and wait for a response.
The third factor was “general principles for selection of an antipsychotic drug,” with six items.
Most PGs for schizophrenia now consider either first-or second-generation antipsychotics (other than clozapine) as potential first-line choices for the treatment depending on factors such as patient’s treatment history (if any), medical comorbidities and concurrent medications, potential cost concerns, history of the adverse effects of particular agents, the existing treatment formulations, capacity for drug-drug interactions, pharmacokinetic considerations, and patients’ or caregivers’ preferences (35, 39). If information about a patient is unavailable, particularly in newly diagnosed patients with no prior treatment, medication history for a first-degree relative with schizophrenia may be beneficial in selecting an agent for the patient in a similar manner (39).
The fifth factor was labelled “indications for the administration of an injectable antipsychotic drug” consisting of four items.
The application of antipsychotic long-acting injections (LAIs) (depots) is suggested when a patient prefers this type of treatment or when there is poor or uncertain adherence to oral treatments (35). The findings of a meta-analysis revealed that LAIs diminished relapse rates compared with oral antipsychotics (56). Also, a Finnish naturalistic investigation demonstrated that using LAIs resulted in a threefold decrease in the rates of rehospitalization compared with the oral application of a similar antipsychotic (57).
5.1. Limitations
The present study has some limitations. First, this study was conducted in the Iranian psychiatrist setting. Therefore, the results may not be generalizable to other countries. Second, in terms of there is a lack of national PG for treatment of schizophrenia in Iran, MATAPSCZ was developed using the European and American guidelines, NICE (38) and APA (35) that are in accordance with the guidelines used by the psychiatrists in the management of patients with schizophrenia in Iran. Using an online questionnaire for data collection was another limitation. Although, online Survey has some benefits, but the lack of face-to-face communication, the inability to verify the participant’s status, and the accuracy of the answers are online survey limitations. The fourth limitation is that confirmatory factor analysis was not performed due to the need for a larger number of participants to fill out the survey. It is recommended to perform it to confirm the construct validity of the tool in future studies.
5.2. Future Research
This study is the first step in forming a structure for further studies to evaluate individual patients, provide feedback to clinicians, identify barriers, and monitor continuous quality improvement in the pharmacotherapy of patients in the acute phase of schizophrenia. Moreover, the MATAPSCZ can be used as a valid tool to survey the association between treatment recommendation conformance to patient outcome, and cost impact in future studies.
5.3. Conclusions
The study showed that the MATAPSCZ consisting of 48 items with six factors had acceptable validity and reliability to evaluate prescribers’ adherence to PGs in the management of patients with APSCZ. The MATAPSCZ will provide evidence-based quality indicators to enhance the care of patients during the acute phase of schizophrenia. To characterize the most updated evidence and recommendations, this tool must be regularly checked when updating the guidelines.
