Psychometric Properties of Persian Version of Structured Clinical Interview for DSM-5-Research Version (SCID-5-RV): A Diagnostic Accuracy Study

1. Background

The diagnostic and statistical manual of mental disorders (DSM), as a semi-structured interview mainly intended to facilitate communications among professionals. Besides, the evolution of each edition of DSM (DSM-1 to DSM-5) indicates a trend toward addressing the limitations of the previous versions (1). Despite its limitations, DSM is a valuable instrument for both practice and research purposes (2).

To conduct better assessments, clinicians should obtain more in-depth information. Moreover, in-depth information improves treatment planning and monitoring that in turn, enhances the reliability and validity of the diagnostic nosology. Further, clinicians should know how to increase clinical utility while addressing the shortcomings of previous versions of the DSM, including high comorbidity rates, the exclusive emphasis on categorical classification, a need for incorporating a dimensional approach, and the excessive use of “not otherwise specified” (3-5). Accordingly, for more than a decade, the DSM workgroup was developing the DSM-5. To this aim, evidence from clinical practice, neurological, genetic literature, epidemiological findings, cognitive neuroscience, and pathophysiology, which yielded important insights about pathology, particularly treat debilitation cases, were used (1).

In line with the goal of Research Domain Criteria launched by the National Institute of Mental Health and to address the above-mentioned limitations, the following items were added to the DSM-5: possible shared etiologies, subthreshold symptoms, and cross-cutting symptoms. Besides, by adding specifiers for “not otherwise specified” (NOS) and other categories, supplementing the DSM categorical system with dimensions, and expansion of categories, its clinical utility was expanded (1, 3).

The workgroup who proposed the diagnostic criteria confirmed the face and construct validity of the DSM-5 (6). To assess the reliability of the DSM-5, a selection of DSM-5 categorical diagnoses was used in a field trial, and it was found that most diagnoses with an adequate sample size had a good to very good reliability. Therefore, since DSM is the most widely used interview structure in adult psychopathology and availability of several revisions of DSM-5, assessing the reliability and validity of the DSM-5 diagnosis is of crucial importance.

Semi-structured interviews, such as the SCID, have several advantages over paper-pencil questionnaires. For instance, probing and clarifying the questions and their answers cannot be distorted by the respondents’ reading level, which allows for assessing a wide area of psychopathology categorically or dimensionally and accurately capturing the multifactorial nature of psychopathology. The structural clinical interview is used to determine DSM-5 axis I disorders, and it is also useful for research purposes (7, 8).

To the best of our knowledge and for linguistic adaptations, this is the first study that translated SCID5-RV in Iran by implementing it in a large and diverse adult clinical sample and assessed the reliability and validity of the SCID-5-RV. Accordingly, the psychometric properties of the latest version of SCID were evaluated to see whether it has adequate reliability and validity in the nonwestern country. Additionally, the current study paves the way for future research to evaluate DSM-5 criteria on the Iranian population or for applying it in other studies and clinical practices.

2. Objectives

Thus, the current study aimed to assess the reliability of the Persian version of SCID-5-RV, including test-retest reliability, and to evaluate the diagnostic accuracy of SCID-5 v, including the criteria validity, sensitivity, and specificity. Besides, construct validity was also assessed.

3. Methods

3.1. Participants

This is a diagnostic accuracy study that aimed to assess the reliability of the Persian version of SCID-5-RV. Participants were selected among clients who had referred with at least one psychiatric complaint to the psychiatric clinics or hospitals (15 facilities for adults in Tehran) during 2018 - 2019. The research proposal, study measures, and informed consent form were approved by the Ethics Committee (no.: IR.USWR.REC.1395.264).

To recruit participants, first, the city of Tehran was divided into northern, southern, eastern, and western districts, and a clinical facility was randomly selected in each district. Finally, based on pre-determined criteria, eligible clinical participants were chosen. Inclusion criteria were as follow: being born in Iran, the ability to speak in Persian, aged more than 18 years, having mental disorders symptoms, and signing the informed consent to participate in the study. The exclusion criteria were having hearing and speech problems, intellectual problems, receiving different treatments, and failure to cooperate with the researcher due to restlessness or severe psychosis. In total, 305 clinical participants were admitted to outpatient or inpatient facilities, and 40 non-clinical participants, mostly staff members of the hospitals, were selected.

While this study is an accuracy study design, the below formula was used to calculate the sample size.

Which the anticipated specificity and the value of L were considered as 0.80 and 0.05, respectively. Besides, according to the literature, the prevalence of mental disorders in the general population was estimated at 15%. Placing these values in the above formula resulted in a sample size of 289. However, due to reasons such as failure of samples, we decided to increase the sample size to 370 n (289 + 25%). Eventually, 305 valid clinical participants were eligible for data analysis.

3.2. Instruments

The SCID-5-RV, Millon Clinical Multiaxial inventory-III (MCMI-III), and Brief Symptom inventory (BSI) were used in this study.

The Persian translation of SCID-5-RV was administered, and the procedure was supervised by the first author, who published the Persian version of SCID-I and is one of the contributor members of reviewing Structured Clinical interview for DSM-5. The SCID-5-RV normally administers in a single session and takes 45 - 90 minutes to administer. To achieve a linguistic adaptation, the following steps were performed: forward translation by bilingual (English/Persian) translators, evaluation, and revision of the translated SCID-5-RV in an expert panel comprised of mental health professionals (three psychiatrists, five clinical psychologists, and three translators). After comparing the English and Persian versions and approving content validity, it was confirmed that both versions are identical, and the only difference about them is the language. Further, to assess the understandability and applicability of the instrument, the Persian version of SCID-5-RV was piloted on a small sample of clinical participants (n = 40).

The MCMI-III is a well-known, true-false, and self-administered questionnaire for assessing DSM-based disorders, which contains 175 items about the severity of symptoms and 24 scales (14 for basic personality, and 10 for the clinical disorder) (9, 10). The present study utilized 10 clinical scales, including generalized anxiety, somatic symptom, bipolar disorder, persistent depressive disorder, alcohol and drug use, post-traumatic disorder, thought disorder, major depression, and delusional disorder. It is proved that MCMI-III subscales have very good validity, including high predictive validity (ranged from 0.92 to 0.98), in the Iranian population (11). In the current study, the internal consistency of the clinical scales was in an acceptable range of 0.72 - 0.96.

Brief Symptom inventory (BSI) is a self-administered questionnaire that comprises 35 items derived from the longer 90-item symptom check-list-90 R (SCL-90-R), and each item has a 4-point Likert-type scale ranging from 0 (not at all) to 4 (extremely). BSI assesses a series of psychological symptoms encompassing depression, irritability, phobic anxiety, psychoticism, interpersonal sensitivity, somatization, obsessive-compulsive, anxiety, hostility, and paranoid ideation (12). Internal consistency for all nine subscales of BSI dimensions ranged 0.75 - 0.89, which is acceptable. BSI also showed satisfactory construct validity (13). The internal consistency of BSI subscales in the Iranian population is reported to be 0.71 - 0.96 and yielded satisfactory construct validity (14). In the current study, the internal consistency of BSI subscales was in an acceptable range of 0.88 - 0.96.

3.3. Training

57 psychologists (by local announcements) with a master’s degree or higher agreed to contribute as an interviewer in the present study. All volunteers were assessed by the first author to ensure their diagnostic competency as well as interviewing skills, and being well experienced. In total 37 psychologists were selected as an interviewer. Then, two one-day training workshops were held with a 45-day interval. The first workshop was focused on the purpose of the study, the main goals of DSM-5, the structure and content of the SCID-5-RV, the changes made in DSM-5 and SCID-5-RV, and the interviewing skills. After the workshop, each psychologist was asked to interview with two clinical participants and fill in the DSM-5 checklist for SCID-5-RV at the next workshop, followed by checking and evaluating all completed interviews. Finally, 23 psychologists, who had good skills in interviewing, were asked to contribute to the current study as interviewers and were continuously supervised throughout the study implementation.

3.4. Procedure

At each facility, a coordinator (with an MSc degree in clinical psychology) arranged all steps before interviews, including checking the inclusion and exclusion criteria, debriefing participants about the purpose of the study, and obtaining written informed consent. Then, the coordinators were providing the interviewers with sufficient information about the interviewees.

From 370 clinical participants and a total of 45 nonclinical participants, 65 clinical participants and five nonclinical interviews were removed due to incomplete information. Ultimately, MCMI-III and BSI were filled by the participants with different order of measures.

Additionally, to assess the test-retest reliability, 50 interviews with clinical participants were randomly selected, and after two weeks, another interview was performed by an independent interviewer who was not aware of the intention of the interviews.

To study the construct validity, SCID-5-RV diagnoses were compared with the clinical diagnoses made by blinded psychiatrists. For this purpose, all 350 participants were interviewed by a psychiatrist or clinical psychologist, the time limit between 2 measures was 7 days. and the agreement between the SCID-5-RV interview and diagnosis made by psychiatrists were compared.

3.5. Data Analysis

Data were analyzed using SPSS software version 21 (15). Descriptive statistics such as mean, standard deviation, and frequency were estimated for categorical diagnoses (disorder) and demographic information. Regarding the reliability and validity, the zero-order correlation was used to examine the concordance between diagnosis made by two distinct examiners and the diagnosis based on the SCID-5-RV interview and the psychiatrist’s interview, respectively. The following standards were applied to examine the agreement between the diagnosis of the psychiatrist and SCID-5 diagnosis:

Intra-class kappa was considered unacceptable, questionable, good, very good, and excellent if it was below 0.20, 0.20 - 0.39, 0.40 - 0.59, 0.60 - 0.79, and above 0.80, respectively (16).

In addition to examining the internal consistency, Cronbach’s alpha was also calculated for each categorical diagnosis. The following rule of thumb was set for an acceptable range:

“> 0.9 = excellent; > 0.8 = good; > 0.7 = acceptable; > 0. 6 = questionable; > 0.5 = poor; < 0.5 = unacceptable” (17).

Regarding test-retest reliability, the Pearson coefficient was applied to evaluate the correlation between scores obtained in rounds one and two for each disorder. Further, the following classification was used to confirm the correlation coefficient:

Up to 0.19 = very weak; 0.2 = 0.39 = weak; 0.40 = 0.59 = moderate; 0.6 - 0.79 = strong; 0.8 - 0.9 = very strong; 1.0 = perfect correlation (18).

To assess the validity, other operating characteristics were calculated, including sensitivity, specificity, positive likelihood ratio, and negative likelihood ratio. To obtain the criterion validity, the degree of agreement between the diagnoses made by the assessor and psychiatrists was evaluated using the Kappa coefficient.

To evaluate the construct validity of the SCID-5-RV diagnoses, multivariate analysis of variance (MANOVA) was employed to assess the significant differences between clinical and non-clinical samples in the subscales scores of MCMI-III and BSI. Significance of differences between clinical and non-clinical groups in the pathological scales of MCMI-III and BSI indicated the construct validity of SCID-5-RV diagnoses (19).

4. Results

4.1. Descriptive Findings

The demographic characteristics of the participants and the frequency of their diagnosis are shown in Table 1. As shown in Table 1, of 305 total clinical participants, 169 (55.4%) were female, and their mean age was 34.31 (SD = 11.96).

4.2. Reliability

Data related to the reliability of diagnosis are shown in Table 2. Internal consistency of all diagnoses ranged from α = .95 to α = .99, which demonstrates an excellent internal reliability. Furthermore, the composite reliability for all diagnoses was α = .99, which is highly acceptable (Table 2). Also, the test-retest reliability for all diagnoses ranged from r = 0.60 to r = 0.79, which indicates that all disorders have good test-retest reliability (Table 2). The Kappa coefficients of two distinct examiners (n = 65) ranged from 0.57 to 0.72 (Table 2), which is acceptable.

4.3. Validity

As shown in Table 3, the agreements obtained using κ statistics were good to excellent for all diagnostic categories (κ = 0.63 to κ = 0.83). Obsessive-compulsive, eating disorders, and delusional disorders had the largest Kappa coefficient degree, while depressive episode, panic disorder, and acute stress disorder were among disorders with the lowest Kappa coefficient degree.

For most psychiatric disorders, specificity values were high (n > 0.85), and the sensitivity values were to somewhat lower (Table 3). In the medical literature, these indexes, which represent the diagnostic validity of the test, are referred to as a form of criterion validity (20). Generalized anxiety disorder and schizophrenia had the highest and the lowest sensitivity, respectively. Schizophrenia, schizoaffective disorder, and bipolar-II had the highest specificity, and panic disorder had the lowest specificity. In addition, LR+ for various categories ranged from 2.26 to 10.89, and LR- of categories ranged from 0.16 to 0.42.

To assess the construct validity of diagnostic categories, clinical and non-clinical participants were compared regarding the subscales of axis-I disorders of MCMI-III and those of BSI by using MANOVA. The multivariate effect was statistically significant for MCMI-III (F (10, 123) = 3.45, P < 0.001, partial η² = 0.22) and BSI (F (9, 149) = 3.45, P < 0.02, partial η² = 0.12) subscales. Further, the results of univariate tests indicated significant differences among the groups on all the subscales of MCMI-III, including generalized anxiety, somatic symptom, bipolar disorder, persistent depression disorder, alcohol and drug use, post-traumatic disorder, thought disorder, major depression, and delusional disorder as well as BSI subscales, including somatization, obsession-compulsive, interpersonal sensitivity, depression, anxiety, hostility, paranoid ideation, and psychoticism, except for the phobic anxiety. The results of the univariate analysis of MCMI-III and BSI subscales are provided in Table 4.

5. Discussion

Diagnostic reliability and validity have an important role in the progress of science and the practice of clinical psychology and psychiatry, without which the accurate identification of risk factors, interpretation of psychopathology, and efficacy of treatment would be erroneous (21, 22). Furthermore, cross-cultural and cross-national evaluation of a universal diagnostic interview, such as DSM, are milestones for further studies, particularly clinical trials, in clinical settings.

The findings of the current study are about the initial psychometric properties of SCID-5-RV categorical diagnosis among the Iranian adult population. Generally speaking, SCID-5-RV categorical diagnosis demonstrated good psychometric properties (i.e., excellent internal consistency, test-retest reliability, criterion validity, and acceptable sensitivity, specificity, and construct validity) which are comparable to the properties of the original SCID-5-RV categorical diagnosis (6, 22).

Investigating the reliability showed satisfactory internal consistency, test-retest reliability, and agreement between two distinct examiners. It worth noting that, in the current study, intra-class kappa coefficient and test-retest reliability of diagnosis categories were higher than those in the previous field trial on categorical disorders that used DSM-5 (22), as well as the DSM-IV among the Iranian population (23). Usually, the test-retest method results in a lower kappa coefficient, as in the current study, compared to the joint or inter-rater method (24). However, the results of the present study are in contrast with such a claim, since the obtained kappa coefficient was extremely higher than that of the studies which used two examiners who observed and rated the same interview independently. This may be related to the larger sample size, and higher homogeneity of the interviewers and clinicians who conducted the interviews (in terms of expertise, age, ethnicity, and other related variables which may affect the intended diagnosis). One of the strengths of the current study was the participation of different clinicians with various clinical disciplines, years of clinical practices, ethnicity/race, age, sex, and other characteristics from all around the country, which enhanced the generalizability of the findings.

Additionally, based on the results of the current study, major depressive disorder, acute stress disorder, and panic disorder had the least Kappa coefficient degree, which is as another important finding of this study that may be related to the lower sample size and comorbidity of these disorders, along with several more severe disorders, including other anxiety or mood disorders, which are common in these diagnosis categories (25-27).

In addition, evaluating the internal consistency of diagnosis categories is an important advantage for the present study, which yielded an excellent alpha coefficient for all disorders, indicating a high correlation between the items of each diagnosis. Reliability coefficients are high and it shows that the designed items measure the target construct with high consistency. Such results are not unexpected for a diagnostic tool, because the items are specifically designed for a specified purpose, and the higher the reliability value, the more reliable the measure (28). Furthermore, the adequate internal consistency was observed even for some disorders that their criteria was changed in DSM-5, including the alcohol use or even several specifiers and MDD with melancholic features. The findings of the current study indicated that the changes from DSM-IV-TR to DSM-5 are acceptable, even in Iranian culture.

To assess the validity of the Persian version of the SCID-5, we compared the results obtained from the questionnaire with clinical diagnoses made consensually by psychiatrists. The results indicated the adequacy of agreement, including acceptable kappa coefficient, high specificity, and almost good sensitivity and likelihood ratio for nearly all diagnoses. The results of the current study are very similar to the findings of studies conducted using previous versions of SCID on Iranian samples (23).

Sensitivity and Specificity should never be interpreted in isolation as a means for evaluating the clinical utility of a measure (29). The sensitivity and specificity are directly related to the diagnosis of positive and negative cases, respectively. A test with 60% sensitivity, correctly classifies 60% of individuals as patients (true positives), and therefore it is unable to identify the remaining 40% of the patients (false negatives). A test with 60% specificity, correctly identify 60% of people who truly are not sick (true negative), but cannot identify 40% of them (false positive). The likelihood ratio is defined as the likelihood that a patient whose test is positive is really sick compared to patients whose test was negative (30). The observed value of these indexes showed that the SCID is sensitive for positive diagnoses and can identify people without disorder correctly. The observed values of LR showed that these diagnoses are of high accuracy. Besides, there was evidence about the construct validity of SCID-5-RV diagnosis categories, representing significant differences between the clinical and non-clinical population in nearly all subscales of the BSI and MCMI-III, which confirm that the criteria of SCID-5-RV disorders could accurately discriminate clinical and non-clinical populations. The results, which represent the variance in the diagnosis criterion of SCID-5-RV categorical diagnoses, reflect the variance in the underlying construct (31).

The present study has several practical and clinical implications. First, it was conducted across a variety of clinical settings, including academic and private clinical settings, thus a diverse clinical population is captured. Besides, it was conducted by interviewers from various fields. In addition, test-retest and multivariate methodological designs are highly effective methods that their application increases the generalizability of the findings. More importantly, the exclusion criteria were minimal and clinical participants were more similar to natural clinical settings. Also, SCID-5-RV categorical diagnoses are highly reliable and valid when applied to the Iranian population and have good cross-cultural psychometric properties. Thus, these categorical diagnoses can be used in different clinical settings by various interviewers.

Despite having several strengths, the current study has some limitations. First, the sample size was inadequate for all disorders, and calculation of some statistics was not possible for some disorders. Therefore, further studies should evaluate the cross-validation analysis of more diverse disorders with a larger sample size. Accordingly, specific pilot studies are recommended for each spectrum.

In the present study, first, validity and reliability of the Persian translation of SCID-5-RV were examined, and the construct validity of SCID-5-RV categorical diagnoses was assessed using the MANOVA method. Then, BSI and MCMI-III subscales were compared between clinical and non-clinical populations as an innovative and pioneer perspective to assess the construct validity of DSM-5, which can pave the way for further elaboration of categorical diagnoses. Overall, the SCID-5-RV categorical diagnosis had reliable and valid diagnoses for almost all diagnostic categories in clinical settings in Iran. These results contribute to the growing body of evidence supporting the reliability and validity of SCID-5-RV diagnosis categories and representing the cross-cultural use of the instrument. The authors suggest conducting further studies concerning testing and cross-validating the diagnostic criteria. Consistent with the National Institute of Mental Health, Research Domain Criteria project aims to identify the symptomatic and biological dimension of psychopathologies. Future studies are required to investigate cross-cutting dimensional measures of DSM-5 and to build a foundation for incorporating dimensional diagnoses into categorical diagnoses, which help to improve case formulation and treatment plan.