Psychometric Study Using Item Response Theory of an Instrument Developed for Assessment of Iranian Mental Health Problems

1. Background

In recent years, the burden of mental health problems, with their significant effects on health, socioeconomic consequences, and human rights in all countries of the world, has been increasing. According to studies, 1 in every 5 individuals suffers from some kind of mental health problem (1). However, lack of appreciation of the need for treatment and financial and other barriers hindering access to it, stigmata associated with mental health problems, disadvantages experienced by minorities, and cultural differences impede early diagnosis and treatment (2), which can lead to increased mental health treatment gap due to limitations in evidence-based mental health intervention resources (3).

Cultural difference is one of the diagnostic and therapeutic barriers to mental health problems (4, 5). This factor has effects on an individual’s understanding of the disease, the need for treatment, and health-seeking behaviors, such as attitudes toward therapeutic and preventive care (3). Cultural and environmental differences cause individuals to describe and prioritize their psychosocial problems in different ways, which can affect receiving therapeutic interventions (6). Therefore, diagnostic-therapeutic interventions in every country should be tailored to the cultural specifics of that particular country (7). Diagnostic and measurement errors are other factors that can arise and cause discrepancies in this regard (8).

A variety of diagnostic and screening instruments are available for the assessment of mental health problems and their interventions (9). However, few of these instruments are specifically designed to evaluate interventions in developing countries, which is one of the concerns and causes of patient neglect in these countries (6). Instruments used for disease screening and diagnosis in Western societies might not be able to detect the symptoms of mental health problems in other cultures, such as the Iranian society, because individuals in these cultures might consider some symptoms as usual and not as indicators of disease (1). Another factor determining the appropriateness of an instrument is the quality of being patient-centered to not impose a burden on the patient in terms of the number of items or complexity of the questionnaire (10). Short questionnaires are more appealing because they are easy to use and time-saving and decrease the burden of responsibility, thereby minimizing information loss (11). Designing shorter instruments is relatively common in all areas of psychology and psychiatry, and it is also desirable to reduce the number of items in the existing questionnaires (12). Several approaches are used in reducing the number of items, including the approaches of concept-retention, the equi-discriminative item-total correlation, and the Rasch model (4).

As part of a larger study on the assessment of culturally appropriate interventions, this paper describes the development of an instrument, based on item response theory (IRT), for the proper assessment of interventions. The IRT method is used to assess the cross-population utility of instruments that models the probability of a latent variable based on a given response as a function of the respondents. The IRT methodology allows how a tool performs in various populations by presenting information on identifying potential item-level bias and item characteristics (3).

2. Objectives

This study was part of the main intervention program on common mental problems. The present study aimed to validate the instrument developed for screening patients with common mental health problems using IRT. The current study was conducted on the adult population in two cities in Semnan province, east of Tehran (the capital), in northern-central Iran. The location was selected due to its proximity to the capital of the country and its low rate of immigration. According to the census of 2016, the population of the province was 352,285 individuals, comprising 36,298 households. In a national survey in 2017, the prevalence of suspected cases of mental health problems was 14.5% (13.1% of male and 15.8% of female individuals) (6). About 20%, 23.8%, and 7.2% of the sample had somatization (13.5% of male and 21.4% of female individuals), anxiety (17.7% of male and 26.8% of female individuals), and depression, respectively (6).

3. Methods

3.1. Preceding Qualitative and Instrument Validation Studies

A qualitative study was conducted to identify local mental health problems through developing, adapting, and validating a native instrument. The qualitative study showed a wide range of internalizing and externalizing symptoms (13). It was shown that the community was mostly concerned about symptoms related to common mental problems of depression, anxiety, and obsessive-compulsive disorder (OCD). All the existing validated measurement instruments for these three common mental disorders were listed and reviewed by the authors, and three instruments that were more symptom-based were selected. Prioritized symptoms/tasks or words/idioms selected in the qualitative study not present in these standard instruments were added with minor adjustments. For function assessment, a checklist, based on the qualitative findings of the study, was developed and reviewed for face validity with a group of the target population. Another review was performed with the interviewers as part of their training courses before the instrument test.

3.2. Measures

The following instruments were selected and modified:

3.2.1. Yale-Brown Obsessive-Compulsive Scale

Yale-Brown Obsessive-Compulsive Scale (Y-BOCS) assesses the symptoms of OCD in terms of severity. Wayne K. Goodman et al. created this scale, which is widely used in research and treatment of patients to assess the severity of OCD symptoms and the response to treatment. This scale has 90 items and rates obsessions and compulsions separately (14). The mean reliability values of the total scale are 0.866, 0.922, and 0.848 for the alpha coefficient, interclass correlations, and test-retest correlations, respectively (9). In the current study, some questions were modified or added based on the preceding qualitative findings. The modified instrument contained 21 questions.

3.2.2. Beck Depression Inventory Scale

The Beck Depression Inventory (BDI) scale was designed by Aaron T. Beck with 21 questions and is self-scored. This psychometric test is extensively used to assess the severity of depression (15). The modified instrument included 28 questions (1). The questions regarding religious beliefs, social networks, and digestive problems were added to the original questionnaire. In addition, the wording of some questions was updated according to the results of the qualitative research, including adding the sentence “I am locked-in due to too much thought” to Q10 and adding the word “penalty” to Q14 (1).

3.2.3. Beck Anxiety Inventory Scale

The Beck Anxiety Inventory (BAI) scale was designed by Aaron T. Beck et al. It is an inventory to determine the degree of anxiety in adults and children and is self-reported (16, 17). The modified instrument had 27 questions. Seven questions concerning aggression, sleep disorders, impatience/haste, disturbing thoughts, lack of focus, and worry were added to the BAI.

3.2.4. Function Checklist

This instrument was developed based on the results of the qualitative research with 25 questions. This checklist assesses different aspects of the individual (e.g., exercise, study, and recreation), familial (e.g., taking care of spouse/children and doing housework), and social (e.g., voluntary activities or helping others) activities. The numbers of questions related to individual, familial, and social aspects were 15, 6, and 4, respectively.

3.2.5. Final Package

The final version of the instrument consisted of 101 questions (depression: 1 - 28, anxiety: 29 -55, obsession: 56 - 76, and function: 77 - 101). The psychometrics of the final instrument was evaluated using face validity, content validity, criterion validity, and reliability as follows:

3.3. Psychometric Characteristics

3.3.1. Face Validity

Qualitative and quantitative methods were used to determine the face validity of the questionnaire. Face-to-face interviews were held with 10 members of the general public to determine the qualitative face validity. Items’ relevance and relationships and ambiguities, and difficulties in understanding the concepts and words were assessed. Modifications were then made according to participants’ views. The item impact score method was used to decrease the number of items, omit the inappropriate ones, and determine their significance. The items were rated using a 5-point Likert scale (from “not important at all” to “very important”) by the respondents in the qualitative section, and the item impact score was separately measured for each item using the following equation:

The items with an impact score ≥ 1.5 were deemed appropriate and were kept for the next stage of the analysis.

3.3.2. Content Validity

The content validity of the designed questionnaire was assessed using both qualitative and quantitative methods. The qualitative content validity assessment was performed based on 10 experts’ feedback on the importance, necessity, item allocation, grammar, wording, and proper scoring of the scale. The content validity index (CVI) and content validity ratio (CVR) were determined subsequently. The copies of the designed questionnaire were distributed among 10 psychiatrists to assess each item considering a 3-point scale (“necessary”, “useful but not necessary”, and “unnecessary”). Based on scores obtained from Lawshe’s table, based on the 10 experts’ assessment, the item was deemed necessary and essential at the statistically significant level of α < 0.05 (106) whether the minimum value of the index was greater than 0.62. An equation was used to find the CVR. Accordingly, the researcher distributed copies of the questionnaire among the 10 experts with the request to determine the simplicity, relevance, and clarity of the items based on Waltz and Bausell’s validity index. The experts assessed all the items using a 4-point Likert scale (e.g., relevancy, from 1 = irrelevant to 4 = totally relevant).

A CVI greater than 0.79 indicated an appropriate item. A CVI within 0.7 - 0.79 indicated a debatable item requiring modification, and a CVI less than 0.7 indicated an unacceptable item that had to be eliminated (15). Then, the mean scale-level content validity index/averaging calculation method (S-CVI/Ave) was utilized based on the average CVI of all items. Polit and Beck have suggested an acceptability score of 90% and higher for the S-CVI/Ave (1).

3.3.3. Criterion Validity and Item Response Theory

This study was performed within August 2017 to February 2018. Two methods were used to recruit the participants. The subjects were selected from the urban health centers and referrals to the psychiatric center in the main hospital in Semnan. The inclusion criteria were age over 18 years, residence in Semnan province for at least 5 years, registration in the Integrated Health System (SIB), and informed consent to participate in the study. The exclusion criteria were inability to communicate and acute psychiatric symptoms.

For comparing the mean scores for each problem of those who had the problem to those who did not (arbitrarily use of 20% difference as the cut-off), 40 subjects were selected for each problem (a total of 120 participants) and 40 subjects without any problem or healthy. The project staff used the community health records repository (SIB online platform) to assemble the list of participants. In the health centers or psychiatric centers, the study interviewers explained the project to the participants, completed the informed consent process, and then administered the study instrument. After completing the instrument, the participants were referred to a trained psychiatrist to be assessed according to the Structured Clinical Interview for DSM Disorders (SCID) measurement.

3.4. Background Variables

Background variables were included in the sociodemographic section of the survey instrument. Age was measured in years. The educational level had three categories, namely primary (less than 5 years of education), intermediate (5 - 12 years of education), and high (more than 12 years of education).

3.5. Statistical Analysis

For the achievement of the overall aim of the study, we developed a short, reliable, and valid measure of three main mental health problems, namely anxiety, depression, and OCD, by shortening a questionnaire, including 28, 21, and 27 items for depression, OCD, and anxiety, respectively. The latent traits (i.e., depression, OCD, and anxiety) were measured using an instrument, which is a collection of items. Each item has a difficulty parameter and a discrimination parameter. The correct answer is given by the gold standard criterion using the SCID-based clinical assessment for each mental disorder. This study investigated the probability of a positive response to each item separately.

The IRT for binary outcome was used as the item-reduction method on cross-sectional field-tested data derived from 160 participants from healthcare centers in Semnan. A series of binary response items for each dimension was created in which the response option “Not at all” was regarded as 0, and response options 1 - 3 were regarded as 1. This study reported the discrimination parameter (a) and difficulty parameter (b) using the IRT parameterization. The items with high discrimination, based on thresholds reported in the literature, were retained in the final version of the questionnaire. Generally, item discrimination was regarded as very low to very high (Baker, 2001).

Reliability analysis was performed on the remaining items for each dimension through Cronbach’s alpha and item-rest correlations. Sensitivity and specificity analyses were performed using the receiver operating characteristic curve to assign a cut-off point. Statistical analyses were carried out using Stata software (version 15.1; StataCorp, College Station, TX, USA).

3.6. Ethical Considerations

The study protocol was approved by the Research Ethics Committee of the National Institute for Medical Research Development (NIMAD) under the code IR.NIMAD.REC.1395.047. All the participants signed the consent form and verbally consented to participate in the study.

4. Results

There were 160 participants in the present study, 58.2% of whom were male. The mean age of the participants was 36.3 ± 11.2 years (range: 17 - 75). More than 50% of the participants had an intermediate education (Table 1). The results are presented in the two following parts:

4.1. Validity of the Common Mental Health Screening Instrument

The initial common mental health scale package consisted of 101 questions (depression: 1 - 28; anxiety: 29 - 55; obsession: 56 - 76; function: 77 - 101). Face validity, content validity, and criterion validity were used to validate the instrument. In the face validity stage, the necessary changes were made after receiving the opinions of the members of the public. The impact factors of the items were within the range of 1.8 - 5, and none of the items had an impact factor of less than 1.5. Therefore, all the items were transferred to the content validity stage.

In the qualitative content validity stage, the necessary changes were made after receiving the opinions of experts in relevant fields. The CVR and CVI results were obtained. For all of the items, CVR and S-CVI/Ave were within the range of 0.7-1 and 97.09 ± 0.63, respectively. The mean values of clarity, relevance, S-CVI/Ave, and simplicity were 96.73 ± 0.70, 98.2 ± 1.9, 97.09 ± 0.63, and 97.64 ± 0.61, respectively. In the criterion validity stage, the final package was examined with the SCID (Table 2).

4.2. Item Reduction

Table 3 shows the results for IRT, including discrimination and difficulty parameters. Discrimination parameters for the depression, OCD, and anxiety dimensions were within the ranges of a = 0.72 (depression 27) to a = 2.59 (depression 9), a = 0.95 (OCD74) to a = 2.48 (OCD70), and a = 1.11 (anxiety 44) to a = 4.55 (anxiety 54), respectively. Based on the threshold criteria, this study excluded items 3, 8, 10, 14, 15, 18, 19, 20, 21, 22, 25, 27, and 28 from the depression dimension, items 29, 30, 44, 46, and 51 from the anxiety dimension, and items 56, 57, 60, 61, 63, 64, 66, 68, 72, 73, 74, and 75 from the OCD dimension. Cronbach’s alpha coefficients were calculated at 0.92, 0.95, and 0.88 for depression, anxiety, and OCD, respectively, suggesting high reliability for all three dimensions.

Sensitivity and specificity analyses were performed to assign a cut-off point. The choice for depression was based on 89% sensitivity and 59% specificity, determining a cut-off point of 14. This finding means that if the sum of the scores on items for the depression dimension exceeds 14, the individual is considered positive in the screening test. Similarly, considering 80% sensitivity and 44% specificity, a cut-off point of 17 was chosen for anxiety. Given 80% sensitivity and 49% specificity, a cut-point of 6 was considered for OCD (Table 4). In this study, Cronbach’s alpha and internal consistency coefficients were 0.88 and 0.7, respectively.

5. Discussion

This study to develop and test an instrument is the first in a series of investigations to assess the performance of screening instruments. This series of studies includes vast epidemiological studies and clinical validations to determine shared constructs between instruments using factor analysis to enhance the assessment of the screening and treatment of mental health problems in Semnan province. This study performed a transcultural translation of the BAI, BDI, and Y-BOCS. This study developed a function assessment instrument locally and learned through the process of qualitative transcultural translation that some items of the BDI and BAI were not applicable in the Persian version of this questionnaire due to the lack of interpersonal interpretation, specificity, or conceptual nonequivalence. In addition, it was understood that various interpretations of somatic idioms affected the proper recognition of the physical problems in mental health screening. Several crucial lessons can be learned from the adaptation and development of screening instruments during the process of expanding healthcare services to include mental health. It has been proposed that to correctly identify those individuals who require mental healthcare, it is essential to recognize practices and beliefs that are considered normal in a particular culture, such as communicating with deceased ancestors, and differentiate them from probable mental illness (12, 18-20).

In the present study, the final mental health scale package was developed as a short, reliable, and valid measure of three main mental health problems consisting of anxiety, depression, and OCD by shortening a questionnaire containing 28, 21, and 27 items for depression, OCD, and anxiety, respectively. Content and face validity studies confirmed the clarity and simplicity of the items. The relevance of the items in the CVI showed a significant degree of agreement among the experts.

The correlation coefficients of the values obtained from this instrument and the SCID were used for the criterion validation. The results presented a correlation between these instruments that measured the same issue.

Exploratory factor analysis was used to reduce the number of items in several studies (21). Specifically, this approach has been utilized to identify items with loadings below 0.4 on any conceived factor. This identification results in the elimination of the pertinent items from the model. The use of this method is limited by the fact that it accounts for neither the structure of the original factors nor the model’s structure. Moreover, exploratory factor analysis is intended to be used to explore novel constructs and not existing scales (22). The IRT method is a kind of latent variable analysis utilized for a better understanding of how an instrument performs in various populations by identifying potential item-level bias and providing information on item characteristics across populations. The IRT analysis includes discrimination and difficulty parameters (3).

The study of any subject needs a suitable tool for data collection with the least number of errors and the most level of precision (23). An instrument designed in a country is influenced by the culture of that country, and its use elsewhere, even via precise translation, might lead to problems due to the content considered not being a good fit (24). In short, psychometrically sound measures introduce more efficient methods of quantifying patient outcomes to researchers and clinicians while retaining the validity and reliability of the longer versions. Such tools offer the advantage of providing the same quality of information with less burden for the patient and easier scoring for the researcher or clinician (4).

The comparison of the results of the present study to other similar studies indicates that this tool has a perfect sensitivity to identify mental health problems. Brief Jail Mental Health Screen generally had a sensitivity of approximately 65% (% 95 CI: 47-48%) (25). England Mental Health Screen has only 50% sensitivity in a small subsample of 18-21-year-old male subjects (26). In the current study, the choice for depression, anxiety, and OCD was 80-89% sensitivity and 44 - 59% specificity.

In this study, Cronbach alpha coefficients were calculated at 0.92, 0.95, and 0.88 for depression, anxiety, and OCD, respectively. Therefore, it presented a high internal consistency among the items confirming the reliability of the instrument. However, the reliability coefficient of the General Health Questionnaire based on Cronbach alpha was calculated at 90 (27).

A few challenges encountered in the studies warrant discussion. Firstly, local supervisors had difficulty finding and/or finishing a pilot case. Feedback received from all supervisors suggested that time was the primary barrier to not completing a case. Other barriers to the projects across both cities were mainly organizational and logistical, such as transport and personnel problems. Future studies are needed to examine organizational facilitators and barriers. With regard to the duration of follow-up, this study did not evaluate the longitudinal effects of the intervention. Although the follow-up was, on average, longer than one month after treatment, additional postintervention assessments of 6-12 months after treatment would be more informative.

5.1. Conclusions

Iranian mental health problems screening tools can provide qualified information with the least error and the most precision in appropriate early diagnosis and decrease the burden of mental health problems in the national healthcare system. This questionnaire has 71 questions, including 15, 9, and 22 items for depression, OCD, and anxiety, respectively, and one checklist with 25 items for the function.

The role of the lack of mental health professionals (i.e., psychiatrists or psychologists) shifting from treating a few cases to supervising the treatment of many individuals through the community health workers is supported as community health workers were able to learn and provide both interventions with fidelity. This approach to task sharing is supported by other studies as a good option for providing sustainable, accessible, and effective services for multiple mental health problems at a scale where there are few professionals.