Abstract
Background:
An estimated 25% to 30% of patients presenting to family physicians have psychiatric disorders (PDs). Accurate diagnosis of various PDs requires valid and well-designed screening tools and psychiatric interviews. There is no rapid and low-cost tool for PD screening in primary health care (PHC).Objectives:
To determine the validity and psychometric properties of the Iranian Rapid Assessment for Psychiatric Illness Screening Instrument (IRA-PISI) in PHC.Methods:
This psychometric study was performed among a convenience sample of 257 outpatients presenting to the first-line health services in Tabriz. Psychiatric interviews via DSM-5 were used as the gold standard. The validation process included the face validity (literature review and expert comments), content validity, estimating the best cutoff point, and measuring the sensitivity and specificity of the questionnaire compared to the psychiatric interview. Temporal and internal reliabilities were measured by test-retest and Cronbach's alpha coefficient, respectively.Results:
The final questionnaire consisted of 14 items (with a score range of 14 to 70) after removing seven questions from the initial questionnaire (21 items) during the validation process. The sensitivity and specificity of the questionnaire compared to psychiatric interviews were 83.62% and 75.17%, respectively. The overall relevance and the best cutoff point (score) were 0.91% and 28, respectively. Interclass correlation coefficient (ICC) and Cronbach's alpha coefficient were 96% and 83%, respectively. The ICC of items/questions ranged from 89.3 to 100.Conclusions:
The IRA-PISI has appropriate validity and reliability for screening common PDs among outpatients presenting to first-line healthcare providers.Keywords
Mental Disorders Primary Health Care Psychometrics Questionnaire Screening
1. Background
Psychiatric disorder (PD) is a major public health concern around the world (1). Currently, communities are exposed to a warning increase in the incidence of PDs all over the world, due to various social and political turmoil and the continuous waves of violence (2). It is estimated that 25% to 30% of patients presenting to family physicians have PDs. In 2020, mental disorders were responsible for 20% of disabilities, compared to only 9% in 1990 (3).
In Iran, the incidence rate of PDs is growing for many reasons, including population growth, social changes, the collapse of the family foundation, and economic problems (4). Recently, a systematic review and meta-analysis study from Iran revealed that the prevalence rates of PDs in studies that used screening tools and clinical interviews were 31.03% and 25.42%, respectively, compared to 28.70% by screening and 18.60% by interviews in 2007 (3). Identifying and screening mental disorders in the first-line health care system are unavoidable (3). Accurate diagnosis of various PDs requires valid and well-designed screening tools and clinical psychiatric interviews (5). Clinical interviews require high costs and time. Patient self-report scales do not provide valid and reliable diagnoses and sometimes lead to false-positives and/or false-negative implications (6).
Screening for early case detection and treatment of mental disorders can improve the quality of life, and reduce health system costs (7). Tools stemming from the Patient Health Questionnaire (PHQ) and other valid and reliable instruments had the most application in many fields and settings (8). However, these numerous tools could not meet the needs of PHC and primary physicians and need too much time and many psychiatric interviews. This psychometric investigation provides a rapid instrument that first-line healthcare providers can use to screen various PDs in the early stage (9, 10). Screening tools should be well-designed, rapid, concise, and understandable, and accurately identify major and common PDs without distress to patients. Access to such a tool is limited. Moreover, screening tools should be localized; thus, tools used in developed countries cannot identify PDs in low and middle-income countries (11).
The primary health care (PHC) system is an effective and important place for detecting and managing mental health disorders. The PHC services are the most readily available means of health care. With high rates of regular contact, primary care providers play a noticeable role in mental health promotion (12, 13).
2. Objectives
There is no rapid and low-cost screening tool for screening common PDs in PHC. This psychometric study aimed to develop and validate the psychometric properties of the Iranian Rapid Assessment for Psychiatric Illness Screening Instrument (IRA-PISI) in the first-line health care system.
3. Methods
3.1. Study Design
This psychometric study was performed to develop and validate the psychometric properties of the IRA-PISI for screening common PDs in first-line outpatient PHC in Tabriz between 2019 and 2020. The study population included outpatients presenting to family physicians and/or first-line providers in three medical centers, including a community health center, Emam Khomeini Clinic, and a private clinic in Tabriz. The validity and reliability were assessed among outpatients presenting to family physicians in the PHC setting. Considering P = 0.2, α = 0.05, d = 0.2P, and 20% compensation for non-respondents, the sample size was calculated at 257 outpatients. A convenience sampling method was used for selecting the sample. Family physicians are the chief first-line healthcare providers in the Iran health system. They provide mainly therapeutic and preventive healthcare services for a large number of community members.
3.1.1. Eligibility Criteria
Inclusion criteria were age 18 - 60 and informed consent, and exclusion criteria were mental disability, cognitive disorder, and major physical illness such as cancer and chronic hepatitis (13).
3.2. IRA-PSI Development Process
3.2.1. Literature Review
We systematically searched Medline (via Ovid), the Cochrane Library, and Psych Info to find the most relevant records, including common psychiatric disorder screening tools or/and questionnaires and their contents. The initial search was performed with the most relevant medical subject heading (MeSH) keywords related to "questionnaire," "psychiatric," and "screening" in the title and/or abstract. The initial search was combined with related terms, including tool, psychological, primary health care, PHC, disorder, diagnosis, case finding, validity, reliability, and family physicians. Boolean operators AND, OR, and NOT were used to combine related words and phrases. Grey literature, such as the WHO or CDC websites, was thoroughly explored. There were no time and language restrictions.
3.2.2. Expert Comments
Twelve psychiatrists were recruited to generate the questionnaire. They had work experience of 16.11 (SD = 6.98) years. Tools/questionnaires and their contents identified in the literature review were discussed in the expert panel (psychiatrists) to generate the questionnaire items. The literature review found instruments, including PRIME-MD, PHQ9 and 15, and K10. A focus group discussion (FGD) was used to generate the questionnaire items. The experts discussed the items and questions of the explored tools, identified relevant items, rated them, removed overlap items, and finalized them to generate the IRA-PIS questionnaire. Common psychiatric disorders included depressive, anxiety, psychotic, obsessive-compulsive, bipolar, and personality disorders. The following principles were considered in the development and scoring steps: (1) each question measures only one variable; (2) the questions should be as simple and short as possible; (3) the questions must be comprehensive; (4) the items use positive sentences; (5) the answers should be on a Likert scale, including always (score = 5), often (score = 4), sometimes (score = 3), rarely (score = 2), and never (score = 1).
3.3. Validity
The first version of the tool was shared among 12 experts to be scored in terms of content validity. The experts were asked to rate instrument items in terms of clarity and relevance to the study constructs on a four-point ordinal scale (1: Not relevant, 2: Somewhat relevant, 3: Quite relevant, and 4: Highly relevant). The number of experts giving a rating of 3 or 4 was divided by the total number of experts to obtain the Content Validity Index (CVI) for relevance and clarity of each item. Furthermore, suggestions from the expert panel and item impact scores were used to examine the instrument's face validity.
Receiver operating characteristic curve (ROC) was used to measure the best cutoff point of the questionnaire. Structured interviews via DSM-5 (a binary outcome) were conducted by a highly experienced psychiatrist as a gold standard. The structured interviews and questionnaires were used for screening the same sample of 257 outpatients referred to the family physician. Then, the best cutoff point, where it had the most sensitivity and specificity, was measured.
After determining the best cutoff point among 257 outpatients referred to the first-line health services (family physicians), the screened outpatients were divided into a binary group (positive screen and negative screen). Table 1 shows the method for calculating the sensitivity and specificity of the questionnaire. By presenting data in a two-way table, the sensitivity and specificity of the questionnaire compared to the gold standard were measured.
| Questionnaire | Gold Standard | |
|---|---|---|
| Positive | Negative | |
| Positive | TP | FP |
| Negative | FN | TN |
3.4. Reliability
Test-retest was used to measure the temporal reliability of the questionnaire among 30 outpatients of family physicians in the PHC at a one-week interval. The questionnaires were given to outpatients, and they were instructed how to complete them. There was no time limit for answering the questions. The questionnaires were not gathered until their announcement. They would mark the best option describing them and could ask the examiner any questions. The Cronbach Alpha method measured the internal reliability of IRA-PISI among 30 outpatients presenting to family physicians in PHC.
3.5. Ethics Approval
This study was extracted from the thesis of Dr. Mohsen Mostofi, a psychiatry student, and approved by the Ethics Committee of Tabriz University of Medical Sciences (IR.TBZMED.REC.1397.186). Written informed consent was obtained before the interviews.
3.6. Statistical Analysis
We used SPSS software (version 19.0, Chicago, IL, USA) for data analysis. The Kolmogorov-Smirnov test assessed the data normality. The ROC curve determined the best cutoff point. Screening data are presented in a two-way table in which columns show the number of positive and negative cases by the gold standard method (interviews) and rows show the number of positive and negative cases through questionnaires. Conditional cells were used to calculate the sensitivity and specificity of the questionnaire. Interclass correlation coefficient (ICC) and Cronbach's alpha coefficient were used to measure the temporal and internal reliability of the questions/items among 30 outpatients at a one-week interval, respectively.
4. Results
4.1. Demographic Characteristics of the Participants
Table 2 indicates some demographic characteristics of outpatients who participated in the psychometric assessment of the IRA-PISI. The mean age of the 257 participants was 37.48 ± 13.84 (range: 18 - 75) years. Besides, 53% and 68.09% of the participants were females and married, respectively.
Demographic Characteristics of Participants in the Psychometric Assessment of the Iranian Rapid Assessment for Psychiatric Illness Screening Instrument
| Variables | No. (%) |
|---|---|
| Gender | |
| Female | 137 (53.3) |
| Male | 120 (46.69) |
| Age (37.48 ± 13.84; range: 18 - 75 years) | |
| 18 - 29 (youth) | 94 (36.57) |
| 30 - 59 (adult) | 154 (59.92) |
| ≥ 60 (elderly) | 9 (3.51) |
| Marital status | |
| Single | 74 (28.8) |
| Married | 175 (68.09) |
| Widow and divorced | 8 (3.11) |
4.2. Validity and Reliability Properties
Table 3 showed that the content validity and reliability of the questionnaire. The initial questionnaire consisted of 21 questions. Seven questions were removed during the validation process by expert opinions (CVI was lower than 0.75). According to the expert opinions, finally, a 14-item screening questionnaire was developed. The minimum and maximum scores for the questionnaire were 14 and 70, respectively. Table 3 indicates the content validity and reliability indices of the questionnaire among 12 psychiatric experts who had high work and research experience. The overall relevance of the questionnaire was established (CVI = 0.91) among 12 experts. Moreover, according to the expert opinions, all items had relevancy over 0.83.
The reliability of the tool was determined in a sample of 30 outpatients presenting to family physicians who responded in two steps at a one-week interval. The ICC was higher than 0.89 in all items. The ICC values of the items/questions ranged from 89.3% to 100%. Furthermore, the overall ICC and Cronbach's alpha were 96% and 83%, respectively (Table 3).
Comparison of Items, Overall Relevancy, and Reliability of Developed Tool (14 items) for Screening Common Psychiatric Disorders in Primary Health Care Settings
| No. Question/Item | Content Validity (Relevancy)a | Reliability b, c | |
|---|---|---|---|
| Number of Observed Agreements Among 12 Experts | Item Relevancy (%) | Interclass Correlation Coefficient | |
| 1. Have you had any problems at work recently? | 12 | 100 | 0.987 |
| 2. Have you had communication problems with friends or colleagues or family members recently? | 12 | 100 | 0.986 |
| 3. Have you had depressed feelings or poor moods over the past two weeks? | 11 | 91 | 0.974 |
| 4. Have you felt a decrease in life enjoyment or futility over the past two weeks? | 10 | 83 | 0.97 |
| 5. Have you felt low energy, sluggish or tired too quickly over the past two weeks? | 11 | 91 | 0.959 |
| 6. Have you felt anxious and apprehensive about various issues during the last six months? | 12 | 100 | 0.781 |
| 7. Do you feel you have to repeat some works or things over and over again? | 12 | 100 | 0.976 |
| 8. Do you have any annoying thoughts that are difficult for you to get rid of? | 11 | 91 | 0.957 |
| 9. Have you experienced the weirdness or ambiguity of the people or environment around you? | 10 | 83 | 0.893 |
| 10. Do you have beliefs that seem strange or unacceptable to those around you? | 11 | 91 | 0.955 |
| 11. Have you wished you had not been born or not been alive over the past few weeks? | 12 | 100 | 1.00 |
| 12. Have there been times in your life when you felt very energetic or overjoyed? | 12 | 100 | 0.959 |
| 13. Have there been times in your life when you could not control your anger? | 12 | 100 | 0.984 |
| 14. Are there times in your life when you feel your body needs sleep much less than ever? | 11 | 91 | 0.954 |
Figure 1 shows the ROC of the developed psychiatric screening tool and the gold standard. Based on the findings, a score of 28 was the best cutoff point where it had the highest sensitivity and specificity. The lowest and highest questionnaire scores after screening 257 outpatients were 14 and 70, respectively.
Receiver operating characteristic curve of the psychiatric screening tool and the gold standard.
After determining the best cutoff point, the screening outcome was categorized as problematic and healthy. The results showed that the sensitivity and specificity of the questionnaire compared to the gold standard were 83.62% and 75.17%, respectively (Table 4).
| Questionnaire | Gold Standard (DSM) | |
|---|---|---|
| Positive | Negative | |
| Positive | 97 | 35 |
| Negative | 19 | 106 |
5. Discussion
This study determined the psychometric properties of the IRA-PISI in PHC settings. Due to the absence of a valid and localized rapid assessment tool for screening common PDs in first-line PHC settings, this questionnaire can be a concise, easy-to-administer screening tool to promote the specificity of early detection among first-line healthcare providers, especially family physicians. Furthermore, early detection and treatment of PDs by family physicians and first-line healthcare providers have many challenges due to various types of PDs, poor education and information about mental health among outpatients, face-to-face interviews, and being subjective features feathers of PDs (13, 14).
Diagnoses are sometimes missed given the time demands of clinical practice. Screening tools and scales have been suggested to improve accurate diagnosis. A problem with most study efforts on screening scales is the confusion between diagnostic instruments, tests, and screening. From a clinical perspective, it is most important that a diagnostic tool has high and acceptable sensitivity so that most cases are detected since more time-intensive/expensive follow-up diagnostic inquiries will presumably only occur in patients who are positive on the initial screening. Physicians and healthcare providers vary in how they analyze their data in identifying mental disorders. The present instrument could be used for screening, early detection, and minimizing missed cases (15, 16). Family physicians frequently diagnose and treat PDs, particularly in patients enrolled in care plans. One study revealed that 25% to 30% of patients presenting to primary care physicians have PDs (17). Investigations showed that 30 to 80% of primary care patients are undetected by primary care physicians (18). To avoid missing a psychiatric diagnosis, family physicians should use a systematic approach when assessing a patient for PDs, such as a valid tool with high sensitivity (19).
The current instrument aids first-line healthcare staff to provide an appropriate framework and path to screen and recognize common psychiatric disorders such as anxiety and mood disorders in primary care patients in PHC or any medical setting. The initial diagnosis of PDs by general medical doctors and other healthcare providers remains uncertain, and a significant proportion of psychiatric patients are undiagnosed or misdiagnosed.
In Iran, there are various psychometric studies for the validation of mental disorders, including the Templer Death Anxiety Scale-Extended (20), the Persian Version of Type D Personality Scale (DS14) (21), The Young Early Maladaptive Schema Questionnaire (YEMSQ) (22), among others (23-25). However, the findings revealed that the screening data may be more readily integrated when specific measures are required and low additional computation on part of the clinician is needed.
It is estimated that PHC physicians did not routinely identify patients with PDs since it is needed to specific-field. Furthermore, algorithm-based diagnoses are more complex than tools based on a simple sum of scores (26). The findings indicated that PHC providers have a shortage in the identification of PDs due to reasons including physical complaints as the major objective of PHC visits, psychiatric stigma, and time constraints (27). Therefore, rapid screening tools with the simple sum of scores can go a long way toward solving these problems (10).
Content validity must be ensured when developing a tool. On the other hand, content validity is a prerequisite for other validities and has a close relationship with reliability. Therefore, evaluating the content validity is very important in designing questionnaires (28). The process of quantitative validating using the opinions of experts and participants makes questionnaires with high relevance, transparency, and comprehensiveness. In this study, the screening tool had acceptable validity and reliability for screening common PDs. The findings revealed that the overall content validity of the tool was 91%, indicating the high agreement between experts and confirming the relevance and appropriateness of the tool. The minimum acceptable value of content validity for the overall tool is 80% (29).
The sensitivity of tools, especially screening tools, is another major criterion of validity. The sensitivity index should be maximized in the first line of the healthcare system (30, 31). The sensitivity and specificity of this questionnaire compared to the gold standard (psychiatric interviews with a structured DSM instrument) were 83.62% and 75.17%, respectively (32).
Regarding the reliability of the questionnaire, internal reliability (Cronbach's alpha coefficient) and temporal reliability (ICC) were higher than 0.89 in all questions. Consequently, overall ICC and Cronbach's alpha coefficient were reported as 0.96 and 0.83, respectively. The results showed that all scales and subscales had high reliability. The internal consistency of all scales and subscales in this study was very high. This means that the items in each test or subscale were correlated with each other, as well as with all items. The highest reliability (ICC) among the questions was related to question 11 (suicide ideation), and the lowest reliability was related to question 9 (psychotic disorders).
There is limited evidence on developing and validating a screening tool in PHC (33). This study is unique as it provides family physicians in PHC settings with a rapid questionnaire. Family physicians frequently identify and treat PDs, particularly in patients enrolled in care programs. It is estimated that 30% to 80% of PDs are undetected by PHC providers (17, 18). Most outpatients with mental disorders are deprived of mental healthcare due to inappropriate case management by first-line healthcare providers, unavailability of valid, reliable, and structured tools, poor awareness of first-line healthcare providers, patient orientation to the private sector, and strong private sector (2, 34). Although most first-line healthcare providers have poor information about PDs, this questionnaire provides a rapid assessment survey for screening common PDs in a short time. On the other hand, completing the questionnaire by the interviewer or patients is possible.
5.1. Limitations
This study is a cross-sectional study that could not assess sensitivity to change over time or responsiveness to changes. Longitudinal studies and high sample size are required to assess this validity. The use of tools in different ethnicities from various cities requires a pre-test study to ensure the feasibility of using the tool in these populations.
5.2. Conclusions
Findings indicated that the Persian version of IRA-PISI (14 items) showed high validity and sensitivity for screening common PDs in the first-line healthcare system of Iran. Given that the patient load is high for family physicians, they can use this rapid and efficient questionnaire without spending much time. Investigations are needed to evaluate whether such a rapid and valid screening questionnaire is generalizable to other contexts and health systems, and how PDs screening should be best introduced into routine health care practice.
Acknowledgements
References
-
1.
Vicente B, Kohn R, Rioseco P, Saldivia S, Baker C, Torres S. Population Prevalence Of Psychiatric Disorders In Chile: 6-Month And 1-Month Rates. Br J Psychiatry. 2018;184(4):299-305. [PubMed ID: 15056573]. https://doi.org/10.1192/bjp.184.4.299.
-
2.
Kosaryan M, Rabiei K. The critical appraisal of the papers published in the "iranian journal of psychiatry and behavioral sciences", 2007-2010. Iran J Psychiatry Behav Sci. 2013;7(2):77-82. [PubMed ID: 24644513]. [PubMed Central ID: PMC3939991].
-
3.
Fatahi A, Doosti-Irani A, Cheraghi Z. Prevalence and Incidence of Metabolic Syndrome in Iran: A Systematic Review and Meta-Analysis. Int J Prev Med. 2020;11:64. [PubMed ID: 32577194]. [PubMed Central ID: PMC7297433]. https://doi.org/10.4103/ijpvm.IJPVM_489_18.
-
4.
Aminorroaya A, Fattahi N, Azadnajafabad S, Mohammadi E, Jamshidi K, Rouhifard Khalilabad M, et al. Burden Of Non-Communicable Diseases In Iran: Past, Present, And Future. J Diabetes Metab Disord. 2020:1-7. https://doi.org/10.1007/s40200-020-00669-z.
-
5.
Ramirez Basco M, Bostic JQ, Davies D, Rush AJ, Witte B, Hendrickse W, et al. Methods to improve diagnostic accuracy in a community mental health setting. Am J Psychiatry. 2000;157(10):1599-605. [PubMed ID: 11007713]. https://doi.org/10.1176/appi.ajp.157.10.1599.
-
6.
Hudson JL, Rapee RM. Parent-child interactions and anxiety disorders: an observational study. Behav Res Ther. 2001;39(12):1411-27. [PubMed ID: 11758699]. https://doi.org/10.1016/s0005-7967(00)00107-8.
-
7.
Mulvaney-Day N, Marshall T, Downey Piscopo K, Korsen N, Lynch S, Karnell LH, et al. Screening for Behavioral Health Conditions in Primary Care Settings: A Systematic Review of the Literature. J Gen Intern Med. 2017;33(3):335-46. [PubMed ID: 28948432]. [PubMed Central ID: PMC5834951]. https://doi.org/10.1007/s11606-017-4181-0.
-
8.
Liegl G, Wahl I, Berghöfer A, Nolte S, Pieh C, Rose M, et al. Using Patient Health Questionnaire-9 item parameters of a common metric resulted in similar depression scores compared to independent item response theory model reestimation. J Clin Epidemiol. 2016;71:25-34. [PubMed ID: 26475569]. https://doi.org/10.1016/j.jclinepi.2015.10.006.
-
9.
McQuaid JR, Stein MB, McCahill M, Laffaye C, Ramel W. Use of brief psychiatric screening measures in a primary care sample. Depress Anxiety. 2000;12(1):21-9. [PubMed ID: 10999242]. https://doi.org/10.1002/1520-6394(2000)12:1<21::aid-da3>3.0.co;2-u.
-
10.
Bhana A, Mntambo N, Gigaba SG, Luvuno ZPB, Grant M, Ackerman D, et al. Validation of a brief mental health screening tool for common mental disorders in primary healthcare. S Afr Med J. 2019;109(4):278-83. [PubMed ID: 31084695]. [PubMed Central ID: PMC7029170]. https://doi.org/10.7196/SAMJ.2019.v109i4.13664.
-
11.
Bernstein GA, Layne AE, Egan EA, Nelson LP. Maternal phobic anxiety and child anxiety. J. Anxiety Disord. 2005;19(6):658-72. [PubMed ID: 15927779]. https://doi.org/10.1016/j.janxdis.2004.09.001.
-
12.
Schulberg HC, Bruce ML, Lee PW, Williams JW, Dietrich AJ. Preventing suicide in primary care patients: The primary care physician's role. Gen Hosp Psychiatry. 2004;26(5):337-45. [PubMed ID: 15474633]. https://doi.org/10.1016/j.genhosppsych.2004.06.007.
-
13.
Azizi H, Davtalab-Esmaeili E, Farahbakhsh M, Zeinolabedini M, Mirzaei Y, Mirzapour M. Malaria situation in a clear area of Iran: an approach for the better understanding of the health service providers' readiness and challenges for malaria elimination in clear areas. Malar J. 2020;19(1):114. [PubMed ID: 32188469]. [PubMed Central ID: PMC7079540]. https://doi.org/10.1186/s12936-020-03188-7.
-
14.
van Rijswijk E, van Hout H, van de Lisdonk E, Zitman F, van Weel C. Barriers in recognising, diagnosing and managing depressive and anxiety disorders as experienced by Family Physicians; a focus group study. BMC Fam Pract. 2009;10:52. [PubMed ID: 19619278]. [PubMed Central ID: PMC2734533]. https://doi.org/10.1186/1471-2296-10-52.
-
15.
Zimmerman M, Holst CG. Screening For Psychiatric Disorders With Self-Administered Questionnaires. Psychiatr Res. 2018;270:1068-73. [PubMed ID: 29908784]. https://doi.org/10.1016/j.psychres.2018.05.022.
-
16.
Zimmerman M, Mattia JI. A self-report scale to help make psychiatric diagnoses: the Psychiatric Diagnostic Screening Questionnaire. Arch Gen Psychiatry. 2001;58(8):787-94. [PubMed ID: 11483146]. https://doi.org/10.1001/archpsyc.58.8.787.
-
17.
Schulberg HC, Burns BJ. Mental Disorders In Primary Care: Epidemiologic, Diagnostic, And Treatment Research Directions. Gen Hosp Psychiatry. 1988;10(2):79-87. [PubMed ID: 3282988]. https://doi.org/10.1016/0163-8343(88)90092-8.
-
18.
Carlat DJ. The psychiatric review of symptoms: a screening tool for family physicians. Am Fam Physician. 1998;58(7):1617-24. [PubMed ID: 9824959].
-
19.
Spitzer RL, Williams JB, Kroenke K, Linzer M, deGruy FV, Hahn SR, et al. Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study. JAMA. 1994;272(22):1749-56. [PubMed ID: 7966923]. https://doi.org/10.1001/jama.1994.03520220043029.
-
20.
Sharif Nia H, Ebadi A, Lehto RH, Mousavi B, Peyrovi H, Chan YH. Reliability and validity of the persian version of templer death anxiety scale-extended in veterans of iran-iraq warfare. Iran J Psychiatry Behav Sci. 2014;8(4):29-37. [PubMed ID: 25798171]. [PubMed Central ID: PMC4364474].
-
21.
Bagherian R, Bahrami Ehsan H. Psychometric Properties of the Persian Version of Type D Personality Scale (DS14). Iran J Psychiatry Behav Sci. 2011;5(2):12-7. [PubMed ID: 24644442]. [PubMed Central ID: PMC3939967].
-
22.
Yousefi N, Shirbagi N. Validating The Young Early Maladaptive Schema Questionnaire (Yemsq) Among Students. Iran J Psychiatry Behav Sci. 2010;4(1):38-46.
-
23.
Dadfar M, Kalibatseva Z. Properties of the Persian Version of the Short Beck Depression Inventory with Iranian Psychiatric Outpatients. Scientifica (Cairo). 2016;2016:8196463. [PubMed ID: 27293979]. [PubMed Central ID: PMC4886104]. https://doi.org/10.1155/2016/8196463.
-
24.
Shirinbayan P, Salavati M, Soleimani F, Saeedi A, Asghari-Jafarabadi M, Hemmati-Garakani S, et al. The Psychometric Properties of the Drug Abuse Screening Test. Addict Health. 2020;12(1):25-33. [PubMed ID: 32582412]. [PubMed Central ID: PMC7291903]. https://doi.org/10.22122/ahj.v12i1.256.
-
25.
Namjoo S, Shaghaghi A, Sarbaksh P, Allahverdipour H, Pakpour AH. Psychometric Properties Of The General Health Questionnaire (GHQ-12) To Be Applied For The Iranian Elder Population. Aging Ment Health. 2016;21(10):1047-51. [PubMed ID: 27333198]. https://doi.org/10.1080/13607863.2016.1196337.
-
26.
Gilbody S, Richards D, Barkham M. Diagnosing Depression In Primary Care Using Self-Completed Instruments: UK Validation Of PHQ-9 And CORE-OM. Br J Gen Pract. 2007;57(541):650-2. [PubMed ID: 17688760]. [PubMed Central ID: PMC2099671].
-
27.
Thielke S, Vannoy S, Unutzer J. Integrating Mental Health And Primary Care. Prim Care. 2007;34(3):571-92. [PubMed ID: 17868760]. https://doi.org/10.1016/j.pop.2007.05.007.
-
28.
Garin O, Ayuso-Mateos JL, Almansa J, Nieto M, Chatterji S, Vilagut G, et al. Validation of the "World Health Organization Disability Assessment Schedule, WHODAS-2" in patients with chronic diseases. Health Qual Life Outcomes. 2010;8:51. [PubMed ID: 20482853]. [PubMed Central ID: PMC2893517]. https://doi.org/10.1186/1477-7525-8-51.
-
29.
Schutz AL, Counte MA, Meurer S. Development Of A Patient Safety Culture Measurement Tool For Ambulatory Health Care Settings: Analysis Of Content Validity. Health Care Manag Sci. 2007;10(2):139-49. [PubMed ID: 17608055]. https://doi.org/10.1007/s10729-007-9014-y.
-
30.
Rashidian A, Kavosi Z, Majdzadeh R, Pourreza A, Pourmalek F, Arab M, et al. Assessing Health System Responsiveness: A Household Survey In 17th District Of Tehran. Iran Red Crescent Med J. 2011;13(5):302-8. [PubMed ID: 22737485]. [PubMed Central ID: PMC3371967].
-
31.
Azizi H. [Indicators for Assessing the Practice and Capacity of COVID-19 Surveillance System]. J Mar Med. 2020;2(1):53-6. Persian. https://doi.org/10.30491/2.1.10.
-
32.
Farahbakhsh M, Fakhari A, Azizi H, Davtalab-Esmaeili E. [Structure, Characteristics and Components of COVID-19 Surveillance System]. J Mil Med. 2020;22(6):534-41. Persian. https://doi.org/10.30491/jmm.22.6.534.
-
33.
El-Den S, Chen TF, Gan YL, Wong E OC. The psychometric properties of depression screening tools in primary healthcare settings: A systematic review. J Affect Disord. 2018;225:503-22. [PubMed ID: 28866295]. https://doi.org/10.1016/j.jad.2017.08.060.
-
34.
Ahmadi J, Ahmadi N, Soltani F, Bayat F. Gender differences in depression scores of Iranian and german medical students. Iran J Psychiatry Behav Sci. 2014;8(4):70-3. [PubMed ID: 25798177]. [PubMed Central ID: PMC4364480].