1. Background
In December 2019, an outbreak associated with a novel, highly contagious coronavirus emerged in Wuhan, Hubei Province, China. It was subsequently labeled as coronavirus disease 2019 (COVID-19) (1). The virus spread created a pandemic, with a mean mortality rate of 3.4% calculated based on the data of 135 nations (2). The current COVID-19 pandemic compares to other pandemics, such as the 2003 severe acute respiratory syndrom (SARS) epidemic and the 2009 H1N1 pandemic. These earlier viruses were similar in transmissibility, hospitalization, severe acute respiratory distress, and mortality (3-5). Different countries around the globe applied restrictions, such as social distancing and lockdown (6), which could cause additional stress (7).
Public and public health officials, clinicians, and researchers have been highly concerned about the mortality rate and effect of the virus on patients with chronic diseases (8). In addition to physical well-being, the COVID-19 pandemic affects behavioral and mental health (9). Restrictions make people feel isolated, lonely, stressed, anxious, and helpless. Individuals may have deep concerns about being in contact with someone who has COVID-19 (2, 10). Moreover, becoming ill or dying (11, 12) and losing a loved one results in stress due to grief and bereavement (13).
It has been observed that stress related to Covid has become widespread during the pandemic (14, 15). High levels of stress during pandemics can predict negative behaviors and psychological problems, such as panic buying and anxiety over mild non-COVID symptoms (16-18). More realistically, low to moderate COVID stress levels may positively influence social and protective behaviors that may lower the infection rate (7, 19). There have been few validated COVID-related stress measures. Ahorsu et al. (10) recently developed the "Fear of COVID-19 Scale" in Iran. However, we are unaware of validity and reliability of this measure. Therefore, this study aimed to develop and initially assess the validity and reliability of a new scale, the COVID-19 Stress Scale (CSS), using data from the Iranian non-clinical population.
COVID-19 and its consequences have caused stress and psychological injuries (15). Stressful conditions, such as epidemics, can increase stress even in healthy people. Improving symptoms in people with mental disorders can lead to various psychological consequences (20). In addition to benefits, such as survival and motivation, stress has potentially detrimental effects and can negatively impact various dimensions, including sleep, learning, endocrine system, cardiovascular health, digestion, and memory (21). COVID-19-related stress is caused by various factors, such as dangerous and contagious diseases, unpredictable conditions, and uncertainty about treatment (22). Furthermore, fear of spreading the disease to others, as well as the social and economic consequences of COVID-19, are other factors that influence the stress caused by this disease. A study conducted in 18 countries in North Africa and the Middle East showed that more than half of the participants experienced the stress of COVID-19 because of domestic issues, and more than one-third of participants experienced COVID-19 stress related to work and financial issues (23). Moreover, the decrease in sports activities in almost half of the participants due to the quarantine of the epidemic was another negative impact of COVID-19 stress. COVID stress and its consequences, including quarantine, social distancing, and financial problems, led to a decline in school and social activities and even changes in eating habits and lifestyle. This stress has also resulted in diminished physical activity, weight gain, and an increased risk of heart disease (24). COVID-19 stress causes people to feel guilty because they worry about getting sick. The death of relatives and the inability to cope with stress leads people to resort to unscientific and unreliable medications and methods. This is harmful to them and worsens the situation (22). Therefore, to provide a suitable platform for research, assessment, and evaluation, it is necessary to develop scales to study the effects of this condition (17).
Information about this epidemic can be more accurately measured, assessed, and evaluated with reliable tools, and appropriate decisions can be made in various areas based on this information and accurate measurements. For this purpose, a valid, accurate, and appropriate psychometric instrument is needed. Considering that Iran is one of the countries with the highest rates of COVID-19 prevalence and death, we attempted to establish a valid CSS.
2. Objectives
This study aimed to assess the validity and reliability of CSS as a new scale using non-clinical Iranian data.
3. Methods
Data were collected between 10 March 2020 and 19 April 2020. Our study design included several survey methodology strategies. We first generated items for the CSS to develop and validate our study after reviewing the literature. Next, an expert panel assessed the items, and items were selected based on the expert panel feedback. The preliminary scale, designed based on the previous step, was then evaluated in a pilot sample. In the next step and after fielding a larger sample, the scale's construct validity was evaluated using exploratory and confirmatory factor analyses. Consistent internal reliability was determined based on Cronbach's alpha. Furthermore, the test-retest reliability was investigated. Convergent validity was evaluated by examining correlations between the CSS and various subscales of the Depression, Anxiety, and Stress Scale 21 (DASS-21) (25).
3.1. Item Generation and Content Validity Assessment
The principal investigator and a co-investigator reviewed the stress literature with a particular focus on the scales and other measures of stress. Afterwards, we generated items based on the knowledge from our review. The items were close-ended declarative statements written in simple language geared towards a basic reading level. Using the process described in the methods section, we developed 18 items for review by ten expert panel members. Panelists were professors and researchers from health education, prevention, psychology, psychiatry, and stress. All had an experience of psychosocial research on the Iranian population. We asked whether each item was unique and related to the research aims. In addition, we requested feedback about the readability of each item (whether clear and at a below diploma educational level). After processing the feedback of experts, we retained seven out of the original 18 items. Each item had a 1 - 5 response set as described above. Therefore, the possible total score was 7 - 35. An expert panel assessed content validity, and the panel members reviewed the items and provided the requested feedback.
3.2. Pilot Test Administration and Feedback
The emergent CSS was tested in a convenience sample. The samples were also invited to provide feedback in a telephone interview. The survey was completed by a mostly female pilot convenience sample (N = 45) with a mean age of 38 years, most of whom had a diploma or above (46.6%). Follow-up telephone interviews revealed that all the pilot participants found the items clear and understandable.
3.3. Tools
3.3.1. CSS
It resulted from the expert panel review. Each item's response set had a range of 1 - 5 (strongly disagree = 1, disagree = 2, neither agree nor disagree = 3, agree = 4, strongly agree = 5).
3.3.2. DASS-21
It comprises three subscales: stress, anxiety, and depression (25). Each item is scored on a 4-point Likert scale with 0 = “Did not apply to me at all–never”, 1 = “Applied to me to some degree, or some of the time–sometimes, 2 = “Applied to me to a considerable degree, or a good part of the time–often”, “3 = Applied to me very much, or most of the time-almost always.” The potential range of the sum of subscales is 0 - 21. The internal consistency reliability of DASS-21 was 0.77 (26) and 0.82 for an Iranian sample (27).
3.4. Setting, Participants, and Data Collection
The online method was used to collect information in this study. This method was chosen due to traffic restrictions when the risk of COVID-19 in Iran was grave. The scale was designed online, and its link was published as invitations on virtual social networks, including Instagram, Telegram, WhatsApp, university E-mail list, and SMS list. People who wanted to participate in the study were asked to answer the scale questions and send their answers to the researcher. These invitations entailed a link to CSS and all elements of informed consent. The inclusion criteria were Iranian citizenship, residence in Iran, access to one of the above platforms, and proficiency in Persian. Participants were required to complete all survey items before submitting the questionnaire.
At the beginning of the questionnaire, participants were given the necessary information about the voluntary nature and the fact that they were not forced to answer the research instruments, they should not give their name and key data, the answers would be kept confidential, how they should answer the questions, they should answer honestly, and they should contribute to valid research. Individuals who wished to participate in this study responded to the research tool. The duration of online data collection was approximately 2 months. The statistical population of this study included all people aged 18 - 60 years old and participated in this study online. In addition, people from different age groups and social classes could participate in the study. All 755 individuals (511 females and 244 males) who completed the questionnaire online were selected as samples. The individual responses were automatically stored in the database after completion. Ultimately, all data were collected and analyzed using appropriate statistical methods. Social demographic data were also collected.
3.5. Data Analysis
Descriptive statistics were generated. The data met factor analysis assumptions. We randomly split the samples into two groups exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). We conducted EFAs using IBM SPSS Statistics version 24.0 (IBM, Inc., Armonk, USA). The results of EFA confirmed the hypothesis of the CFA model, which was evaluated using the AMOS 26 software (IBM® SPSS® Amos TM26). We used a maximum likelihood approach to estimate the parameters. Six indices were used to assess the goodness of fit of the resulting factor structure. The indices included root mean square error of approximation (RMSEA; criterion ≤ 0.05), the goodness of fit index (GFI, criterion > 0.08), relative fit index (RFI; criterion > 0.99), normed fit index (NFI; criterion > 0.99), and incremental fit index (IFI criterion ≥ 0.9) (28).
3.6. Ethical Considerations
All the procedures conducted in this study were consistent with the National Research Committee ethical standards in Iran, the Helsinki Declaration of 1964 (29), subsequent revisions, or equivalent ethical norms. As written informed consent elements were incorporated into the online invitation, participants provided consent when they returned the survey.
4. Results
A total of 755 Iranians completed the survey. They were mainly women (69%), over half were married, and over 60% held a bachelor's or a graduate degree as their highest education level. We observed that 11.8% had tested positive for COVID-19. We randomly divided the participants into an EFA or a CFA sample, as depicted in Table 1. There were no statistically significant sociodemographic differences between the EFA and CFA participants.
Table 1.
Sociodemographic Characteristics of EFA and CFA Participants
| Total (N = 755) | EFA (N = 304) | CFA (N = 451) | |||
|---|---|---|---|---|---|
| Male | Female | Male | Female | ||
| Marital status | |||||
| Single | 366 (48.5) | 37 (12.17) | 135 (44.4) | 46 (10.19) | 148 (32.81) |
| Married | 389 (51.5) | 56 (18.42) | 76 (25) | 105 (23.28) | 152 (33.7) |
| Educational status | |||||
| High school | 36 (4.8) | 4 (1.31) | 9 (2.96) | 7 (1.55) | 16 (3.54) |
| Diploma | 182 (24.1) | 9 (2.96) | 78 (25.65) | 26 (5.76) | 69 (15.29) |
| Associate degree | 47 (6.2) | 12 (3.94) | 13 (4.27) | 7 (1.55) | 15 (3.32) |
| Bachelor’s degree | 259 (34.3) | 34 (11.18) | 73 (24.01) | 51 (11.3) | 101 (22.39) |
| Higher than bachelor’s degree | 231 (30.6) | 34 (11.18) | 38 (12.5) | 60 (13.3) | 99 (21.95) |
| COVID-19 status | |||||
| No | 666 (88.2) | 83 (27.3) | 177 (58.22 | 138 (30.59) | 268 (59.42) |
| Yes | 89 (11.8) | 10 (3.28) | 34 (11.18) | 13 (2.88) | 32 (7.09) |
Sample size sufficiency was assessed by calculating a Kaiser-Meyer-Olkin (KMO) and Bartlett's test of sphericity. The statistics indicated that the sample size was adequate [KMO = 0.89 (criterion > 0.7), and Bartlett's test was significant at P < 0.05 (28, 30, 31). Table 2 displays the items and descriptive statistics. Once we determined that the data met factor analysis assumptions, we continued the analytic plan.
Table 2.
Descriptive Statistics for All the CSS Items
| Items | Mean ± SD | Median | Min. | Max. |
|---|---|---|---|---|
| 1. I am overreacting to situations since the coronavirus outbreak | 2.16 ± 1.46 | 2 | 0 | 5 |
| 2. Due to thoughts about coronavirus, my appetite has significantly changed (increased or decreased) | 0.98 ± 1.44 | 0 | 0 | 5 |
| 3. I cannot sleep well due to worries about coronavirus | 0.86 ± 1.39 | 0 | 0 | 5 |
| 4. I feel that my accuracy in tasks has decreased due to thinking about COVID-19 | 1.06 ± 1.45 | 0 | 0 | 5 |
| 5. I cannot relax easily because of COVID-19 | 1.27 ± 1.47 | 0 | 0 | 5 |
| 6. Since the COVID-19 outbreak, I have been feeling tired sooner than usual | 1.35 ± 1.57 | 0 | 0 | 5 |
| 7. I get distracted by seeing and hearing the news of COVID-19 | 1.28 ± 1.58 | 0 | 0 | 5 |
| Total items | 8.96 ± 8.26 | 6 | 0 | 35 |
Table 3 displays the relationships between the sociodemographic characteristics and the total CSS scores. The only statistically significant variable was gender, with women scoring higher than men.
Table 3.
Association Between the Total CSS Score and Sociodemographic Characteristics (N = 755)
| No. (%) | Mean ± SD | t | P-Value | |
|---|---|---|---|---|
| Marital status | 0.11 | 0.9 | ||
| Married | 389 (51.52) | 9 ± 8.37 | ||
| Single | 366 (48.47) | 8.93 ± 8.14 | ||
| Gender | -3.09 | 0.002 a | ||
| Male | 244 (32.31) | 7.62 ± 7.19 | ||
| Female | 511 (67.68) | 9.61 ± 8.6 | ||
| COVID-19 status | -0.07 | 0.94 | ||
| Yes | 89 (11.78) | 8.91 ± 8.28 | ||
| No | 666 (88.21) | 8.97 ± 8.26 | ||
| Educational status | F = 0.56 | 0.69 | ||
| High school education | 36 (4.76) | 7.69 ± 7.19 | ||
| Diploma graduation | 182 (24.1) | 9.03 ± 7.98 | ||
| Associate degree | 47 (6.22) | 8.97 ± 7.8 | ||
| Bachelor’s degree | 259 (34.3) | 8.62 ± 8.09 | ||
| Higher than a bachelor’s degree | 231 (30.59) | 9.49 ± 8.91 |
a Significance at 5% or lower level.
4.1. EFA and CFA
We assessed the scale's underlying factor structure by first conducting EFAs using IBM SPSS 24.0 software (IBM Inc., Armonk, USA). We used primary factor analysis with a varimax rotation approach. Our EFA sample comprised 304 participants. According to Table 4, the first EFA produced one factor, a 7-item solution. Each item's factor loading was > 0.4, and each item was loaded only on a single factor.
Table 4.
EFA of 7-item CSS (N = 304)
| Items | Factor Loading | Eigenvalue | ||
|---|---|---|---|---|
| Total | Variance % | Cumulative % | ||
| Item 1: I am overreacting to situations since the coronavirus outbreak | 0.56 | 4.41 | 63 | 63 |
| Item 2: Due to thoughts about coronavirus, my appetite has significantly changed (increased or decreased) | 0.53 | 0.692 | 9.88 | 72.88 |
| Item 3: I cannot sleep well due to worries about coronavirus | 0.68 | 0.572 | 8.16 | 81.05 |
| Item 4: I feel that my accuracy in tasks has decreased due to thinking about COVID-19 | 0.67 | 0.502 | 7.17 | 88.23 |
| Item 5: I cannot relax easily because of COVID-19 | 0.73 | 0.347 | 4.96 | 93.19 |
| Item 6: Since the COVID-19 outbreak, I have been feeling tired sooner than usual | 0.75 | 0.239 | 3.41 | 96.6 |
| Item 7: I get distracted by seeing and hearing the news of COVID-19 | 0.66 | 0.237 | 3.39 | 100 |
These EFA findings provided the hypothesized model for the first CFA. The one factor, a 7-item solution, resulted from the first CFA rotation (Table 5).
Table 5.
CFA of 7-item CSS (N = 451)
| Items | Standardized Factor Loading | Eigenvalue | |||
|---|---|---|---|---|---|
| P-Value | Total | Variance % | Cumulative % of the Variance | ||
| Item 1: I am overreacting to situations since the coronavirus outbreak | 0.64 | 0.001 | 4.41 | 63 | 63 |
| Item 2: Due to thoughts about coronavirus, my appetite has significantly changed (increased or decreased) | 0.68 | 0.001 | 0.692 | 9.88 | 72.88 |
| Item 3: I cannot sleep well due to worries about coronavirus | 0.79 | 0.001 | 0.572 | 8.16 | 81.05 |
| Item 4: I feel that my accuracy in tasks has decreased due to thinking about COVID-19 | 0.81 | 0.001 | 0.502 | 7.17 | 88.23 |
| Item 5: I cannot feel relaxed easily because of COVID-19 | 0.85 | 0.001 | 0.347 | 4.96 | 93.19 |
| Item 6: Since the COVID-19 outbreak, I have been feeling tired sooner than usual | 0.75 | 0.001 | 0.239 | 3.41 | 96.6 |
| Item 7: I get distracted by seeing and hearing the news of COVID-19 | 0.73 | 0.001 | 0.237 | 3.39 | 100 |
The 7-item scale was evaluated using criterion fit statistics, all of which were met (Table 6). This provided assurance that construct validity was established.
Table 6.
Model Fit Indices of the 7-item 1-factor CSS
| Fit Index | Obtained Value | Recommended Value |
|---|---|---|
| Root mean square error of approximation (RMSEA) | 0.05 | < 0.08 |
| The goodness of fit index (GFI) | 1 | > 0.9 |
| Normed fit index (NFI) | 0.99 | > 0.9 |
| Incremental fit index (IFI) | 1 | > 0.9 |
| Relative fit index (RFI) | 0.99 | > 0.9 |
| The goodness of fit index (GFI) | 0.96 | > 0.9 |
4.2. Convergent Validity
We examined Pearson’s correlations between the CSS and DASS-21 subscales to investigate a theoretical relationship between the two scales. We hypothesized a moderate to strong correlation of 0.4 - 0.79 (32). A theoretical relationship between the DASS-21 subscales was hypothesized. Descriptive statistics for DASS-21 are found in Table 7. Table 8 summarizes the correlations between the CSS and DASS-21 subscales. Correlations had a range of 0.53 - 0.73 and were statistically significant at P < 0.01. Consequently, our criterion for moderate to strong relationships was met.
Table 7.
Descriptive Statistic for DASS-21 Subscales (N = 451)
| DASS-21 | Mean ± SD | Potential Range | Actual Range | Skew | Kurtosis |
|---|---|---|---|---|---|
| DASS-depression | 5.36 ± 4.54 | 0 - 21 | 0 - 21 | 0.84 | 0.25 |
| DASS-stress | 6.74 ± 4.51 | 0 - 21 | 0 - 20 | 0.49 | -0.46 |
| DASS-anxiety | 4.61 ± 3.82 | 0 - 21 | 0 - 18 | 0.83 | 0.15 |
| DASS-Total Scale | 16.71 ± 9.46 | 0 - 63 | 6 - 57 | 0.71 | 0.09 |
Table 8.
Pearson’s Correlation of Satisfaction with CSS and DASS-21 Components Among Participants (CFA Sample, N = 451) a
| 1 | 2 | 3 | 4 | 5 | |
|---|---|---|---|---|---|
| 1. CSS | 1 | ||||
| 2. DASS-21, depression | 0.53** | 1 | |||
| 3. DASS-21, stress | 0.64** | 0.42** | 1 | ||
| 4. DASS-21, anxiety | 0.59** | 0.56** | 0.69* | 1 | |
| 5. DASS-21, total | 0.73** | 0.82** | 0.71** | 0.69** | 1 |
a P < 0.05, **P < 0.01
3.3. Internal Consistency Reliability and Temporal Stability
Cronbach's alpha was calculated to assess internal consistency reliability (criterion > 0.7) (33). Using Pearson's correlation, we evaluated temporal stability reliability by a test-retest strategy in a small sub-sample. We hypothesized a moderate to strong correlation of 0.4-0.79 (32). Internal consistency reliability was assessed using Cronbach's alpha. The entire scale alpha was 0.903, exceeding the criterion > 0.7. Alpha-if-item removed statistics, indicating that the alpha was best if the seven items remained (Table 9).
Table 9.
Reliability Indices (7-item Version)
| Items | Item to Total Correlations | Alpha If Item Deleted | Total Scale Alpha |
|---|---|---|---|
| Item 1: I am overreacting to situations since the coronavirus outbreak | 0.601 | 0.901 | 0.903 |
| Item 2: Due to thoughts about coronavirus, my appetite has significantly changed (increased or decreased) | 0.644 | 0.896 | |
| Item 3: I cannot sleep well due to worries about coronavirus | 0.726 | 0.888 | |
| Item 4: I feel that my accuracy in tasks has decreased due to thinking about COVID-19 | 0.749 | 0.885 | |
| Item 5: I cannot relax easily because of COVID-19 | 0.784 | 0.881 | |
| Item 6: Since the COVID-19 outbreak, I have been feeling tired sooner than usual | 0.756 | 0.884 | |
| Item 7: I get distracted by seeing and hearing the news of COVID-19 | 0.738 | 0.886 |
Temporal stability was assessed using a test-retest strategy in a small sub-sample of 45 participants. This sample took the CSS in two sessions 14 days apart. The correlation between the total scores of each session was 0.80 (CI = 0.79 - 0.83). Therefore, our criterion for a strong relationship was met.
5. Discussion
This study aimed to develop and psychometrically assess CSS as a new scale to measure the stress response of individuals to the COVID-19 pandemic among the Iranian non-clinical population. An expert panel process resulted in a 7-item scale. Pilot test participants found the scale to be clear and understandable. The scale was assessed for its factor structure. Both EFA and CFA demonstrated that the scale had a one-factor structure. Construct validity was supported by fit statistics that met the criteria. The scale's convergent validity was shown in the correlations between the total CSS score and the DASS-21 subscale scores. Correlations were positive, moderate to strong, and statistically significant. Participants who scored higher on the CSS were more likely to experience depression, anxiety, and stress as measured by the DASS-21. The seven items on the CSS had internal consistency reliability. A test-retest strategy supported temporal stability. Overall, the CSS indicates strong validity and reliability, providing confidence that the scale accurately measures the stress associated with COVID-19. A review of the CSS items shows that they measure symptoms such as worrying, changing appetite, being tired, being more distractible, and poor sleep quality. These symptoms partly explain the strong correlations between CSS and the scales of DASS-21.
Women scored significantly higher than men on the CSS. This finding is consistent with other studies on gender and stress (34-38). Perceived stress is usually higher in women than men, and some psychological explanations are given, such as rumination, trauma, biology/hormones, brain, and connectivity across regions and networks (36). Perhaps in some cultures, women's expressions of stress are more socially acceptable than those of men. Moreover, as caretakers, women may be additionally troubled by family members' stress (34).
5.1. Strengths and Limitations
The strengths of the present study included multi-stage rigorous development and psychometric assessment, a full sample size, findings that display construct and convergent validity, and internal consistency and temporal reliability. One of the limitations was the convenience sample comprising only Iranian citizens. As a result, the findings are not generalizable. In addition, there were no scales available in this study that measured Iranians' negative reactions to COVID, such as the Fear of COVID-19 Scale. Therefore, statistical comparisons between scales could not be made. Item comparisons of the two scales indicate each measure of sleep quality. Other symptom measurements were different (eg, cardiac reactions and clammy hands compared to the change of appetite and accuracy in tasks). The CSS is a self-report measure, and only the individual can convey the stress they are feeling. Social acceptability bias potentially may have affected our findings (39). However, this bias is also potentially present in well-used and well-regarded measures for patient states, such as pain or anxiety (40).
5.2. Implications
Other studies concluded that disease outbreaks (eg, SARS) resulted in feelings of trauma and stress (15, 41). There is a similar problem with COVID-19. Although not widely tested, CSS could be a helpful screening tool for Iranians amid the pandemic. Once found valid and reliable in other investigations with randomized samples, it could be a standard of care to measure individual or group stress caused by COVID-19. Furthermore, researchers could use the scale as an antecedent or co-variate in outcome studies or as an outcome in intervention studies. Stress is associated with physical health, such as susceptibility to infection. Consequently, policy decision-makers, health authorities, and healthcare professionals must assess the existence and severity of adverse psychological reactions related to the coronavirus pandemic. Once psychometrically assessed in larger, randomly selected samples, the CSS may be used in a pre-and post-test design to examine the effectiveness of stress-reducing interventions.
5.3. Conclusions
The CSS developed here has high validity and reliability. Given the global impact of the pandemic on people's mental health, CSS might be a helpful tool for research. Moreover, we anticipate that the scale can be widely used after being assessed in randomized samples.
