Abstract
Background:
There is evidence of high rates of occupational burnout across many health practitioner groups. However, most measures used to assess burnout are lengthy, time consuming to administer, or difficult to interpret. Hence there is a need for the development and assessment of a short easily administered burnout measure for use in research and human resource contexts.Objectives:
To assess the psychometric performance of a single item burnout measure (SIB) and its association with salient outcome measures in a sample of general practitioners.Patients and Methods:
This cross sectional study collected survey data from 92 general practitioners practising in the Northern Rivers region of NSW, Australia. SIB scores were compared with maslach burnout inventory-emotional exhaustion (MBI-EE) scores and examined for association with early retirement intentions, psychological distress and general health. Sensitivity analyses were calculated using the MBI-EE subscale as the reference standard.Results:
SIB scores were highly and positively correlated with MBI-EE scores (r = 0.8, P < 0.0001) and significantly associated with all outcome measures. A cut-off score of 5 on the SIB yielded a Kappa of 0.70 (95% CI: 0.49 to 0.90), sensitivity of 79% (95% CI: 0.63 to 0.95), specificity of 87% (95% CI: 0.79 to 0.95), positive predictive value of 68%, and negative predictive value of 92%.Conclusions:
The results of this study indicates that the single item burnout measure has potential as a brief, sensitive screening measure of burnout due to its brevity, ease of administration and sound psychometric properties and needs to be validated further in larger studies and with other professional groups.Keywords
1. Background
While the scientific literature is rife with reports of high burnout amongst health care professionals and in particular GPs (1, 2), there is surprisingly little consensus on the conceptual definition of ‘burnout’, the exact components of it and hence its measurement. However, we know that burnout is a multifaceted phenomenon marked by a gradual depletion of a person’s physical, emotional and cognitive resources in response to his or her work.
While there is considerable debate over the dimensions of burnout, most researchers agree that emotional exhaustion appears to be at the heart of burnout (3-5) or that it represents the only component of burnout (6, 7) and that it is related to a host of occupational and personal outcome measures such as intention to quit, medical errors, mental quality of life, and self-rated health (2, 8-11). Indeed, the ICD-10 defines burnout as a “state of vital exhaustion” (12).
The maslach burnout inventory (MBI) (13) consisting of 3 subscales measuring emotional exhaustion (EE), depersonalisation (D), personal accomplishment (PA), has traditionally been the most commonly used tool for the measurement of burnout, with its’ widespread use being self-perpetuated by researchers’ need to compare their findings to those of others. However, the usefulness of the MBI as a screening tool is limited by its length, and the less than intuitive underpinnings of its development. A critique by Kristensen et al. (2005) (7) highlights the problem of having the singular construct ‘burnout’ measured by three distinct and different scales, which scores cannot be combined to yield a single burnout score. Additionally, this multi-dimensional measurement of burnout raises the important issue of interpretation; e.g., is a person scoring low on emotional exhaustion and low on depersonalisation actually burnt out, or is high emotional exhaustion a prerequisite for a “burnout diagnosis”. Such inherent problems similarly exist with West et al.’s (10, 14) proposed use of 2 key items from the EE (“How often do you feel burned out from your work?”) and D subscale (“How often do you feel you’ve become more callous towards people since you took this job?”) as a brief measure of burnout.
It appears from the literature that using a person’s own definition of burnout may be a valid way of assessing burnout (15, 16). Pick and Leiter (17) also found in their qualitative study that nurse’s self-definition of burnout was strongly related to emotional exhaustion but not depersonalisation or personal accomplishment, suggesting that these may not be salient aspects of the lived experience of burnout. Indeed, some researchers (e.g. 7) have argued that depersonalisation may rather represent a coping strategy applied in situations of burnout and personal accomplishment a consequence of burnout rather than parts contained within the construct. This idea is fully consistent with Leiter and Maslach’s (18) own model as well as that proposed by Lee and Ashforth (19) in which EE is conceptualised as the first burnout dimension to develop, with D and PA developing as a direct result of EE. Further testing of these models carried out by Taris et al. (20) confirmed that EE triggers D, and that D in consequence affects PA.
Previous validation research has assessed single item burnout measures amongst physicians on a nine point rating scale. The study found that physicians assess their global burnout in terms of emotional exhaustion (21). Previous validations of a single item global measure of burnout initially developed for the physician work life study (22) have been carried out against the MBI EE only and have yielded promising results (15, 16, 23) but previously not assessed in terms of association with relevant outcome measures. This measure asked respondents to rate their current level of burnout by endorsing one of five statements describing the gradual development of (self-defined) burnout. However, the limitations imposed by the restricted number of response categories may affect its usefulness in situations where detection of minor changes in burnout or over shorter time frames is required. Therefore, allowing respondents to define and label their experience of burnout on an 11-point verbal numeric rating scale may prove to be a quick and sensitive measure with applicability in human resource or screening contexts, as well as research.
A single zero to ten global burnout items is brief and easily administered, thereby increasing survey completion rates. By not applying an artificial cut-off point, it allows an assessment (and timely intervention) of the gradual development of burnout.
2. Objectives
This paper reports on the validation of such an item against a comprehensive emotional exhaustion measure (MBI-EE), its sensitivity, specificity and positive and negative predictive value, as well as association with a range of outcome measures (early retirement intentions, psychological distress, and self-rated general health).
3. Patients and Methods
3.1. Subjects and Recruitmen
Participants in this study were rural GPs who were members of the Northern Rivers general practice network (NRGPN) and practicing in the Northern Rivers region of NSW, Australia. Potential participants received a study package from NRGPN containing a covering letter, a participant information sheet, and the anonymous survey. All 165 eligible participants received two reminders 2 and 4 weeks after the initial invitation. Data collection took place between October 2011 and February 2012. The study protocol conforms to the ethical guidelines of the 1975 Declaration of Helsinki as reflected in a priori approval by The University of Sydney Human Research Ethics Committee (approval number: 14112).
3.2. Instruments
The survey included the following measures:
3.2.1. Demographic and Work Factors
Age, gender, retirement intentions (planned age of retirement from direct patient care in general practice).
3.2.2. Professional Burnout
Burnout was assessed by two different measures;
3.2.2.1. Maslach Burnout Inventory (Human Services Survey) (13) 9-Item Emotional Exhaustion subscale
The MBI is a very well-validated scale with sound psychometric properties and considered, by many, to be a benchmark measure of burnout, with the EE subscale often used as a stand-alone measure. Each item is rated on a seven-point Likert Scale. MBI-EE sub-scale burnout score categories are: low (< 18), medium (19 - 26) and high (≥ 27) as recommended by the scale developers.
3.2.2.2. The Single Item Burnout measure (SIB)
SIB was developed by the authors for this study, asking respondents to rate their current level of burnout on a scale from zero to ten (“not at all burnt out” to “extremely burnt out”).
3.2.2.3. Psychological Distress
Psychological Distress: The six-item Kessler Psychological Distress Scale (K-6) (24) was included as a brief measure of non-specific psychological distress.
3.2.2.4. General Health
General Health: The global health question from the SF-36 was included to measure self-rated general health (“In general, would you say your health is;” rated on a five-point Likert scale ranging from “poor” to “excellent”). This item has consistently been found to possess strong psychometric properties compared to validated multi-item measures (25) and be a good predictor of mortality and health care utilization (26, 27).
3.3. Statistical Analyses
Analyses were conducted using SAS 9.3. Mean scores and prevalence of burnout on the MBI-EE and the SIB were calculated. An ANOVA analysis compared SIB scores to MBI-EE sub-scale score categories of low (< 18), medium (19 - 26) and high (≥ 27) with the data displaying normally distributed residuals and homogeneity of variance. A Pearson correlation coefficient was also calculated as a measure of association.
A Bland-Altman analysis was used to assess the level of agreement between the two methods to compare the shorter single item technique to the established emotional exhaustion subscale of the MBI. A range of agreement was defined as mean bias ± 2 SD. A scatterplot of the average of the two scales against the difference between the two scales for each participant was used. The EE scale was transformed to be the same as the SIB scale (0 to 10).
The plot was used as a visual check that the magnitudes of the differences were constant throughout the range of measurement, with the expectation that approximately 5% of the points would lie outside the limit lines if the differences were normally distributed (28).
Raw SIB scores were examined for their association with a number of outcome measures. Early retirement was treated as a nominal variable and defined as a planned retirement age before 65 years of age, psychological distress (K6) was categorized into low, moderate, high and very high applying cut-off values recommended by the scale developers (24) while self-rated general health similarly was treated as an ordinal variable retaining its five response categories.
Sensitivity, specificity, positive and negative predictive value, and prevalence and bias adjusted weighted Kappa coefficients were calculated using the MBI-EE as the standard.
4. Results
A total of 92 GPs completed the survey, representing a response rate of 56%, with all completing both the index and reference test. This constitutes a high response rate in this population given the relatively sensitive natures of the survey and the well-known difficulty in recruiting time-poor GPs (29) with recent studies with Australian GPs having reported response rates of between 12% and 59% (30-32). The mean age was 51.3 years (SD = 10.7 years; 95% CI: 48.75 to 53.84). Sixty percent (95% CI: 50 to 70) were male which is slightly lower than the general rural GP population which is 71% male (33). The level of burnout of the GPs in this study as measured on the MBI-EE (Mean = 18.9, SD = 13.5 (95%CI: 15.67 - 22.11), were slightly lower than published norms (Mean = 22.19, SD = 9.53, (95%CI: 21.55 to 22.83) from 1104 physicians and nurses in the USA (13). A quarter of our sample (26%) was identified as having high levels of burnout.
The mean score on the SIB was 3.1 (SD = 2.5). An ANOVA analysis showed that the Mean SIB scores increased with increased level of burnout as per MBI-EE burnout categories: Mean SIB scores (SD) in the low, average and high burnout categories on the MBI-EE were 1.6 (1.7), 3.5 (1.7), and 6.0 (2.0) respectively (P < 0.0001). The Pearson correlation coefficient was r = 0.8 (P < 0.0001).
The Bland-Altman analysis indicated that the 95% limits of agreement between the two methods ranged from -2.78 to 3.73. The difference of the mean bias was 0.48 (SD = 1.62). The mean difference was different from zero (P = 0.0069). A visual check demonstrated that the magnitudes of the differences were reasonably constant throughout the range of measurement. The differences were approximately normally distributed, and as expected about 5% of the points lay outside the limit lines.
Construct validity was demonstrated by examining the SIB for its association with a number of salient outcome measures (Table 1) and showed high positive associations with early retirement intentions and psychological distress, and a high negative association with self-rated general health.
Associations of the Single Item Burnout Measure (SIB) with Retirement Intentions, Psychological Distress and General Health Outcomes
| Outcome | No. | SIB Mean | SIB SD | Test Statistic | P value |
|---|---|---|---|---|---|
| Early retirement intentions | t (90) = 2.68 | 0.0089 | |||
| Yes | 52 | 2.5 | 2.4 | ||
| No | 40 | 3.9 | 2.5 | ||
| Psychological distress | F (3, 88) = 16.23 | < 0.0001 | |||
| Low | 64 | 2.2 | 2.0 | ||
| Medium | 20 | 4.5 | 2.5 | ||
| High | 6 | 6.5 | 1.2 | ||
| Very high | 2 | 8.0 | 0.0 | ||
| General health | F (4, 85) = 8.83 | < 0.0001 | |||
| Poor | 3 | 6.3 | 2.9 | ||
| Fair | 17 | 4.5 | 2.3 | ||
| Good | 24 | 4.0 | 2.9 | ||
| Very good | 26 | 1.9 | 1.8 | ||
| Excellent | 20 | 1.6 | 1.3 |
Characteristics of the cut-off values on the SIB, applied to the 24 GPs who displayed high burnout on the MBI-EE (score ≥ 27), are reported in Table 2.
Prevalence of Burnout According to the Single Item Burnout Measure (SIB) and MBI-EE, Prevalence and Bias Adjusted Weighted Kappa Coefficient, Sensitivity, Specificity, and Positive and Negative Predictive Value of the SIB Compared to Those With a High Burnout Score on the MBI-EE (≥ 27) Amongst 92 GPs
| Burnout Prevalence | Observed | Prevalence and bias adjusted kappa | Positive | Negative | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MBI-EE (High) | SIB | Agreement | Sensitivity | Specificity | predictive Value | predictive Value | |||||||
| SIB Score | % | 95% CI | % | 95% CI | % | K | 95% CI | % | 95% CI | % | 95% CI | % | % |
| 26 | (17 to 35) | ||||||||||||
| ≥ 0 | NA | NA | 100 | (100 to 100) | 26 | -0.48 | (-0.68 to -0.27) | 100 | (100 to100) | NA | NA | NA | NA |
| ≥ 1 | NA | NA | 85 | (78 to 92) | 41 | -0.17 | (-0.38 to -0.03) | 100 | (100 to100) | 21 | (11 to 30) | 31 | 100 |
| ≥ 2 | NA | NA | 64 | (54 to 74) | 62 | 0.24 | (0.0.3 to 0.44) | 100 | (100 to100) | 49 | (37 to 60) | 41 | 100 |
| ≥ 3 | NA | NA | 48 | (38 to 58) | 78 | 0.56 | (0.36 to 0.80) | 100 | (100 to100) | 71 | (60 to 81) | 55 | 100 |
| ≥ 4 | NA | NA | 38 | (28 to 48) | 79 | 0.59 | (0.38 to 0.79) | 83 | (68 to 98) | 78 | (68 to 88) | 57 | 93 |
| ≥ 5 | NA | NA | 30 | (11 to 27) | 85 | 0.70 | (0.49 to 0.90) | 79 | (63 to 95) | 87 | (79 to 95) | 68 | 92 |
| ≥ 6 | NA | NA | 19 | (7 to 12) | 82 | 0.63 | (0.43 to 0.83) | 50 | (30 to 70) | 93 | (86 to 99) | 71 | 84 |
| ≥ 7 | NA | NA | 12 | (5 to 19) | 84 | 0.67 | (0.47 to 0.88) | 42 | (21 to 61) | 99 | (96 to 100) | 91 | 83 |
| ≥ 8 | NA | NA | 9 | (3 to 15) | 83 | 0.65 | (0.45 to 0.86) | 33 | (15 to 52) | 100 | (100 to 100) | 100 | 81 |
| ≥ 9 | NA | NA | 2 | (0 to 5) | 92 | 0.52 | (0.31 to 0.73) | 2 | (0 to 5) | 100 | (100 to 100) | 100 | 76 |
| ≥ 10 | NA | NA | 0 | NA | 74 | 0.47 | (0.27 to 0.68) | NA | NA | 100 | (100 to 100) | NA | 74 |
The proportion of observed agreement between high MBI-EE and the various SIB cut-off scores was highest between scores of 3 and 9, ranging from 78% to 92%. Kappa showed good agreement at a cut-off score of 5.
Sensitivity declined and specificity increased with increasing SIB cut-off scores. Similarly, positive predictive values increased with higher SIB scores, whereas negative predictive values decreased.
Generally, the trade-off between sensitivity and specificity reached most optimal levels at a score of 5 or more, yielding sensitivity of 79%, specificity of 87%, positive predictive value of 68%, and negative predictive value of 92%, indicating that 79% of GPs truly were burnt out, and 87% were truly not burnt out, according to the MBI-EE when using a SIB cut-off score of 5 or more.
5. Discussion
Due to high prevalence of burnout in health care providers and the significant associations it has with important variables relating to mental and physical health and intention to leave, it is imperative that a brief, sensitive screening measure is available which allows for early identification of burnout. The results of this current study indicate that the SIB has significant potential to fulfill this gap due to its brevity, ease of administration and sound psychometric properties. Firstly, scores on the SIB were found to be highly associated with several person outcome variables previously documented to be related to burnout; self-rated general health, psychological distress, and early retirement intentions, thus lending evidence to its construct validity. Upon examination of its performance against the reference standard (maslach burnout inventory-emotional exhaustion subscale) (13), concurrent validity of the SIB was evidenced by findings of a high positive association between SIB and MBI-EE scores. This association was in this case higher than those reported by Hansen and Girgis (15) and Rohland et al. (16) while almost identical to that found by Dolan et al. (23) in their validation study amongst a large sample of primary care staff. The results from the bland-altman analysis similarly indicated that only a minimal bias exists of nearly half a point between the SIB and the EE, with EE scores being higher. However, a difference of this magnitude has minimal clinical implications in a screening context and suggest that the SIB and the EE provide reasonably similar measures across the scale. The sensitivity and specificity analyses confirmed the SIB’s ability to correctly identify highly burnt out participants with a high degree of accuracy, when using the full MBI-EE subscale as the standard.
Some potential limitations to the current study should be noted. Firstly, a multi-faceted assessment of burnout incorporating the inclusion of objective measures of burnout (e.g. third-party assessment, absenteeism) in addition to the MBI-EE would be a stronger standard against which to test the utility of the SIB. Secondly, no inferences about the test-retest reliability of the SIB can be made on the basis of the current study, as it involved assessment at only one time point. Thirdly, the small sample size needs to be acknowledged.
However, the promising results from this study lend support to the utility of the Single Item Burnout Measure, with potential applicability in both human resource (including organizational scans) and research contexts, and calls for further testing of this tool in a larger sample across other health care settings.
Acknowledgements
References
-
1.
Orton P, Orton C, Pereira Gray D. Depersonalised doctors: a cross-sectional study of 564 doctors, 760 consultations and 1876 patient reports in UK general practice. BMJ Open. 2012;2:e000274. [PubMed ID: 22300669]. https://doi.org/10.1136/bmjopen-2011-000274.
-
2.
Soler JK, Yaman H, Esteva M, Dobbs F, Asenova RS, Katic M, et al. Burnout in European family doctors: the EGPRN study. Fam Pract. 2008;25(4):245-65. [PubMed ID: 18622012]. https://doi.org/10.1093/fampra/cmn038.
-
3.
Maslach C. Burnout, the cost of caring. Englewood Cliffs, NJ: Prentice-Hall; 1982.
-
4.
Pines O, Yoon HJ, Inouye M. Expression of double-stranded-RNA-specific RNase III of Escherichia coli is lethal to Saccharomyces cerevisiae. J Bacteriol. 1988;170(7):2989-93. [PubMed ID: 3290193].
-
5.
Shirom A. Burnout in work organizations. In: Cooper CL, Robertson IT, editors. International review of industrial and organizational psychology. New York: Wiley; 1989. p. 25-48.
-
6.
Koeske GF. Construct validity of the maslach burnout inventory: A critical review and reconceptualization. J Appl Behav Sci. 1989;25(2):131-44. https://doi.org/10.1177/0021886389252004.
-
7.
Kristensen TS, Borritz M, Villadsen E, Christensen KB. The copenhagen burnout inventory: A new tool for the assessment of burnout. WORK STRESS. 2005;19(3):192-207. https://doi.org/10.1080/02678370500297720.
-
8.
Ilhan MN, Durukan E, Taner E, Maral I, Bumin MA. Burnout and its correlates among nursing staff: questionnaire survey. J Adv Nurs. 2008;61(1):100-6. [PubMed ID: 18034813]. https://doi.org/10.1111/j.1365-2648.2007.04476.x.
-
9.
Shanafelt TD, Balch CM, Bechamps G, Russell T, Dyrbye L, Satele D, et al. Burnout and medical errors among American surgeons. Ann Surg. 2010;251(6):995-1000. [PubMed ID: 19934755]. https://doi.org/10.1097/SLA.0b013e3181bfdab3.
-
10.
West CP, Dyrbye LN, Satele DV, Sloan JA, Shanafelt TD. Concurrent validity of single-item measures of emotional exhaustion and depersonalization in burnout assessment. J Gen Intern Med. 2012;27(11):1445-52. [PubMed ID: 22362127]. https://doi.org/10.1007/s11606-012-2015-7.
-
11.
Williams ES, Konrad TR, Scheckler WE, Pathman DE, Linzer M, McMurray JE, et al. Understanding physicians' intentions to withdraw from practice: the role of job satisfaction, job stress, mental and physical health. 2001. Health Care Manage Rev. 2010;35(2):105-15. [PubMed ID: 20234217]. https://doi.org/10.1097/01.HMR.0000304509.58297.6f.
-
12.
International statistical classification of diseases and related health problems. 10th ed. Geneva, Switzerland: World Health Organization; 1992.
-
13.
Maslach C, Jackson SE. Maslach Burnout Inventory Manual. 2nd ed. CA: Consulting Psychologists Press; 1986.
-
14.
West CP, Dyrbye LN, Sloan JA, Shanafelt TD. Single item measures of emotional exhaustion and depersonalization are useful for assessing burnout in medical professionals. J Gen Intern Med. 2009;24(12):1318-21. [PubMed ID: 19802645]. https://doi.org/10.1007/s11606-009-1129-z.
-
15.
Hansen V, Girgis A. Can a single question effectively screen for burnout in Australian cancer care workers? BMC Health Serv Res. 2010;10:341. [PubMed ID: 21162747]. https://doi.org/10.1186/1472-6963-10-341.
-
16.
Rohland BM, Kruse GR, Rohrer JE. Validation of a single-item measure of burnout against the Maslach Burnout Inventory among physicians. Stress Health. 2004;20(2):75-9. https://doi.org/10.1002/smi.1002.
-
17.
Pick D, Leiter MP. Nurses' perceptions of burnout: a comparison of self-reports and standardized measures. Can J Nurs Res. 1991;23(3):33-48. [PubMed ID: 1802377].
-
18.
Leiter MP, Maslach C. The impact of interpersonal environment on burnout and organizational commitment. J Organ Behav. 1988;9(4):297-308. https://doi.org/10.1002/job.4030090402.
-
19.
Lee RT, Ashforth BE. A meta-analytic examination of the correlates of the three dimensions of job burnout. J Appl Psychol. 1996;81(2):123-33. [PubMed ID: 8603909].
-
20.
Taris TW, Le Blanc PM, Schaufeli WB, Schreurs PJG. Are there causal relationships between the dimensions of the Maslach Burnout Inventory? A review and two longitudinal tests. Work Stress. 2005;19(3):238-55. https://doi.org/10.1080/02678370500270453.
-
21.
Rafferty JP, Lemkau JP, Purdy RR, Rudisill JR. Validity of the Maslach burnout inventory for family practice physicians. J Clin Psychol. 1986;42(3):488-92. https://doi.org/10.1002/1097-4679(198605)42:3<488::aid-jclp2270420315>3.0.co;2-s.
-
22.
Schmoldt RA, Freeborn DK, Klevit HD. Physician burnout: recommendations for HMO managers. HMO Pract/HMO Group. 1994;8(2):58-63.
-
23.
Dolan ED, Mohr D, Lempa M, Joos S, Fihn SD, Nelson KM, et al. Using a single item to measure burnout in primary care staff: a psychometric evaluation. J Gen Intern Med. 2015;30(5):582-7. [PubMed ID: 25451989]. https://doi.org/10.1007/s11606-014-3112-6.
-
24.
Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SL, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959-76. [PubMed ID: 12214795].
-
25.
DeSalvo KB, Fisher WP, Tran K, Bloser N, Merrill W, Peabody J. Assessing measurement properties of two single-item general health measures. Qual Life Res. 2006;15(2):191-201. [PubMed ID: 16468076]. https://doi.org/10.1007/s11136-005-0887-2.
-
26.
DeSalvo KB, Bloser N, Reynolds K, He J, Muntner P. Mortality prediction with a single general self-rated health question. A meta-analysis. J Gen Intern Med. 2006;21(3):267-75. [PubMed ID: 16336622]. https://doi.org/10.1111/j.1525-1497.2005.00291.x.
-
27.
Idler EL, Angel RJ. Self-rated health and mortality in the NHANES-I Epidemiologic Follow-up Study. Am J Public Health. 1990;80(4):446-52. https://doi.org/10.2105/ajph.80.4.446.
-
28.
Altman DG. Practical statistics for medical research. London: Chapman & Hall; 1991.
-
29.
Pit SW, Vo T, Pyakurel S. The effectiveness of recruitment strategies on general practitioner's survey response rates - a systematic review. BMC Med Res Methodol. 2014;14:76. [PubMed ID: 24906492]. https://doi.org/10.1186/1471-2288-14-76.
-
30.
Bonevski B, Magin P, Horton G, Foster M, Girgis A. Response rates in GP surveys - trialling two recruitment strategies. Aust Fam Physician. 2011;40(6):427-30. [PubMed ID: 21655493].
-
31.
Brett TD, Arnold-Reed DE, Hince DA, Wood IK, Moorhead RG. Retirement intentions of general practitioners aged 45-65 years. Med J Aust. 2009;191(2):75-7. [PubMed ID: 19619089].
-
32.
Crouch S, Robinson P, Pitts M. A comparison of general practitioner response rates to electronic and postal surveys in the setting of the National STI Prevention Program. Aust N Z J Public Health. 2011;35(2):187-9. [PubMed ID: 21463418]. https://doi.org/10.1111/j.1753-6405.2011.00687.x.
-
33.
National health workforce. 2013; Canberra, Australia. AIHW;