An Evaluation of the Online Quality of Content of United States Accredited Pain Medicine Fellowship Training Programs

1. Background

With the increasing availability of comprehensive fellowship information, applicants exploring career plans are relying heavily on using the Internet to identify and evaluate information (1-3). The application process to pain medicine training programs is carried out through online platforms such as the Electronic Residency Application Service (ERAS) and the National Resident Matching Program (NRMP). In addition, the Fellowship and Residency Electronic Interactive Database (FREIDA) consolidates and standardizes the online information available to prospective residents applying to pain medicine fellowship programs. FREIDA provides generalized information about training programs from location, addresses and contacts to program size and positions available. This information allows applicants to gather more information about programs and characteristics that may influence their program choice. Consequently, all three databases lead prospective pain medicine fellowship applicants directly to the programs or their sponsoring institutions.

Investigations of website content have been conducted for a variety of graduate medical education programs: surgery, emergency medicine, otolaryngology, radiation oncology, neurosurgery, and orthopedics. The surgical subspecialties studies demonstrate underutilization of training program websites for education and recruitment of potential candidates by the recruiting program (2-8). Furthermore, studies in various specialties have demonstrated that during the application process there is a notable importance of program websites (9-11). Given the reliance of online websites: the marketing and sales domains suggest that website quality, rather than mere existence, proves beneficial to company competitiveness, consumers attraction, and purposeful intentions (12-14). The are no published studies that have evaluated the accessibility, availability, and quality of online content for pain medicine fellowship training programs.

2. Objectives

The objective of this cross-sectional study is to evaluate the content, design, organization, and user friendliness by using a previously defined composite score to determine the quality of the pain medicine fellowship websites (PMFW). Our hypothesis was to explore PMFW to assess identifiable areas of strength and weakness within the categories of content, design, organization, and user friendliness.

3. Methods

3.1. Study Design

This study was performed under an exempt status granted by the Institutional Review Board of Rhode Island (IRB#1744837). The IRB determined that the study qualified for exemption under 45 CFR 46.101(b). The study is reported following the STROBE guidelines for reporting observational studies (15).

A comprehensive list of accredited board-certified pain medicine fellowship programs by the Accreditation Council for Graduate Medical Education (ACGME) (16) was obtained by cross referencing three databases for pain medicine education: (1) ERAS (17), (2) the FREIDA (18), and (3) the NRMP (19). Pain fellowship programs that were included in one or more of the databases qualified for the study. Following the identification of the Pain medicine fellowship websites (PMFW), all websites were accessed and evaluated by three independent reviewers for the accessibility of recruitment and educational content items and for the quality of PMFW as determined by a combination of four dimensions: (1) content, (2) design, (3) organization, and (4) user friendliness. As part of website training, each reviewer examined an optimal website, an average website, and a marginal website. The three authors selected the three examples based on their cumulative assessment and scoring. The first quartile was attributed to an optimal website, a mean scaled score was granted to an average website, while a marginal website scored in the fourth quartile. Each reviewer assessed each website autonomously of the other reviewers maintaining individual integrity. The sequence of websites was randomized by using a computer random number generator.

3.2. Availability of Pain Medicine Fellowship Websites

Availability of PMFW was determined by examining educational databases (ERAS, FREIDA, and NRMP) for a complete number of pain medicine fellowship programs listed while qualifying the existence or deficiency of website links. The hyperlink was considered as either functional or nonfunctional if present. Functional was described as whether or not a website hyperlink worked. The hyperlink was considered functional if selecting the link produced any website. However, the hyperlink was considered nonfunctional if clicking on a link led to an inactive webpage or produced an error. All working functional hyperlinks were consequently categorized as being either “direct” (linking directly to the pain medicine fellowship website) or “indirect” (linking to a website of the hospital, academic affiliation or departmental website and not the pain medicine fellowship program homepage).

3.3. Accessibility of Content

PMFW were investigated for the accessibility of knowledge about the programs used to update and attract prospective applicants. Variables based on ACGME program requirements for graduate medical education in pain medicine (16) and previous literature in the field (3, 6-8) related to resident recruitment or resident education, were identified, and charted as either present or absent. A total of 32 pre-defined variables were used to assess the PMFW for program recruitment variables that may influence recruitment of future fellows as referenced by the ACGME (16).

PMFW were also assessed for comprehensiveness in specifying how their practicing site met core components of an ACGME pain medicine fellowship program. A total of 16 program education variables were used to assess each individual PMFW. Each PMFW scored within the listed domains of program recruitment and education, regarding the accessibility of content. PMFW content was the specific content evaluated, unless a direct URL link to the specific information was present in the PMFW.

3.4. Website Quality

A predefined composite of four dimensions: (1) content, (2) organization, (3) design, and (4) user friendliness was used to appraise website quality (20). Each domain contained subcategories to specifically assess website quality. Each PMFW was evaluated on a 4-point Likert scale (1 = poor, 2 = acceptable, 3 = good, 4 = great) (14). Neutral positions were excluded to determine whether a website achieved a score of ‘good’ or if there was room for improvement (Figure 1).

The quality of pain medicine fellowship websites was assessed using a predefined composite of four domains and their corresponding subcategories. Each domain was appraised on a 4-point Likert scale (1 = poor, 2 = acceptable, 3 = good, 4 = great).

Figure 1.

The quality of pain medicine fellowship websites was assessed using a predefined composite of four domains and their corresponding subcategories. Each domain was appraised on a 4-point Likert scale (1 = poor, 2 = acceptable, 3 = good, 4 = great).

Data are presented as counts and percentages. Inter-rater reliability was assessed using Cohen’s Kappa (Microsoft Excel for Microsoft 360 2021, Version 16.0.13901.20366 64-bit).

4. Results

When accessed in November 2020, 103 of the 105 ERAS, 110 NRMP, and 111 FREIDA listings contained a unique PMFW. Cross-referencing the three databases provided a list of 103 pain medicine fellowship programs that met eligibility criteria for the assessment of the 32-program recruitment and 16 program education variables.

FREIDA provided hyperlinks to 105 fellowship programs (95%), with 90 functional (81%) and 18 direct (16%) hyperlinks. NRMP had 95 hyperlinks (86%), 91 (83%) were found to be functional, and 24 (22%) linked directly to the pain medicine programs. Of the total PMFW, ERAS specified hyperlinks to only 84 programs (80%), with only 64 programs (61%) that were functional and 30 programs (29%) with direct access.

For program recruitment, there was an average of 12 of 32 content items (38%; range 0 - 22, ± 4.0) for fellowship programs (Figure 2). Nearly all PMFW provided a description of the fellowship training. Information found in the upper quartile of PMFW (100 - 75% of websites) included program description, program administrator contact, eligibility, application listing requirements, and available positions. The second quartile of PMFW (75 - 50% of websites) reported faculty listing, salary, fellowship director contact, USMLE requirement, interview dates, and an ERAS hyperlink. In the third quartile (50 - 25% of websites) was: (1) listings of current fellows or residents (44% fellowships); (2) information about the city (41% fellowships); (3) vacation (37% fellowships); (4) descriptive profiles for faculty (36% fellowships); (5) direct contact information for faculty (29% fellowships); (6) along with their education history (29% fellowships); (7) department and faculty research interests (27%); and (8) a listing of program alumni (25% fellowships). Data analysis showed: (1) a welcome message from the program director (19% fellowships); (2) alumni career placement (17% fellowships); (3) alumni education history (8.7% fellowships); (4) interview details (6.8% fellowships); (5) contact information for current trainees (4.9% fellowships); (6) specific factors considered during the selection process (4.9% fellowships); (7) details about work hours (3.9% fellowships); and (8) descriptive profiles (2.9% fellowships) were found to be in the fourth quartile (25 - 0% of websites. Almost no PMFW offered details about board examination pass rates (0.97% fellowships) or alumni contact information (0% fellowships).

The percentage of pain medicine fellowship websites that displayed the recruitment criteria content as referenced by the ACGME. Of the 103 websites evaluated, an average of 38% recruitment content items were present. Program description and administrator contract (98%) was present in almost all websites whereas board exam and alumni contact information was negligible (< 1%). The 32 recruitment criteria are used to attract applicants and potential pain medicine candidates to the respective program.

Figure 2.

The percentage of pain medicine fellowship websites that displayed the recruitment criteria content as referenced by the ACGME. Of the 103 websites evaluated, an average of 38% recruitment content items were present. Program description and administrator contract (98%) was present in almost all websites whereas board exam and alumni contact information was negligible (< 1%). The 32 recruitment criteria are used to attract applicants and potential pain medicine candidates to the respective program.

In regards to program education, an average of 7 out of 16 content items (44%; range 0 - 14, ± 3.4) were listed on program websites. A list of affiliated hospitals (89% fellowships) is the only program information found in the top quartile of PMFW (100 - 75% of websites) (Figure 3). The second quartile of PMFW (75 - 50% of websites) included: rotation schedule (70% fellowships), details about didactics (67% fellowships), specified research requirements (59%), acute pain coverage (53% fellowships), national organization links (52% fellowships), and journal club meetings (53% fellowships). Evaluation methods and criteria (47%), weekly conferences (46%) and interventional experiences (49%), could be found in the third quartile (50 - 25% of websites). The topics that appeared in the fourth quartile (25 - 0% of websites) were academic meetings frequently attended by trainees (21%), simulation training (16%), while case logs were only mentioned in 4.6% of PMFW.

The percentage of pain medicine fellowship websites that displayed the educational criteria content as referenced by the ACGME. Of the 103 websites evaluated, an average of 44% education content items were present. Affiliated hospitals (89%) and rotations (70%) were present in most of the websites whereas simulation training (15%) and case logs (< 5%) were marginally present. The 16 education criteria are used to attract applicants and provide an overview of the educational components of the respective program.

Figure 3.

The percentage of pain medicine fellowship websites that displayed the educational criteria content as referenced by the ACGME. Of the 103 websites evaluated, an average of 44% education content items were present. Affiliated hospitals (89%) and rotations (70%) were present in most of the websites whereas simulation training (15%) and case logs (< 5%) were marginally present. The 16 education criteria are used to attract applicants and provide an overview of the educational components of the respective program.

Using the composite of four domains, the quality of PMFW was less than good with an average score of 2.6 out of 4.0, Table 1. The Cohen's Kappa coefficient which was 0.92.

5. Discussion

The most important finding of our study is an existing deficit in online education and recruitment content for PMFW. In addition, we found that overall quality of PMFW were below the standard of “good.” PMFW are not generally accessible and are not reliable in providing essential information for prospective applicants. These pitfalls may have serious implications particularly in a time when the field of pain medicine is looking to fill shortages in the future workforce. With the recent growth of technology and enhanced virtual presence of services, a large amount of information and engagement occurring worldwide takes place over the Internet. As a result, online presence and virtual visits provide prospective applicants with additional program information along with word of mouth. The decision of where to apply is the first step of the application process. Given the recent COVID pandemic, many known obstacles including financial resources, time, and geographic separation that would make visiting and applying to many training programs difficult, may be curtailed. Program web presence serves as the most available means for prospective applicants to learn more about programs. Conversely, the Internet gives programs the ability to easily distribute information to recruit prospective applicants. Previous literature in other medical specialties [dermatology (1), orthopedic surgery (2), emergency medicine (5), otolaryngology (6), neurosurgery (8), and anesthesiology (10)] has demonstrated that the majority of applicants utilize online resources when researching programs. Such studies have also shown web-based content often influencing where applicants decide to apply and subsequently, the order in which programs are ranked. Program websites may be comparable in importance to information obtained from mentors and peers especially with changing program leadership and structures, websites may even be the most significant source of information for deciding on program matriculation (9, 11).

It is necessary to ask two important questions when evaluating program websites: (1) What is actually available online, and does the information prove to be helpful? (2) Does the PMFW present the information in a comprehensive, aesthetically pleasing, and navigable format? For information to be useful, it must be available. Our findings demonstrate that national databases (ERAS, NRMP and FREIDA) maintain lists of accredited pain medicine fellowship training programs, however each database provides functional direct website hyperlinks to less than a third of fellowship programs. It should be obligatory for programs to check their hyperlinks yearly on appropriate databases to ensure that their link is functional and direct. Because applying to fellowship programs often leads to consulting online databases before and during the application process (9), our findings suggest that PMFW may not be readily attainable from national official databases. Regardless of accessibility, our findings show that information on PMFW is not complete.

Deficiencies in resident recruitment domains negatively influences a program’s potential to entice high-quality applicants. Three of the most important factors influencing fellows, chief residents, and medical students regarding program education content is the use of imaging technology, challenging procedures, and faculty mentorship (9). For PMFW with existing deficits in the program education, potential applicants may be less likely to consider and rank the corresponding programs.

Relating to our proposed second question, our results demonstrate that PMFW for fellowship programs were below the quality threshold of “good” when assessed using the four dimensions. Previous studies in retail, marketing, and eCommerce have demonstrated that website quality attracts consumers, influence decisions, and enhance a company’s presence on the market (13). In our study, the quality of PMFW likely influences applicants’ decisions about program applications, accepting interview offers, and ranking programs. Website quality likely affects programs’ ability to entice competitive applicants. Our investigation demonstrates the need for more attention to develop and maintain web-based content for pain medicine fellowship programs.

The internet’s presence in everyday life is even more so with handheld devices therefore pain medicine training programs should maximize their web presence given the setting of reduced travel and virtual interactions triggered by the COVID pandemic. Inadequate websites can negatively affect a program’s recruitment efforts during the application process. A survey of residency applicants found that nearly half of the applicants chose not to apply to certain programs because of poorly maintained websites ranging from inadequate information, aesthetic quality, or a combination of both (5). The focus on program websites may have promising consequences for all parties involved. Pain medicine fellowships can improve recruitment while applicants can better determine the best fit for their career aspirations. Beyond specific PMFW, it could prove beneficial to create a central database within a pain medicine society to provide a comprehensive listing with functional links of all accredited programs. This database would serve two primary purposes: (1) a reliable source for applicants; and (2) efficient access to PMFW. There is no single comprehensive resource even though multiple databases exist. While acknowledging the inherent challenges in an up-to-date database, both on individual PMFW and within centralized databases, organizations may not have full-time information technology support to regularly perform updates. We suggest all PMFW should be updated by their institutions by November 30th when planning to participate or enlist in the Fellowship Match. In accordance with the application timeline published by the Association of American Medical Colleges through ERAS (16), this proposal allows applicants time to solicit information and make decisive choices with program application. As competition grows in graduate medical education and adoption of virtual interviews, the number of fellowship programs that applicants are applying to continues to increase every year. Applicants may face decisions about which interview offers to accept and forgo based on limited resources such as time or money. While these decisions can be challenging, having access to high quality information on PMFW may mitigate this process. At a minimum, a PMFW should include all program recruitment and education content listed in Figures 1 and 2.

There are several limitations in this study. First, we used ACGME program requirements for graduate medical education to determine the content features for PMFW analysis. It is conceivable that others may interpret certain program characteristic features with varying importance. Second, this cross-sectional study only allows inspection of available information during the performed search in November 2020. There may have since been changes to databases and individual PMFW. Third, there were 2 programs that were listed on the databases, but no website or contributing information could be found, even with a general Internet query; these 2 programs were omitted from website grading by the evaluators due to the inability to find the associated website. Finally, inherent bias was not controlled and may exist as the PMFW were assessed without being blinded. It is important to note that our study does not evaluate where pain medicine applicants apply, interview, or ultimately match. Future qualitative studies could be performed to provide further insight on this topic.

5.1. Conclusion

Pain medicine training programs serve to inform, attract, and recruit prospective applicants; however, they underuse web-based resources. For PMFW, deficiencies exist in the areas of both content and quality. Overall, current PMFW scored less than good while only 1% of PMFW met the criteria of “great.” Improving the quality of information presented in PMFW would allow applicants the ability to identify potential pain medicine fellowship programs that may influence their decision when pursuing a career in pain management.