1. Background
Many researchers in the academic settings are unfamiliar with methods of searching for articles, databases, and scientific search engines (1). As they are less likely to be asked to speak as a lecturer, they rarely make the effort to learn appropriate presentation methods (2) and have an inadequate knowledge of the proper presentation of a scientific article (3). Another issue in this regard is the misconception that any article that has been published or written by famous authors has no weaknesses (4-6). On the other hand, educational environments are responsible for the continuous monitoring of their current status through the analysis of their issues and identifying their weaknesses and their causes in order to discover scientific solutions to improve the quality of education (7).
Journal club (JC) is an accepted method to improve knowledge and information by reviewing and discussing scientific texts (8). JC has long been employed and was first organized by William Osler in 1875 at McGill University (9). Today, it is used for the continuation of education (10). JC is an approach to updating knowledge, promoting critical thinking, evaluating the validity and applicability of previous research, developing critical thinking skills, and increasing the use of clinical research findings (11, 12).
In general, the objectives of JC are classified into three categories, including the study of recent research, making functional changes based on the results of new research, and increasing the ability of the participants to critically evaluate research (13). Several factors contribute to the success of JC, the most important of which is the regularity of the meetings, having a clear purpose, continuous and effective attendance of the participants, and selecting the appropriate methods of holding the meetings (14). Today, an important concern of managers in the successful implementation of such programs is achieving the goals of the program (15). Moreover, the evaluation of programs is essential to improve educational actions (16).
In this regard, CIPP is considered to be an effective evaluation model, which was developed by Daniel Stufflebeam et al. in the 1960s. Based on the CIPP model, the most important goal of evaluation is to improve and modify the course rather than stabilize the program. CIPP is an acronym for context, input, process, and product; the model seeks to find a tool that could improve and prepare a program for better use (17). The purpose of context evaluation is to provide a rational basis for setting learning goals, as well as an analytical effort to identify the relevant elements in the learning environment and the problems, needs, and opportunities within an educational context.
Input evaluation contributes to the design and selection of appropriate methods to achieve the goals of a program, while process evaluation aims to diagnose the prediction of executive problems during the performance of educational activities and the desirability of the implementation process of these activities. Product evaluation is performed to determine the desirability of the efficiency of educational activities (18). The CIPP evaluation model has been used in several studies aiming to assess educational programs and environments (19-21).
2. Objectives
The present study aimed to evaluate JC meetings from the perspective of postgraduate students of Kermanshah University of Medical Sciences (KUMS) in Kermanshah, Iran using the CIPP evaluation model.
3. Methods
3.1. Participants and Procedure
This descriptive-analytical study was conducted on 62 postgraduate students selected from the schools of health, nutritional sciences, and food industry of KUMS in 2019. The participants were selected via convenience sampling. The research objectives were explained to the selected participants, and they were assured of confidentiality terms regarding their personal information to enter the study willingly. After eliminating the incomplete questionnaires, 58 questionnaires were analyzed with the response rate of 93.5%.
The postgraduate students selected from the schools of health, nutritional sciences, and food industry of KUMS who attended JC meetings at least three times were considered eligible for enrollment, and the exclusion criteria were unwillingness to partake in the research and incomplete questionnaires.
3.2. Measures
The data collection instrument consisted of two sections, which were background information and the CIPP model domains. Data were collected in a self-report manner.
3.2.1. Background Information
The first section of the questionnaire included background information, such as age (year), gender, academic school (health, nutritional sciences, food industry), marital status (single, married), grade point average (GPA), semester, history of participation in research methodology workshops (yes/no), and number of the times attending the JC meetings.
3.2.2. CIIP Evaluation Model
The items of the CIIP evaluation model have been designed based on standard questionnaires (17-24). In total, the model has 33 items in four domains of context, input, process, and product. In order to facilitate the responses of the participants, all the items were standardized to a five-point Likert scale (slightly = 1, extremely = 5). The reliability of the questionnaire was estimated by conducting a pilot study on 10 students using the coefficient alpha (0.82).
Three items were designed to measure the context domain (e.g., appropriateness of the timing of JC meetings), and five items were designed to measure the input domain (e.g., updating level of article presentation). In addition, 11 items were designed for the process domain (e.g., convenience of JC meeting place), and 14 items were designed to evaluate the product domain (e.g., motivating research in students); the higher scores indicated the better status of each domain. The estimated reliability of the context domain was α = 0.73, while it was α = 0.62 in the input domain, α = 0.70 in the process domain, and α = 0.65 in the product domain. The face validity of the model was also determined using the qualitative approach. To this end, a panel of 10 health education and medical education experts were interviewed face-to-face to confirm the difficulty, relevance, clarity, and ambiguity of the questionnaire. Afterwards, the comments of the panelists were applied to some of the items with minor modifications.
3.3. Research Ethics
The study protocol was approved by the KUMS Research Ethics Committee (code: IR.KUMS.REC.1396.673). In addition, the research procedures and objectives were explained to the participants, and they were assured of the confidentiality of information.
3.4. Statistical Analysis
Data analysis was performed in SPSS version 16 using bivariate correlation, one-way analysis of variance (ANOVA), and t-test.
4. Results
In this study, the students were within the age range of 23 - 38 years with the mean age of 25.18 ± 2.09 years. In total, 62.1% of the participants were female, 37.9% were male, and 24.1% were married. Table 1 shows the findings regarding the background information of the respondents. Accordingly, there were positive, significant correlations between the domains of the CIPP model. In addition, the GPA of the students had positive, significant correlations with the input and process domains. Table 2 shows the correlations between the domains of the CIPP model, age, and GPA of the students.
| Variables | Number | Percent |
|---|---|---|
| Gender | ||
| Female | 36 | 62.1 |
| Male | 22 | 37.9 |
| Marital status | ||
| Married | 14 | 24.1 |
| Single | 44 | 75.9 |
| Field of study | ||
| Health education | 12 | 20.7 |
| Environmental health | 16 | 27.6 |
| Biostatics | 10 | 17.2 |
| Epidemiology | 11 | 19 |
| Nutrition | 9 | 15.5 |
| School | ||
| Health | 49 | 84.5 |
| Nutrition sciences and food industry | 9 | 15.5 |
| Semester | ||
| 1 | 10 | 17.2 |
| 3 | 22 | 37.9 |
| 5 | 23 | 39.7 |
| 7 | 3 | 5.2 |
| History of participating in research method workshops | ||
| No | 12 | 20.7 |
| Yes | 46 | 79.3 |
| Frequency of attending in the JC meetings | ||
| 3 | - | 48.3 |
| 4 | - | 36.2 |
| 5 and more | - | 15.5 |
Background Variables Among the Participants
| Mean ± SD | Context | Input | Process | Product | Age | The Average Percentage of the Maximum Achievable Score | |
|---|---|---|---|---|---|---|---|
| Context | 7.39 ± 2.32 | 1 | 49.2 | ||||
| Input | 15.86 ± 3.69 | 0.333* | 1 | 63.4 | |||
| Process | 33.48 ± 5.98 | 0.370** | 0.500** | 1 | 60.8 | ||
| Product | 41.12 ± 5.09 | 0.290* | 0.402** | 0.559** | 1 | 58.7 | |
| Age | 25.31 ± 2.50 | 0.056 | -0.080 | -0.028 | -0.097 | 1 | - |
| GPA | 16.47 ± 1.27 | 0.041 | 0.347** | 0.281** | 0.189 | -0.178 | - |
Correlation Between CIPP Model Domains with Students' Age and GPA
According to the obtained results, some domains of the CIPP model had positive, significant correlations at the significance level of 0.01 and 0.05%. However, age had no significant associations with the domains of the CIPP model, while GPA had positive, significant correlations with the input and process domains. In addition, the context domain had the lowest percentage of the mean maximum achievable score.
Table 3 shows the mean scores of the studied items in the CIPP model domains. Correspondingly, the items ‘appropriateness of the time of JC meetings’, ‘academic rank of the individual presenting the article’, ‘final evaluation of each session’, and ‘creating a spirit of participation among the students to improve the quality of education and research’ had the lowest average score respectively in the domains of context, input, process, and product from the perspective of the students. On the other hand, the items ‘relevance of the presented articles to the current issues of the field of study’, ‘regular and continuous JC schedules’, ‘time allocation to the individual reading of the articles at the beginning of the session’, and ‘skills in answering questions’ respectively had the highest average scores in the domains of context, input, process, and product in the viewpoint of the students. Table 4 shows the correlations between the background variables and the domains of the CIPP model.
| Domains/Items | Mean ± SD |
|---|---|
| Context | |
| The relevance of the presented articles to the issues of the day | 2.67 ± 1.03 |
| Appropriateness of the timing of the JC meetings | 2.17 ± 0.84 |
| Feeling the need to attend the JC meetings | 2.55 ± 0.99 |
| Input | |
| Existence of a regular and continuous program of holding the JC | 3.22 ± 1.06 |
| Develop of a regular and continuous program of evaluation the JC | 3.19 ± 1.30 |
| Develop appropriate regulations to encourage students and faculty to participate in the JC meetings | 3.18 ± 1.25 |
| Academic rank of the person presenting the article | 3.10 ± 1.10 |
| The level of up-to-date of the articles presenting | 3.17 ± 1.11 |
| Process | |
| Informing students about the JC | 3.02 ± 1.13 |
| How to send articles to participants | 3.28 ± 0.91 |
| State the goals at the beginning of each JC meetings | 3.24 ± 1.03 |
| Up-to-date selected articles | 3.17 ± 0.92 |
| Send articles at least one week before the presentation | 3.16 ± 1.02 |
| Discipline in the JC meetings | 2.95 ± 0.98 |
| Convenient the JC meetings place | 2.98 ± 1.11 |
| Audience group participation rate | 2.90 ± 1.25 |
| Structured critique of articles | 2.84 ± 1.07 |
| Final evaluation of each session | 2.59 ± 1.14 |
| Allocate time for individual reading of the article at the beginning of the session | 3.36 ± 1.26 |
| Product | |
| Motivate research in students | 3.14 ± 0.83 |
| Creating a spirit of participation among students to improve the quality of education and research | 2.55 ± 0.82 |
| Ability to select keywords and targeted search to find relevant articles | 2.86 ± 0.76 |
| Ability to select an optimal article from among the searched articles | 3.17 ± 0.88 |
| Understand the statistical results of data analysis | 2.88 ± 0.77 |
| Skills of criticizing scientific articles | 2.98 ± 0.86 |
| Understand study design | 2.86 ± 0.99 |
| Understanding of scientific goals and assumptions | 2.93 ± 0.74 |
| Skills to prepare appropriate slides | 3.05 ± 0.84 |
| Improving lecture technique | 3.17 ± 0.81 |
| Skills in answering questions | 3.28 ± 1.04 |
| Skills in how to summarize scientific articles | 2.78 ± 0.75 |
| Creating interest in participating in the next meetings | 2.59 ± 0.85 |
| Keep up to date with world scientific developments | 2.88 ± 0.90 |
Mean ± SD of the Items in CIPP Model Domains
| Variables | Context | Input | Process | Product |
|---|---|---|---|---|
| Gender | ||||
| Female | 7.11 ± 2.27 | 15.86 ± 3.89 | 33.97 ± 6.15 | 41.50 ± 4.46 |
| Male | 7.87 ± 2.37 | 15.87 ± 3.42 | 32.68 ± 5.74 | 40.50 ± 6.05 |
| t | -1.201 | -0.003 | 0.794 | 0.167 |
| P | 0.235 | 0.998 | 0.430 | 0.474 |
| Marital status | ||||
| Married | 6.71 ± 2.30 | 13.50 ± 4.51 | 31.07 ± 4.85 | 38.21 ± 3.80 |
| Single | 7.61 ± 2.31 | 16.61 ± 3.08 | 34.25 ± 6.14 | 42.02 ± 5.14 |
| t | -1.268 | -2.405 | -1.763 | -2.566 |
| P | 0.210 | 0.028 | 0.083 | 0.013 |
| Field of study | ||||
| Health education | 6.50 ± 2.57 | 15.16 ± 4.52 | 33.41 ± 7.91 | 39.33 ± 4.49 |
| Environmental health | 7.75 ± 1.87 | 15.75 ± 3.43 | 32.87 ± 4.88 | 40.12 ± 4.73 |
| Biostatics | 7.30 ± 1.94 | 15.10 ± 3.28 | 31.00 ± 3.97 | 41.00 ± 2.58 |
| Epidemiology | 7.36 ± 2.61 | 16.00 ± 3.94 | 34.81 ± 5.36 | 43.81 ± 5.11 |
| Nutrition | 8.11 ± 2.80 | 17.66 ± 3.20 | 35.77 ± 7.32 | 42.11 ± 7.57 |
| F | 0.743 | 0.742 | 0.936 | 1.417 |
| P | 0.567 | 0.566 | 0.450 | 0.241 |
| School | ||||
| Health | 7.26 ± 2.23 | 15.53 ± 3.71 | 33.06 ± 5.69 | 40.93 ± 4.59 |
| Nutrition sciences and food industry | 8.11 ± 2.80 | 17.66 ± 3.20 | 35.77 ± 7.32 | 42.11 ± 7.57 |
| t | -1.004 | -1.616 | -1.258 | -0.631 |
| P | 0.320 | 0.112 | 0.213 | 0.531 |
| Semester | ||||
| 1 | 6.60 ± 2.50 | 14.90 ± 3.87 | 32.60 ± 7.19 | 40.00 ± 4.80 |
| 3 | 7.72 ± 2.33 | 17.22 ± 2.11 | 33.00 ± 5.90 | 41.54 ± 5.38 |
| 5 | 7.39 ± 2.36 | 15.39 ± 4.36 | 34.60 ± 5.69 | 41.26 ± 5.38 |
| 7 | 7.66 ± 1.52 | 12.66 ± 4.72 | 31.33 ± 6.02 | 40.66 ± 2.08 |
| F | 0.540 | 2.234 | 0.507 | 0.216 |
| P | 0.657 | 0.095 | 0.679 | 0.885 |
| History of participating in research method workshops | ||||
| No | 7.25 ± 2.34 | 15.58 ± 2.57 | 31.08 ± 4.87 | 39.25 ± 3.04 |
| Yes | 7.43 ± 2.35 | 15.93 ± 3.95 | 34.10 ± 6.13 | 41.60 ± 5.43 |
| t | -0.243 | -0.291 | -1.580 | -1.440 |
| P | 0.809 | 0.772 | 0.120 | 0.155 |
| Frequency of attending in the JC meetings | ||||
| 3 | 7.00 ± 2.32 | 14.89 ± 3.97 | 31.64 ± 5.53 | 39.67 ± 4.05 |
| 4 | 7.47 ± 2.01 | 16.61 ± 3.62 | 34.23 ± 5.98 | 40.80 ± 4.49 |
| 5 and more | 8.44 ± 2.87 | 17.11 ± 2.14 | 37.44 ± 5.59 | 46.33 ± 6.42 |
| F | 1.351 | 1.983 | 3.805 | 7.120 |
| P | 0.267 | 0.147 | 0.028 | 0.002 |
Background Variables and the Domains of the CIPP Model a
5. Discussion
The present study aimed to evaluate the quality of JC meetings from the perspective of the postgraduate students of KUMS using the CIPP evaluation model, and the findings indicated positive, significant correlations between the CIPP domains. Consistently, Hosseini et al. conducted a study on 473 Iranian faculty members to evaluate Shahid Motahari Educational Festival during 2008 - 2013 using the CIPP model, and the obtained results indicated significant correlations between the CIPP domains (17). Therefore, it could be inferred that various domains of this evaluation model are correlated, and all the domains should be considered to achieve optimal outcomes.
According to the current research, the context domain had the lowest mean score, which indicated the poorer evaluation status compared to the other domains from the perspective of the students. In the study conducted by Sajjadi et al. (25) regarding the evaluation of patient registries and complain systems using the CIPP model, the highest scores were reported in the input, process, product, and context domains, respectively, which is consistent with the results of the present study. In addition, Ali-Mohammadi et al. (26) assessed female secondary school students in Ahvaz city (Iran) using the CIPP model and identified some issues in the domains of context, input, process, and product, respectively. The low mean score of the context domain in the present study could be due to the dissatisfaction of the students with the time of the event and the presented contents in the JC meetings, which require special attention by JC planners.
In the current research, the items of ‘appropriateness of the time of JC meetings’, ‘academic rank of the individual presenting the article’, ‘final evaluation of each session’, and ‘creating a spirit of participation among the students to improve the quality of education and research’ respectively achieved the lowest mean scores in the domains of context, input, process, and product in the viewpoint of the students. Therefore, it could be inferred that the students did not consider the time of holding the JC meetings to be desirable, and this issue requires attention for the more appropriate implementation of JC meetings.
With regard to the input domain, the item ‘academic rank of the individual presenting the article’ was not considered desirable by the students; since JC is presented by students, this was quite predictable. Therefore, the presentation of JC should not be completely left to students, and the presenting student should do the presentation with the help of the supervisor. It is also suggested that presentations be prepared by academic members with an emphasis on presenting updated, high-quality articles.
The final evaluation at the end of a JC meeting by the academic members could increase the quality of JC. In this regard, Masjedi et al. (24) conducted a study to assess the effects of JC through new methods (presence of specialists in community medicine and/or statistics) and traditional methods (without the presence of specialists in community medicine and/or statistics) on the attitude of the anesthesiology residents of Shiraz University of Medical Sciences (Iran), reporting a statistically significant difference in the group using the new methods compared to routine JC meetings. In the mentioned study, the students claimed that the most important effects of the new JC methods compared to the traditional methods were the better understanding of statistical terms, study methods, positive attitudes toward the usefulness of interdisciplinary communication article reviews, article critique in applying the results in clinical practice, developing article critique skills, motivating more research, interest in attending the next JC, and advising other colleagues to attend the meetings. This is in congruence with the results of the present study, highlighting the need for the summarization and evaluation of JC meetings by academic members.
Our findings also indicated that the frequency of attending JC meetings had a significant correlation with the increased score of the product domain among the participants. This is consistent with the study by Masjedi et al. (24), which demonstrated that the higher frequency of attending JC meetings was associated with the enhanced knowledge of the subjects, as well as a higher demand for more meetings. This finding highlights the need for attention and participation in JC meetings in order to increase the educational and research capabilities of students. Given the importance of holding JC to promote the educational and research skills of students and the need for careful planning in this regard, it is suggested that the results of the present study be applied in other KUMS schools and the outcomes be presented to other KUMS departments for better planning.
One of the limitations of the present study was data collection using a questionnaire due to the possibility of bias on behalf of the respondents. In addition, our study was conducted in two of the seven schools of KUMS, and the findings may not be generalized to the other schools of the university.
5.1. Conclusion
From the perspective of the students, the context domain encompassed the relevance of the presented articles to the issues of the day, appropriateness of the time of JC meetings, and feeling the need to attend the JC meetings. This domain received the lowest mean score compared to the other domains, which highlights the need for more careful planning in the implementation of JC meetings.
