Repeated item development training helps faculty better
predict and adjust item difficulty in medical education
assessments
BUSAN,
South Korea, July 2, 2024
/PRNewswire/ -- Evaluation is essential to learning, as it
determines whether students have grasped the taught concepts. The
item difficulty index (the ratio of correct responses to total
responses) plays a key role in this process, providing a measure of
test difficulty. Educators' ability to create items and predict
their difficulty indices significantly impacts evaluation.
Educators often overestimate item difficulty, underscoring the need
for better predictive skills. However, research on faculty
development programs to enhance this ability is limited.
To address this gap, a team of researchers at Pusan National University led by Professor Sang Lee, Vice President for Medical Affairs,
Professor of Medical Education, and Professor of Family Medicine,
conducted a study, the findings of which were published online in
BMC Medical Education on May 30,
2024. The study investigated whether repeated item
development training for medical school faculty improved their
ability to predict and adjust the difficulty level of
multiple-choice questions (MCQs).
Explaining the background of their study, Prof. Lee elucidates,
"Just like the final stroke completes and perfects a painting,
education is perfected through evaluation. Medical school
faculty members cannot create high-quality items without proper
training. Item development training is essential, and this study
demonstrates that the effectiveness of such training increases with
repetition."
Item development workshops were conducted with 62 participants,
first in 2016 and later in 2018. The estimated accuracies of item
difficulty predictions were compared. Before the workshop, the
teachers developed newly drafted items which were then reviewed. An
item development committee trained the faculty members by offering
continuous feedback and helping them revise the newly drafted items
according to the national exam standards, with an ideal difficulty
range and an application-based focus. Furthermore, the difficulty
indices predicted by the participants were compared with
fourth-year medical student evaluation analyses.
The study found that before the training, significant agreement
between the predicted and actual item difficulty indices was
observed for only cardiology. After training, significant agreement
was observed for cardiology, neurology, internal medicine, and
preventative medicine. These findings suggest systematic and
effective training can improve the quality of MCQ assessments in
medical education.
Repeated training sessions significantly enhanced faculty
members' ability to predict and adjust item difficulty levels
accurately, leading to effective assessments and better educational
outcomes. Despite the benefits, sustaining them might be
challenging due to the workshop's three-day duration and busy
faculty schedules. However, trained educators will be better
equipped to create and adjust items, improving assessment practices
across all academic fields. In conclusion, continuous faculty
development programs are essential to ensure the creation of
appropriate items aligned with evaluation purposes.
Talking about the potential applications of their study, Prof.
Lee shares, "Repeated item development training not only helps
adjust the difficulty level but also enhances the construction of
the items, increases their discriminating power, and properly
addresses the issue of validity." He further adds, "Soon
there will be an era of item development using AI. For that,
studies like ours are important for providing necessary information
about existing items and students' answer data, which will help in
developing an AI-powered automated item development
program."
Reference
Title of original paper: The impact of repeated item development
training on the prediction of medical faculty members' item
difficulty index
Journal: BMC Medical Education
DOI: https://doi.org/10.1186/s12909-024-05577-x
About the institute
Website:
https://www.pusan.ac.kr/eng/Main.do
Contact:
Jae-Eun
Lee
82 51 510 7928
379851@email4pr.com
View original content to download
multimedia:https://www.prnewswire.com/news-releases/pusan-national-university-researchers-assess-the-impact-of-repeated-item-development-faculty-training-on-item-difficulty-prediction-302187097.html
SOURCE Pusan National University