The material was created within the framework of the project “Do not Believe Myths” with the support of the International Renaissance Foundation.
We all want to receive treatment from a highly qualified, involved doctor, who really wants us to get healthy and never need their services again. But where do you find a specialist like that, where do they come from? There is only one possible answer to this question: first of all, a good doctor in the future is a good medical student in the present.
Of course, a lot depends on the student's motivation, their talent and the ability to learn and analyze. So what should be the instruments which can be used to assess academic performance of a medical student and their future as a doctor? We are referring to the system of exams and tests. They help determine whether today's student will make a good doctor tomorrow.
VoxUkraine analysts tried to learn about the exams we have in Ukraine and, most importantly, the way they affect future qualification of doctors.
The system of assessment at medical schools is an extremely important factor in medical education – both in Ukraine and abroad. Ukrainian medical students have been passing KROK exam since 1998:
The exams have been developed in close international cooperation and are based on the best world practices. It was the first success story of independent assessment development in Ukraine, which was also used when External Independent Assessment was launched in early 2000s.
Yet, despite the use of best world practices, students, teachers and doctors are often critical about Krok exams.
Most complaints concern the quality of tests. In response to criticism, Testing Centre was continuously saying that the tests were prepared by university teachers so they should be the ones to answer questions. In their turn, the teachers, are complaining, and rightfully so, about lack of payment for the complicated task of developing tests. Obviously, not all quality issues have to do with the level of teacher motivation. There are questions about the syllabi that the exam questions are based on and inconsistencies of answers to the same question in different exam tasks. All things considered, it becomes apparent that the existing model of preparation for the test exam cannot guarantee quality result.
Apart from complaints about the quality of tests, students and teachers constantly express doubt as to the level of transparency and objectivity of Krok assessment. They are voicing various sums needed to `deal with this`, giving names of people that can allegedly help and retelling stories of those who allegedly `got everything handled`. At the same time, over the 20 years of Krok existence no one was able to prove any of those suspicions or allegations.
In the summer of 2018, Ministry of Healthcare of Ukraine (MoH) asked Testing Centre to provide complete information on Krok exam results. In view of planned changes in the assessment of medical students, MoH wanted to do a more comprehensive analysis of the academic results. Authors of this article got the resulting depersonalized data.
All in all, it includes information about more than 197,000 exams taken between 2009 and 2018 (yet, there are years or exams for which data was not supplied).
One of the key issues that most people pay attention to when analyzing tests is how natural the distribution of points is. ‘Normal (or natural) distribution’ is one of the key concepts in statistics. Back in the 19th century mathematicians noticed that for numerous samples of observations graphic presentation of probability is bell-shaped. Such distribution is called ‘normal’ or ‘Gaussian’ after the mathematician who formulated this key concept of statistics for the first time.
What does this mean for test studies? There is a certain ‘normal distribution’ of the results: a small number of persons receive the lowest points, a slightly higher result goes to a slightly bigger number of people and as the result increases the number of people who get them increases. As it reaches a certain point, the value starts decreasing so a very small number of people gets the highest point.
When analyzing the distribution of Krok results, a few things come to the foreground. A small number of students gets the lowest number of points which corresponds to the number of correct answers. As the number of points increases so does the number of people who get them. Then, however, the number of people who got a certain number of points rapidly drops.
Figures for Krok 1 (Fig. 2) show a `gap` around 50% mark for the years 2009-2014. However, the number of people who got 50.5% in 2009-2014 is disproportionately high – higher than the number of people who got 51%. In 2015 (Fig. 1) the same situation may be observed but for a different part of the curve: 60% - a rapid decrease of the number of people; 60.5% - a high number of people.
As a matter of fact, we are witnessing a disproportionately higher number of students who got a specific number of points. At the same time, the `abnormal` increase clearly coincides with the `pass/not pass` threshold every time. Thus, in 2015 the `pass/not pass` threshold increased from 50.5% to 60.5% - and the jump reached exactly the same level. A similar situation is seen with Krok 2 and Krok 3.
What we are seeing on the chart is called discontinuity distribution. Scientists have already researched this phenomenon. In particular, such research was conducted to assess the values of regulatory procedures compliance. Research shows that an inexplicably large share of companies find themselves right before the threshold value which signifies implementation of regulations. Researchers have also analyzed such distribution to see how people optimize their taxes. Many people have lower than threshold value income, where the threshold value stands for the next level of tax rate.
What are the trends that we see in analyzed data?
Such a distribution shows that the test is comparatively easier and most students reach the ‘pass/not pass’ threshold easily. In general, Krok 2 results show that a very small share of students get less points than what is required to reach the ‘pass/not pass’ threshold.
|Year/Krok||Krok 1||Krok 2||Krok 3|
Since Krok 2 is taken during the final year of studies, the price of failure is a lot higher. A student who failed Krok 1 during the third year, continues with their studies and can retake the exam in the process. If a student fails Krok 2 during the sixth year, they cannot be awarded a diploma and need to postpone their studies in internship programme. Perhaps, the high price of failure causes formulation of such exam tasks which will not create problems for too many students.
The results of previous analysis were sent to the Ministry of Healthcare of Ukraine to receive official explanation for such a distribution. In several letters, sent upon request of MoH, Testing Centre explained the deviation by the following:
Analysis of Krok exam for medical students showed outliers in result distribution. A rapid jump on ‘pass/not pass’ threshold is observed practically for all the years. Yet, there are differences: in some years there are no students who got 0.5 less that the threshold value, in other years the number of such students is smaller than could be expected if we look at the distribution curve. Since terms of reference for software development that calculates the results is not available, it is impossible to determine the reason for such results for sure.
Response of the Testing Centre shows that the decisions about assessment procedures were taken ad hoc, depending on the situation. Thus, the Centre could make decisions about force majeure circumstances as they saw fit and adjust the approach to the assessment. According to Testing Centre representatives, that was the procedure developed in 2015 for the displaced universities. At the same time, the very possibility of making such decisions demonstrates absence of a single established approach toward determining exam results and the possibility to change the approach toward internal decisions of the Testing Center. Additional evidence of this is absence of the terms of reference for software which is used to determine the results.
Yet, what is more important is how Krok exams can actually eliminate the less bright students. The analysis also shows that Krok 2 is easier in comparison with Krok 1, and it is failed by a very small percentage of students.
Even though Krok 1 is more complicated, when students retake it they use test compilations for the previous years so the fact that they take the test cannot guarantee proper assessment of their knowledge as medical students.
As a result, test architecture causes doubts as to their efficiency as a method of assessment. If Krok 1 can be retaken under simplified conditions and Krok 2 is intentionally made easier, does it mean that Krok exams are not efficient as preventive method to keep unqualified students away from medical profession.
A year ago, upon initiative of the Ministry of Healthcare of Ukraine, a Decree was passed, which changes the system of student assessment completely. In addition to Krok exams, during the third year of studies, students will also pass English language test and IFOM – International Foundation of Medicine exam (basic disciplines) which is conducted by the National Board of Medical Examiners of the USA. This organization runs licensed exams for American doctors. During graduation year, in addition to Krok, the students will take IFOM (clinical disciplines) and an objective structured clinical exam that is aimed at checking the level of their practical skills.
On March 15 Ukrainian medical students will be passing International Foundations of Medicine Exam (IFOM) on the national scale for the first time. The results will not affect their grade but will give a chance to compare their knowledge with that of students from other countries.
Already a year ago, the possibility of giving up Krok exams was discussed. Finally, a decision was made to keep them but create preconditions for improvement of test quality, primarily via change of funding mechanisms. In particular, funds have been allocated to pay for the development of such tests. Apparently, approaches towards administration of exams should also be reviewed to guarantee transparency of procedures and approaches toward assessment.
Ми віримо, що слова мають силу, а ідеї – визначний вплив. VoxUkraine об’єднує найкращих економістів та допомагає їм доносити ідеї до десятків тисяч співвітчизників. Контент VoxUkraine безкоштовний (і завжди буде безкоштовним), ми не продаємо рекламу та не займаємось лобізмом. Щоб проводити більше досліджень, створювати нові впливові проекти та публікувати багато якісних статей, нам потрібні розумні люди і гроші. Люди є! Підтримай VoxUkraine. Разом ми зробимо більше!