A Tale of a Bad Quiz


Recently, my BBA 407 class took its first quiz. The average score was 11.5 points below that of last semester with a standard deviation of 4.3 points less than last semester. Moreover, realized learning to date, based on progress in areas assessed at the beginning of the semester, was just under 50% of last semester’s comparable figure (19.0% vs. 38.6%). The latter figure was, by far, the more concerning, especially as it is a standardized measure. In addition, six students were “flagged” as being at risk of possibly failing the course vs. one last semester.

Why did this happen? Were the students in this class notably less prepared than those in the preceding class? If so, had they reached a learning barrier that would significantly undermine their progress?

Statistically, the prospects of such an outcome should be quite low if admissions standards are held constant or slowly increasing and course standards are held reasonably constant. Attrition over the prior three years should result in classes that do not deviate too much by the time they reach their fourth year of study.

Understanding what happened would have large implications for the remainder of the course. Therefore, it made a lot of sense to look deeper into the issue.

To address the issue as to whether the current class is much less ready than the preceding class, I went back to the pre-semester diagnostic exam results. The data showed that both classes were remarkably similar in terms of their incoming characteristics. The learning potential for the fall 2015 class was 0.592 vs. 0.600 for the spring 2015 class. Even against classes of the past five semesters, the current class was very similar in terms of its knowledge at the entry point of the course. These outcomes would be very unlikely if the class were qualitatively less prepared than preceding classes. Therefore, some other factor or factors had to explain the disparity of outcomes, especially as the instructional materials and teaching approach were similar across the semesters.

As a result, attention had to be focused on what students were doing to prepare for the class. Although such information is inherently limited, as a lot of preparation takes place outside the classroom, two proxies can provide a lot of insight: homework and attendance. As the quiz was focused on Chapters 1-3, the completion rate for homework assignments for those chapters and the absentee rate on the day the homework was due were compared. The differences were large.

The current class had completed only 56.3% of its homework assignments (chapters 1-3). Last semester, 80.2% of homework assignments were completed. None of my previous classes had such a low homework completion rate.

In terms of attendance, the absentee rate on the days on which the homework assignments were due was 19.8%. Put another way, nearly 1-in-5 students was absent. Last semester, the absentee rate on those days was 11.5%.

Assuming that other factors may explain 10%-40% of a student’s learning, the differences in homework completion rates and attendance were put into a simple model to test the expected realized learning. The preceding class was used as the reference point. The simple model indicated that realized learning to date for the current class would fall between 17.3% and 22.0%. The actual figure fell well within that range. That outcome suggests that the deterioration in student performance relative to last semester’s outcome is largely related to students’ lack of completion of assignments and reduced attendance.

Addressing those matters is an easier challenge than one that the one that could have been presented by a lack of ability. The outcomes will be discussed in tomorrow’s class. In addition, interventions will be required. For example, the treatment of homework assignments will need to be modified. Typically, if a student completes an assignment, it adds to his or her grade. Failure to complete it results in no accumulation of points. An adjustment would entail a failure to complete the assignment resulting in the negative value of the points an assignment is worth. For example, if completion of an assignment led to 1.5 points toward a student’s grade, failure to complete it would result in -1.5 points.

The data will be re-evaluated at the end of the semester.