Modelling approaches to combining and comparing independent adaptive comparative judgement ranks

Abstract

The use of Adaptive Comparative Judgement (ACJ) for educational assessment addresses one need within technology education for the reliable assessment of responses to open-ended activities which are characteristic within the field. The output of an ACJ session is a rank order of the piece of student work with relative “ability scores”. However, the use of ACJ has been limited to date in that ranks are not directly comparable. For example, a rank produced from one class group has no reference information against which to compare a rank produced of the work of another class group. In this type of case a solution has been to combine the work of both classes into one ACJ session, but this has limitation when considering scaling up.

A new goal for the use of ACJ involves solving this issue. The ability to compare or merge ranks presents a new capacity for ACJ – to use a rank as a “ruler” against which other ranks can be compared. In practice this would allow for two possibilities. The first is that a single rank could be developed which presents a national standard against which teachers could compare the work of their students to see where they are performing on a national level. The second is that communities of practice could complete ACJ sessions within their own classrooms, and when meeting as a group they could merge and compare relative performance of their own students to support professional development.

In a previous article a proof of concept of this process conducted via simulation was presented (Buckley and Canty, 2022). In this article we present the results of a project with authentic data – student work completed in response to meaningful activities with teachers acting as ACJ judges – which indicate that the use of ACJ in this way is now possible.