Modelling approaches to combining and comparing independent adaptive comparative judgement ranks
Keywords:Adaptive Comparative Judgement, Assessment, Steady State, Authentic Evidence
The use of Adaptive Comparative Judgement (ACJ) for educational assessment addresses one need within technology education for the reliable assessment of responses to open-ended activities which are characteristic within the field. The output of an ACJ session is a rank order of the piece of student work with relative “ability scores”. However, the use of ACJ has been limited to date in that ranks are not directly comparable. For example, a rank produced from one class group has no reference information against which to compare a rank produced of the work of another class group. In this type of case a solution has been to combine the work of both classes into one ACJ session, but this has limitation when considering scaling up.
A new goal for the use of ACJ involves solving this issue. The ability to compare or merge ranks presents a new capacity for ACJ – to use a rank as a “ruler” against which other ranks can be compared. In practice this would allow for two possibilities. The first is that a single rank could be developed which presents a national standard against which teachers could compare the work of their students to see where they are performing on a national level. The second is that communities of practice could complete ACJ sessions within their own classrooms, and when meeting as a group they could merge and compare relative performance of their own students to support professional development.
In a previous article a proof of concept of this process conducted via simulation was presented (Buckley and Canty, 2022). In this article we present the results of a project with authentic data – student work completed in response to meaningful activities with teachers acting as ACJ judges – which indicate that the use of ACJ in this way is now possible.
How to Cite
Copyright (c) 2023 Jeffrey Buckley, Niall Seery, Richard Kimbell
This work is licensed under a Creative Commons Attribution 4.0 International License.
This journal provides immediate open access to its content with no submission or publications fees. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a LicenceCreative Commons Attribution License that allows others to read, download, copy, distribute, print, search, or link to the full text of works in this journal. It also allows others to remix, adapt and build upon the work, as long as credit is given to the author(s).