Item ordering and computerized classification tests with cluster-based scoring

An investigation of the countdown method

Matthew D. Finkelman, Sarah Lowe, Wonsuk Kim, Oliver Gruebner, Niels Smits, Sandro Galea

Research output: Contribution to journalArticleResearchpeer-review

1 Citation (Scopus)

Abstract

The countdown method is a well-known approach to reducing the average length of screening instruments that are presented by computer. In the countdown method, testing is terminated once the result of the screener ("positive" or "negative") has been unambiguously determined from prior answers. Previous research has examined whether presenting dichotomously scored items in order from "least to most frequently endorsed" or "most to least frequently endorsed" is more efficient when the countdown method is used. The current study describes the Mean Score procedure, an extension of the above item ordering procedures to polytomously scored items, and evaluates its efficiency relative to the distribution of other possible item orderings in 2 real-data simulations. Both simulations involve item responses to the Posttraumatic Stress Disorder (PTSD) Checklist for DSM-5 (PCL-5). In the first simulation, items were scored polytomously, and a single cutoff point was used to determine the screening result. In the second simulation, items were converted to dichotomous scores, as well as categorized into 4 clusters; a positive result for the entire assessment was obtained if and only if a positive result was obtained for each cluster. The latter simulation also investigated the effect of reordering the clusters themselves on the efficiency of the countdown method. Results indicated that the Mean Score procedure does not necessarily produce the optimal ordering, but tends to assemble an efficient item ordering relative to the distribution of possible orderings. In the second simulation, reordering the clusters themselves affected efficiency. Future research directions are suggested.

Original languageEnglish
Pages (from-to)204-219
Number of pages16
JournalPsychological Assessment
Volume30
Issue number2
DOIs
StatePublished - 1 Feb 2018

Fingerprint

Post-Traumatic Stress Disorders
Checklist
Research
Direction compound

Keywords

  • Computerized adaptive testing
  • PCL-5
  • Respondent burden
  • countdown method

Cite this

Finkelman, Matthew D. ; Lowe, Sarah ; Kim, Wonsuk ; Gruebner, Oliver ; Smits, Niels ; Galea, Sandro. / Item ordering and computerized classification tests with cluster-based scoring : An investigation of the countdown method. In: Psychological Assessment. 2018 ; Vol. 30, No. 2. pp. 204-219.
@article{2a0999ffdffb4e5ca87c865a2e26128c,
title = "Item ordering and computerized classification tests with cluster-based scoring: An investigation of the countdown method",
abstract = "The countdown method is a well-known approach to reducing the average length of screening instruments that are presented by computer. In the countdown method, testing is terminated once the result of the screener ({"}positive{"} or {"}negative{"}) has been unambiguously determined from prior answers. Previous research has examined whether presenting dichotomously scored items in order from {"}least to most frequently endorsed{"} or {"}most to least frequently endorsed{"} is more efficient when the countdown method is used. The current study describes the Mean Score procedure, an extension of the above item ordering procedures to polytomously scored items, and evaluates its efficiency relative to the distribution of other possible item orderings in 2 real-data simulations. Both simulations involve item responses to the Posttraumatic Stress Disorder (PTSD) Checklist for DSM-5 (PCL-5). In the first simulation, items were scored polytomously, and a single cutoff point was used to determine the screening result. In the second simulation, items were converted to dichotomous scores, as well as categorized into 4 clusters; a positive result for the entire assessment was obtained if and only if a positive result was obtained for each cluster. The latter simulation also investigated the effect of reordering the clusters themselves on the efficiency of the countdown method. Results indicated that the Mean Score procedure does not necessarily produce the optimal ordering, but tends to assemble an efficient item ordering relative to the distribution of possible orderings. In the second simulation, reordering the clusters themselves affected efficiency. Future research directions are suggested.",
keywords = "Computerized adaptive testing, PCL-5, Respondent burden, countdown method",
author = "Finkelman, {Matthew D.} and Sarah Lowe and Wonsuk Kim and Oliver Gruebner and Niels Smits and Sandro Galea",
year = "2018",
month = "2",
day = "1",
doi = "10.1037/pas0000470",
language = "English",
volume = "30",
pages = "204--219",
journal = "Psychological Assessment",
issn = "1040-3590",
publisher = "American Psychological Association Inc.",
number = "2",

}

Item ordering and computerized classification tests with cluster-based scoring : An investigation of the countdown method. / Finkelman, Matthew D.; Lowe, Sarah; Kim, Wonsuk; Gruebner, Oliver; Smits, Niels; Galea, Sandro.

In: Psychological Assessment, Vol. 30, No. 2, 01.02.2018, p. 204-219.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Item ordering and computerized classification tests with cluster-based scoring

T2 - An investigation of the countdown method

AU - Finkelman, Matthew D.

AU - Lowe, Sarah

AU - Kim, Wonsuk

AU - Gruebner, Oliver

AU - Smits, Niels

AU - Galea, Sandro

PY - 2018/2/1

Y1 - 2018/2/1

N2 - The countdown method is a well-known approach to reducing the average length of screening instruments that are presented by computer. In the countdown method, testing is terminated once the result of the screener ("positive" or "negative") has been unambiguously determined from prior answers. Previous research has examined whether presenting dichotomously scored items in order from "least to most frequently endorsed" or "most to least frequently endorsed" is more efficient when the countdown method is used. The current study describes the Mean Score procedure, an extension of the above item ordering procedures to polytomously scored items, and evaluates its efficiency relative to the distribution of other possible item orderings in 2 real-data simulations. Both simulations involve item responses to the Posttraumatic Stress Disorder (PTSD) Checklist for DSM-5 (PCL-5). In the first simulation, items were scored polytomously, and a single cutoff point was used to determine the screening result. In the second simulation, items were converted to dichotomous scores, as well as categorized into 4 clusters; a positive result for the entire assessment was obtained if and only if a positive result was obtained for each cluster. The latter simulation also investigated the effect of reordering the clusters themselves on the efficiency of the countdown method. Results indicated that the Mean Score procedure does not necessarily produce the optimal ordering, but tends to assemble an efficient item ordering relative to the distribution of possible orderings. In the second simulation, reordering the clusters themselves affected efficiency. Future research directions are suggested.

AB - The countdown method is a well-known approach to reducing the average length of screening instruments that are presented by computer. In the countdown method, testing is terminated once the result of the screener ("positive" or "negative") has been unambiguously determined from prior answers. Previous research has examined whether presenting dichotomously scored items in order from "least to most frequently endorsed" or "most to least frequently endorsed" is more efficient when the countdown method is used. The current study describes the Mean Score procedure, an extension of the above item ordering procedures to polytomously scored items, and evaluates its efficiency relative to the distribution of other possible item orderings in 2 real-data simulations. Both simulations involve item responses to the Posttraumatic Stress Disorder (PTSD) Checklist for DSM-5 (PCL-5). In the first simulation, items were scored polytomously, and a single cutoff point was used to determine the screening result. In the second simulation, items were converted to dichotomous scores, as well as categorized into 4 clusters; a positive result for the entire assessment was obtained if and only if a positive result was obtained for each cluster. The latter simulation also investigated the effect of reordering the clusters themselves on the efficiency of the countdown method. Results indicated that the Mean Score procedure does not necessarily produce the optimal ordering, but tends to assemble an efficient item ordering relative to the distribution of possible orderings. In the second simulation, reordering the clusters themselves affected efficiency. Future research directions are suggested.

KW - Computerized adaptive testing

KW - PCL-5

KW - Respondent burden

KW - countdown method

UR - http://www.scopus.com/inward/record.url?scp=85015166612&partnerID=8YFLogxK

U2 - 10.1037/pas0000470

DO - 10.1037/pas0000470

M3 - Article

VL - 30

SP - 204

EP - 219

JO - Psychological Assessment

JF - Psychological Assessment

SN - 1040-3590

IS - 2

ER -