Argument chains as a tool to improve automated scoring of argumentative writing

The study introduces argument chains, a theory-driven analytic tool we developed to improve the automated assessment of argumentative writing. By attending to both the structure and content of written arguments, the argument-chains approach overcomes key limitations of structural analyses commonly used in writing assessment. Drawing on a dataset of 471 argumentative essays written by Grade 5 students, we used argument chains to reconstruct students’ multi-step reasoning, identify implicit assumptions, and assign acceptability and relevance scores to individual propositions. This analysis enabled us to pinpoint specific weaknesses in argumentation at the level of individual propositions, generating precise diagnostic information to support actionable instructional feedback. It also helped us evaluate multiple essay-level dimensions of student performance, including overall argument quality, acceptability, relevance, and consideration of opposing perspectives. Reliability studies indicated high agreement between human raters (Quadratic Weighted Kappa =.84) and high agreement between human raters and large language models (Quadratic Weighted Kappa =.81). Most discrepancies involved assumed propositions, highlighting an area that requires further investigation. While this work is in its early stages, we suggest that argument chains have the potential, with further automation, to become a broadly applicable tool for assessing argumentative writing.